What Organizations Must Implement When Employees Use Generative AI to Draft or Respond to Customer Service Requests Containing Personal Information

The Subject, Stated Plainly

Generative AI tools used by customer service teams to draft or respond to requests frequently ingest personal information: names, contact details, account numbers, transaction histories, health information, financial data. The organization using the tool remains accountable for how that information is collected, processed, shared with third parties, retained, and eventually disposed of. The AI vendor's privacy policy does not satisfy the organization's obligation. Leadership must establish what data may be submitted, what vendor commitments are required, what training staff receive, and what activity is logged.

Why It Matters to the Business

The Federal Trade Commission holds organizations accountable when they fail to honor their privacy promises or maintain security appropriate to the sensitivity of the data they possess. If customer service staff submit personal information to a generative AI tool without proper agreements, logging, or training, the organization has created a data handling pathway it cannot account for, cannot audit, and may not be able to explain if regulators or customers inquire.

The consequences are regulatory risk, reputational harm, and the discovery that leadership was accountable for a practice it did not design and cannot measure. Customer service operations move quickly. Individual staff make real-time decisions about what to submit to AI tools. Without clear rules, vendor agreements that address data use, and documented training, those decisions accumulate into organizational exposure.

What Data Handling Obligations Apply

Organizations that make privacy claims in their policies or customer communications must honor them. The FTC Act prohibits unfair and deceptive practices. If an organization's privacy policy states that customer data will be handled in a certain way, submitting that data to a third-party AI vendor without appropriate agreements or controls contradicts the stated practice.

Beyond express promises, organizations have an obligation to maintain security appropriate to the nature of the data they possess. Personal information submitted to a generative AI tool enters a processing environment the organization does not control. Without contractual commitments governing how the vendor uses, retains, or shares that data, the organization cannot demonstrate appropriate protection.

If the organization handles health-related information and experiences a breach, the Health Breach Notification Rule may apply, requiring specific steps following the breach. If the organization is a financial institution or offers financial products, the Gramm-Leach-Bliley Act requires safeguarding sensitive data and explaining information-sharing practices to customers.

The NIST Privacy Framework describes privacy risk management as identifying and managing risks to individuals arising from data processing. Submitting personal information to a generative AI tool is a data processing activity. Leadership must identify what risks that activity creates, what controls reduce those risks to an acceptable level, and how compliance is verified.

What Vendor Agreements Must Include

Standard consumer terms of service for generative AI tools do not establish the commitments required when an organization submits customer personal information. Leadership must ensure that agreements with AI vendors address the following:

• What data the vendor may collect and for what purposes
• Whether submitted data is used to train or improve the vendor's models
• How long the vendor retains data and under what conditions it is deleted
• Whether the vendor shares data with other parties and under what terms
• What security controls the vendor maintains to protect submitted data
• What notification the organization receives in the event of a breach
• What audit or verification rights the organization retains
• How the agreement terminates and what happens to data upon termination

If the vendor's standard agreement does not address these elements, leadership must negotiate supplemental terms or prohibit submission of personal information until acceptable commitments are in place. Relying on a vendor's general privacy policy does not satisfy the organization's accountability for how customer data is handled.

What Training Is Required

Customer service staff must understand what information may and may not be submitted to generative AI tools. Training must cover:

• What constitutes personal information in the context of the organization's operations
• What generative AI tools are approved for use and under what conditions
• What data may be submitted to approved tools and what must be excluded or redacted
• What to do when uncertain whether a request involves personal information
• What logging or documentation is required when AI tools are used
• Who to contact when questions arise about appropriate use

Training must be documented, including who received it and when. Periodic refresher training is necessary as AI tools, vendor agreements, and organizational policies change. Leadership must verify that training occurs, not assume it.

What Must Be Logged

Organizations must maintain records sufficient to demonstrate what personal information was submitted to AI tools, when, by whom, and for what purpose. Logging serves three functions: it enables detection of inappropriate use, supports incident response if a breach occurs, and provides evidence of compliance if regulators inquire.

At a minimum, logging should capture:

• User identity and timestamp for each interaction with approved AI tools
• Whether the interaction involved customer data
• What type of request was being handled
• Any redaction or anonymization applied before submission

Log retention must align with regulatory requirements and the organization's data retention policies. Leadership must verify that logs are reviewed, not simply collected.

Who Inside the Organization Is Accountable

This subject sits at the intersection of customer service operations, privacy compliance, information security, vendor management, and legal review. Without a single owner, responsibility diffuses and controls do not get implemented.

Adequate ownership looks like a designated executive who is accountable for:

• Defining what personal information may be submitted to AI tools and under what conditions
• Ensuring vendor agreements contain the required data handling commitments before tools are approved for use
• Verifying that training occurs and is documented
• Confirming that logging is in place, reviewed, and retained appropriately
• Reporting to leadership on compliance and any identified gaps
• Coordinating response if a breach or regulatory inquiry occurs

In many organizations, this accountability is assigned to a Chief Information Security Officer or Chief Privacy Officer. In organizations without dedicated privacy or security leadership, the accountability often falls to the General Counsel, Chief Compliance Officer, or Chief Operating Officer. What matters is that one person is clearly responsible and has authority to enforce decisions across customer service, IT, legal, and vendor management.

Where no internal executive has the time, expertise, or authority to own this accountability, a [virtual CISO (vCISO)](/vciso/) provides executive-level ownership on a fractional basis. A vCISO defines the governance structure, negotiates vendor terms, designs training programs, establishes logging requirements, and reports to leadership on compliance and risk.

How This Relates to AI and Emerging Technology Governance

Generative AI in customer service is one application of a broader governance requirement. Organizations adopting AI tools across any function face similar questions: what data is submitted, what vendor commitments are required, what staff training is necessary, what logging is maintained, and who is accountable.

Effective AI governance establishes a framework that applies across use cases. Rather than addressing each AI tool in isolation, leadership defines principles and controls that apply organization-wide. The NIST Cybersecurity Framework and NIST Privacy Framework provide structures for integrating AI governance into enterprise risk management.

Customer service AI is often among the first use cases leadership encounters. The controls established here—vendor agreements, training, logging, clear accountability—serve as a model for AI adoption in other functions.

What Leadership Should Do Next

First, designate a single executive accountable for AI use in customer service. Confirm that this person has authority to approve tools, enforce policies, and coordinate across departments.

Second, inventory what generative AI tools customer service staff currently use or have access to. Determine whether those tools are authorized, whether vendor agreements address data handling, and whether staff have received training.

Third, define what personal information may be submitted to AI tools and what must be excluded or redacted. Document this in a policy that customer service staff can reference.

Fourth, review vendor agreements for approved AI tools. Verify that agreements address data use, retention, sharing, security, breach notification, and termination. If they do not, negotiate supplemental terms or suspend use until acceptable commitments are in place.

Fifth, implement training for customer service staff. Document who received training and when. Schedule periodic refresher training.

Sixth, establish logging for AI tool use. Verify that logs capture user identity, timestamps, and whether customer data was involved. Confirm that logs are reviewed and retained.

Seventh, report to the board or senior leadership on AI use in customer service, including what controls are in place, what gaps remain, and what timeline exists for closing those gaps.

If your organization lacks internal security or privacy leadership with capacity to own this work, a vCISO provides the executive oversight, technical judgment, and regulatory fluency required. Heights offers confidential consultations to chief executives, boards, and senior leaders navigating AI governance, privacy compliance, and emerging technology risk. A single conversation clarifies what must be done, who should own it, and how progress is measured. Contact Heights directly to schedule a consultation.