Privacy & data
Test the AI Without Handing It Your Customer List
You can test missing receipts, awkward dates and routing errors with invented records. Start there, and check where the tool stores what you give it.

The first warranty claim can be invented. Give the test a purchase date and a fault to assess against the warranty rule. Keep customer records out of the first trial and check where the test files and replies will be stored before you consider using sensitive data.
In a study presented at USENIX Security in 2021, Nicholas Carlini and colleagues confirmed that 604 unique samples among 1,800 selected GPT-2 outputs contained text reproduced word for word from its training data. Some included people’s contact details. The researchers deliberately selected promising candidates, which makes the count specific to their search.
A reply can contain text seen during training
GPT-2 was trained on material collected from the public web. The researchers compared candidate passages with their sources and confirmed the matches with help from the model’s creators. The paper also describes outputs that combined details from unrelated sources, creating false associations.
The US National Institute of Standards and Technology describes training-data disclosure as one privacy risk. A customer file uploaded during a warranty check can expose information through a different route from the model’s training data. Check whether the provider or your own systems retain prompts or uploads in logs separately from any use for training.
Invent the case instead of changing a customer’s name
Suppose a support team wants to see whether an AI assistant can sort warranty requests. Make up order references, purchase dates and fault descriptions. Include a case with a missing receipt and one that falls just outside the warranty window. Those examples can test routing without taking an actual customer’s address or account history into the trial. Build the examples from scratch so a distinctive combination of a customer’s details does not survive a name change.
List where prompts, uploads and replies go during the trial, including the systems that store or log them. Record who can access each location and confirm the rules for storage and training use with the person responsible for the tool. If a later test needs customer records, agree which fields it needs and who can approve access before collecting them.
Sources and further reading
- Carlini et al., Extracting Training Data from Large Language Models (USENIX Security, 2021)
- NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (July 2024)
Sources checked 7 November 2026. Edited and reviewed by AI Boss.