Data Leakage Through LLMs: A Board-Level Question

TL;DR

Two distinct leakage classes, requiring different controls:

Class	What it is	Mitigation
Inference-time	Data sent in a prompt is logged, retained, or used by the model provider	Contractual no-retention zones, BAA-equivalent commitments, on-prem or VPC deployment
Training-time	Data used to train the model is later reproduced in outputs to other users	Opt-out from training, closed-tenant deployments, careful fine-tuning data curation

Most “AI policies” address only the first class. The second class — model memorization and cross-tenant leakage — requires different controls and is rarely covered.

Most AI policies address only one of the two leakage classes. The two require different controls.

The board question on AI data leakage usually starts with “are we protected?” The honest answer is “from one class of leakage, mostly; from the other, probably not.” The two classes are inference-time leakage and training-time leakage, and they require different mitigations. Most enterprise AI policies cover the first because it’s the obvious one; the second is more subtle and, in some ways, more dangerous because it can be undetectable until it’s too late.

This piece is the two classes, the controls that work for each, and what to ask in your next AI vendor review.

Class 1: Inference-time leakage

What it is: data you send to a model in a prompt is logged, retained, or used in ways you don’t intend. The data flow is: user → your application → model API → logs/retention.

Common leakage paths:

Model provider stores prompts for “model improvement” — your data ends up in their system, accessible to their employees.
Model provider retains prompts for abuse-detection — even if not used for training, the data sits in their environment.
Your own logging captures prompts that include sensitive data, then those logs leak via a separate breach.

Mitigations that work:

Contractual no-retention agreements with the model provider (Anthropic, OpenAI, Google all offer enterprise agreements with zero-retention or short-retention for data).
VPC or on-prem deployment for highly sensitive workloads.
BAA (Business Associate Agreement) with the model provider for healthcare data.
Prompt-level PII filtering before the data leaves your environment.

What gets missed: relying on default consumer-tier API access for enterprise workloads. The default terms allow logging and retention; the enterprise terms don’t. Make sure your contract is the right one.

Class 2: Training-time leakage

What it is: data used to train the model is reproduced in outputs to other users. The data flow is: someone’s data → training set → model → outputs to anyone.

Common leakage paths:

Your data was scraped from the public web (your blog, customer reviews, public filings) and trained into the model.
Your data was used by your model provider for training under a previous, less restrictive contract.
A third-party fine-tune of an open model used your data without authorization, and the fine-tune leaks.

Mitigations that work:

Opt-out from training (specifically opt-out, in writing, in the contract).
Closed-tenant deployments where your data is fenced from any training process.
Careful curation of fine-tuning data — whatever you fine-tune on, the model can reproduce.
Periodic memorization testing (does the model repeat your proprietary content verbatim when prompted?).

What gets missed: assuming opt-out is the default. It usually isn’t, especially for older contracts. Check your existing terms and negotiate explicit opt-out at renewal.

The asymmetry

Inference-time leakage is observable. If your data shows up where it shouldn’t, you can usually trace it back. The mitigations are auditable.

Training-time leakage is largely unobservable. If your data is in someone else’s training set, you may never know — until it shows up in a competitor’s chatbot output six months later. The mitigations are preventive; once the leakage has happened, you can’t undo it.

This asymmetry means training-time leakage deserves more attention than it typically gets. The policy emphasis should be on prevention, not detection.

What to ask vendors

Five specific questions for any AI vendor you’re considering.

1. What’s your data retention policy on prompts and outputs? The answer should be “zero retention by default for enterprise customers” or “specific retention period with deletion guarantees.” Vague answers are red flags.

2. Do you use customer data for training? Default-no, with explicit written confirmation. If the answer is “yes by default, but you can opt out,” opt out in writing.

3. How is customer data isolated between tenants? For shared infrastructure deployments, the answer should describe specific technical controls. For VPC or dedicated deployments, the isolation is the deployment model.

4. What happens to our data if we terminate the contract? Specific deletion timelines and certification of deletion.

5. Have you had any data-related incidents in the last 24 months? Forces disclosure. A “no” with details is more credible than a “no” without.

What to do this quarter

Audit your existing AI vendor contracts. For each, classify by retention posture and training-data posture.
Negotiate the gaps at renewal. Inference-time no-retention should be standard; training-time opt-out should be standard.
Add the five questions to your AI vendor evaluation template.
Run a memorization test. Pick a proprietary document; query the major LLMs to see if any reproduce it. You’re testing for past exposure, but the test catches the policy gaps.

FAQ

Can the same data be subject to both leakage classes? Yes. Data sent in a prompt is subject to inference-time leakage; if the provider also uses it for training, it’s subject to training-time leakage as well. The mitigations are independent — controlling one doesn’t control the other.

Are open-weight models more or less risky on data leakage? Mixed. Open-weight models you host yourself eliminate inference-time leakage to a third party. Training-time leakage is bounded to whatever data was in the training set you can’t change. Many enterprise teams move sensitive workloads to open-weight self-hosted for this reason.

Is the model provider liable if our data leaks? Depends on the contract. Most provider contracts have liability caps that are well below the cost of a meaningful data breach. Negotiate higher caps for sensitive-data workloads, or accept the residual risk explicitly.

How do we know if our data is in a model’s training set? You can test (memorization probing, attribution queries), but you can’t be certain. The mitigations are forward-looking: don’t put sensitive data in places where it could be scraped or shared.

Do data residency requirements (GDPR, etc.) extend to AI prompts? Yes. A prompt containing personal data is a transfer of personal data, subject to all the usual cross-border rules. Verify your model provider’s data-flow architecture before sending EU person data.

Working with JAIN on AI data-leakage controls? We help executive teams classify their data, negotiate the right vendor terms, and close the policy gap on training-time leakage. Book a 30-minute call.

Related reading: