Generative AI Data Governance: Controlling Prompts, RAG Access, Training Data, and Logs

Generative AI gives company data new places to go and new ways to come back out. A prompt sends data to a vendor. A retrieval system copies documents into a vector index. A fine-tuned model can repeat its training examples. Logs keep a copy of all of it. And unlike a database, a model will follow instructions that arrive inside the data it reads.

This guide covers those data flows and the controls that work for each. It assumes you already have a governance program; for frameworks, risk tiers, and EU AI Act dates, see our AI governance guide. For GDPR and privacy law questions, see AI and SaaS data privacy, and for catalogs and classification in general, cloud data governance.

Where company data goes in a generative AI system

The OWASP Top 10 for LLM Applications 2025 maps closely to this table: prompt injection is number one, sensitive information disclosure number two, data and model poisoning number four, and vector and embedding weaknesses number eight.

Vendor terms: training, retention, and legal holds

The first control is contractual. The same vendor can treat your data very differently depending on the tier:

• OpenAI's enterprise privacy page says it does not train on business data from ChatGPT Enterprise, Team, Edu, or the API by default, and offers zero data retention for eligible API use.
• Anthropic's August 2025 consumer terms update lets Free, Pro, and Max users choose whether their chats train models, with retention extended to five years for those who opt in and 30 days otherwise. It does not apply to commercial terms: Claude for Work, the API, or access through Amazon Bedrock and Google Cloud Vertex AI.
• Microsoft says Microsoft 365 Copilot prompts, responses, and Graph data are not used to train foundation models, and that Copilot only surfaces content the user already has at least view permission for.

Retention promises can also change without your vendor choosing to. In New York Times v. OpenAI, a magistrate judge ordered OpenAI on May 13, 2025 to preserve output logs that would otherwise have been deleted, covering consumer ChatGPT and API customers without zero data retention. ChatGPT Enterprise and zero-retention API customers were excluded. OpenAI says the obligation ended on September 26, 2025, with some historical data still held. Business tiers with contractual retention terms were the ones that stayed out of it.

When you review an AI contract, check training use, retention period, zero-retention eligibility, subprocessors and hosting region, deletion at termination, breach notice timing, and whether the vendor indemnifies you against copyright claims over outputs.

Retrieval: the permission problem

Enterprise assistants and custom RAG systems answer questions by searching your documents. That makes every old oversharing mistake searchable. Suppose a spreadsheet of salaries was shared with "everyone in the company" five years ago. It stayed effectively hidden because nobody knew to look for it. An assistant that respects that permission will find it for anyone who asks about pay.

For off-the-shelf assistants such as Copilot, the fix is cleaning up permissions before rollout: find sites and folders shared with the whole organization, remove stale sharing links, and apply sensitivity labels to the files that matter. For custom RAG, the requirements are specific:

• Enforce access at query time. Store each chunk with the access list of its source document and filter the vector search by the user's identity and groups. Filtering only at indexing time leaves you with an index that ignores later permission changes.
• Sync revocations quickly, and measure the lag. If someone loses access to a folder at 9:00, the index should reflect it within minutes or hours, not at the next weekly rebuild.
• Delete everywhere. When a source document is deleted, remove its chunks, embeddings, and any cached answers built from it.
• Classify the index like the source. Embeddings are not anonymized data. The vec2text study (EMNLP 2023) reconstructed 92% of 32-token inputs exactly from their embeddings and recovered full names from a set of clinical notes.

Indirect prompt injection

Retrieved content can carry instructions. In June 2025, researchers at Aim Security disclosed EchoLeak (CVE-2025-32711, CVSS 9.3), a zero-click flaw in Microsoft 365 Copilot. An attacker sent an ordinary-looking email with hidden instructions. When the user later asked Copilot a question, retrieval pulled in that email, and the instructions made Copilot embed sensitive data from the user's context into a link that sent it to the attacker. The attack got past Microsoft's injection classifier and link redaction, according to the researchers' paper. Microsoft fixed it on the server side and reported no exploitation in the wild, then described its layered defenses in a July 2025 MSRC post.

No filter catches every injection, so the controls are about limiting what an injected instruction can do:

• Treat retrieved text, email, and web content as untrusted input, and keep external content in a separate retrieval source from internal documents where you can.
• Block or strip outbound links and images in responses that include sensitive data, since rendered links and images are a common way to send data out.
• Require a human confirmation before an assistant sends email, shares files, or calls external tools.
• Red-team with documents and emails written by outsiders, not only with prompts typed by users.

Training and fine-tuning data

Models memorize some of what they are trained on. Carlini and colleagues extracted hundreds of verbatim sequences from GPT-2 in 2021, including names, phone numbers, and email addresses. In 2023, Nasr and colleagues extracted gigabytes of training data from open models and used a "divergence" attack to make ChatGPT emit training data at 150 times its normal rate. Fine-tuning on a small, repetitive set of company records raises the risk, because repeated examples are the ones most likely to be memorized.

Deletion is the harder problem. Once data is in the weights, the dependable way to remove it is to retrain without it. That is a strong argument for keeping personal data and secrets out of fine-tuning sets altogether and putting frequently changing or deletable information in a retrieval index, where deleting the source document removes it. European regulators have said a model trained on personal data is not automatically anonymous; see EDPB Opinion 28/2024.

Differential privacy, with its limits

Differential privacy adds calibrated noise during training so the model's behavior barely changes whether or not any single unit of data was included. Google's VaultGemma, released in September 2025, is a 1-billion-parameter model trained from scratch this way, with a guarantee of epsilon 2.0 (delta 1.1e-10) per 1,024-token sequence. Google reported that its benchmark scores were roughly those of non-private models from about five years earlier, such as GPT-2 1.5B. That is the current price of the guarantee.

Two points matter when a vendor or team claims differential privacy. First, ask for epsilon and the privacy unit: a per-sequence guarantee does not protect a customer whose records appear in thousands of sequences as well as a per-user guarantee would. Second, check that the claim covers the step that touched your data, since many systems apply it only to fine-tuning. NIST's SP 800-226 (March 2025) explains how to evaluate these claims.

Poisoning

Anyone who can write to your training or retrieval sources can try to change the model's behavior. A 2025 study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that about 250 poisoned documents were enough to plant a backdoor in models from 600 million to 13 billion parameters, regardless of how much clean data they were trained on. The tested backdoor was narrow (it made the model output gibberish after a trigger word), and the authors note it is unknown whether the pattern holds for much larger models. The practical lesson: record the source of every fine-tuning example, restrict who can edit indexed wikis and knowledge bases, and be careful indexing inbound email or web content, which outsiders control.

Classification rules people can follow

Most companies already classify data. The gap is saying what each class may do with AI. An example policy:

A prompt gateway can enforce the pattern-matchable part: card numbers, national ID numbers, API keys, and customer IDs can be detected and either blocked or replaced with tokens before the prompt leaves, then restored in the response. It cannot recognize a paraphrased acquisition plan. Aggressive blocking also backfires. Verizon's 2026 Data Breach Investigations Report found 45% of employees using AI tools regularly, two-thirds of them through personal accounts, and IBM's 2026 breach study found that 92% of organizations with an AI-related breach had no access controls on their AI systems. Approved tools that work are the more effective control.

Logs are a new data store

Prompt and response logs are useful for debugging, abuse detection, and audits, and they contain everything users typed plus every chunk retrieval returned. Govern them like the most sensitive data they hold:

• Set a retention period and enforce it, including in observability tools and vendor dashboards.
• Restrict access to the few people who investigate incidents or quality problems.
• Store retrieved chunks as document references instead of full text where you can.
• Include AI logs in your legal hold process. Courts treat them as records, and a hold can override your deletion schedule, as OpenAI's did in 2025.

Our zero trust guide covers identity and access controls for these systems, and the MLOps guide covers model versioning and lineage.

Standards and rules to map against

• The NIST Generative AI Profile (AI 600-1, July 2024) lists data privacy, information security, and intellectual property among its generative AI risks, with suggested actions for each.
• ISO/IEC 42001:2023 is the certifiable AI management system standard.
• Under the EU AI Act, general-purpose model providers have had to publish a summary of their training content since August 2, 2025. The data governance duties for high-risk systems in Article 10 now start December 2, 2027 after the Digital Omnibus delay. The AI governance guide and the AI regulatory compliance guide have the full timeline.

Who does what

Data owners decide which sources may be indexed and approve fine-tuning sets. Security runs the prompt gateway, the injection red-team, and incident response for AI systems. Legal negotiates vendor terms and runs legal holds. The platform team keeps permission sync, deletion, and logging working. A small review group that meets on a schedule and can approve a new tool within days works better than a council that meets quarterly and becomes the reason people use personal accounts.

A first 90 days

1. List every AI tool and connector in use, including which data sources each one can read.
2. Move users onto business tiers and confirm training and retention terms in the contract.
3. Fix oversharing in the sources before turning on enterprise search or an assistant.
4. Build or configure permission-aware retrieval, then test it with accounts at different access levels.
5. Red-team for indirect prompt injection using emails and documents written from outside.
6. Set log retention and add AI logs to the legal hold procedure.
7. Write fine-tuning data rules: no restricted data, and a provenance record for every set.
8. Track a few numbers: share of AI use through approved tools, oversharing findings closed, blocked or tokenized prompts, and the time to remove a deleted document from every index.

This guide is for informational purposes only and does not constitute legal or security advice. Vendor terms, regulations, and vulnerabilities change; figures and terms are as of September 2026. Review your contracts and controls with your security, privacy, and legal teams.