Training-Data Extraction

Training-data extraction is an attack in which an adversary recovers actual examples from a model's training set using only query access to the model. Large language models memorize some of what they are trained on, and Carlini and colleagues showed in 2021 that an attacker can prompt a model so that it regenerates memorized strings verbatim, including names, phone numbers, addresses, and other sensitive text that appeared in the training corpus.

Memorization and extractability

In Carlini et al.'s definition, an example is extractable if there exists a prefix that, when used to condition generation, makes the model output that exact string. Their demonstration against GPT-2 recovered hundreds of memorized sequences, and follow-up work confirmed that larger models and frequently repeated data are more prone to memorization. Unlike membership inference, which only confirms whether a record was used, extraction reconstructs the content itself, making it the more severe confidentiality failure.

Why it matters for self-hosted models

Any model fine-tuned on private data, internal documents, customer records, or proprietary code, can leak that data through ordinary generation if it memorized rare or repeated entries. This is a direct privacy and secrets-management risk for operators who fine-tune local models on sensitive material.

Mitigations include deduplicating training data, applying differential privacy during training, scrubbing secrets before fine-tuning, and limiting how much raw generation an interface exposes. For related privacy attacks, see our entries on membership inference attacks and model inversion attacks.