Fine-tuning language models on sensitive text (medical notes, private messages, internal documents) risks the model memorizing and later regurgitating specifics from the training data — a well-documented failure mode for even small models. Differential privacy (DP), via DP-SGD, gives a formal, quantifiable guarantee against this by clipping per-example gradients and adding calibrated noise during training. In this project you will implement DP fine-tuning properly, then empirically characterize the privacy/utility tradeoff i.e. as the privacy budget is tightened, how much does language model utility degrade, and does DP training actually succeed in suppressing memorization of specific training examples, measured directly rather than assumed.
Model and data
- Model: A small-ish tranformer model e.g. GPT-2 small (124M params).
- Data: A few thousand text examples, large enough to get a meaningful language modeling signal, small enough that model can be trained quickly. Before training, you will insert a handful of synthetic “canary” sequences into the training set e.g. fake sentences containing a random-looking string like a fake phone number or ID. These will give us a direct, checkable memorization signal: after training, can the model be prompted to regurgitate the canary?
Outline workplan:
- Standard (non-private) fine-tuning baseline — fine-tune the model with ordinary SGD/Adam, no DP. This is the utility upper bound and also the “does memorization happen at all without protection” baseline (canary extraction should succeed here).
- DP-SGD fine-tuning — using Opacus (the standard PyTorch DP library) or an equivalent, implement fine-tuning with per-example gradient clipping and calibrated Gaussian noise addition.
- Privacy vs utility — train the same model/data at several ε values (e.g. ε = 1, 3, 8, and ∞ as the non-private baseline) and hold everything else fixed, so the only variable across runs is the privacy budget. Evaluate performance on held-out text for each ε, plus qualitative inspection of generated samples (does the text stay coherent, or degrade into noise as ε tightens). Also measure the wall-clock time for training since DP-SGD is likely slower than plain SGD.
- Memorization evaluation — for each trained model attempt to extract the canary sequences via prompting (feed the canary’s prefix, see if the model completes it correctly) and/or via exposure metrics (comparing the canary’s likelihood under the model against random alternative sequences).