Abstract: We propose the notion of empirical privacy variance and study it in the context of differentially private fine-tuning of language models. Specifically, we show that models calibrated to the same $(\varepsilon, \delta)$-DP guarantee using DP-SGD with different hyperparameter configurations can exhibit significant variations in empirical privacy, which we quantify through the lens of memorization. We investigate the generality of this phenomenon across multiple dimensions and discuss why it is surprising and relevant. Through regression analysis, we examine how individual and composite hyperparameters influence empirical privacy. The results reveal a no-free-lunch trade-off: existing practices of hyperparameter tuning in DP-SGD, which focus on optimizing utility under a fixed privacy budget, often come at the expense of empirical privacy. To address this, we propose refined heuristics for hyperparameter selection that explicitly account for empirical privacy, showing that they are both precise and practically useful. Finally, we take preliminary steps to understand empirical privacy variance. We propose two hypotheses, identify limitations in existing techniques like privacy auditing, and outline open questions for future research.

Lay Summary:

Training large language models (LLMs) while protecting the privacy of the data they learn from is a significant challenge. A popular technique called differential privacy (DP) offers strong theoretical guarantees, but we found a surprising issue: even when models are trained with the same level of theoretical privacy protection using a common method (DP-SGD), they can leak significantly different amounts of private information in practice. Our research introduces the concept of "empirical privacy variance" to measure this difference in practical privacy under the same theoretical privacy guarantee. We show that how you set the training parameters in DP-SGD has a big, and often overlooked, impact on practical privacy. Standard ways of picking these parameters focus on making the model more useful while adhering to a theoretical privacy budget; we show that this practice unfortunately makes the model "remember" more about the training data than necessary, undermining practical privacy. To address this, we propose new strategies for choosing training parameters that consider this practical privacy alongside model performance. These strategies are shown to be effective in producing models that are not only theoretically private but also offer better practical privacy. Our work highlights the gap between theoretical and practical privacy, and calls for a careful reflection of the prevalent usage of DP in LLMs and beyond.