Abstract: This paper aims to provide differential privacy (DP) guarantees for Markov chain Monte Carlo (MCMC) algorithms. In a first part, we establish DP guarantees on samples output by MCMC algorithms as well as Monte Carlo estimators associated with these methods under assumptions on the convergence properties of the underlying Markov chain. In particular, our results highlight the critical condition of ensuring the target distribution is differentially private itself. In a second part, we specialise our analysis to the unadjusted Langevin algorithm and stochastic gradient Langevin dynamics and establish guarantees on their (Rényi) DP. To this end, we develop a novel methodology based on Girsanov's theorem combined with a perturbation trick to obtain bounds for an unbounded domain and in a non-convex setting. We establish: (i) uniform in $n$ privacy guarantees when the state of the chain after $n$ iterations is released, (ii) bounds on the privacy of the entire chain trajectory. These findings provide concrete guidelines for privacy-preserving MCMC.

Lay Summary:

Differential privacy is a framework which verifies that a statistical procedure is not too sensitive to individual components of the data. Specifically, it considers statistics that are randomised in such a way that changing a given entry in the data set only changes the (random) statistic a suitably small amount. This means one can prove mathematically that the (random) statistic does not give too much information about any one data point.In this paper we investigate the differential privacy of a class of widely used algorithms in statistics and optimisation. The class of algorithms we consider are often used for deriving additional information given a certain amount of prior information (sampling from Bayesian posteriors). We show firstly that the idealised theoretical privacy (the true posterior) and the implementable approximation (MCMC method) must broadly agree in their differential privacy if the numerical approximation is accurate. In the next part we show that certain implementable approximations (MCMC methods) can be differentially private under weaker (non-convex) technical assumptions than previously considered in the literature.