Abstract: We propose a novel method for measuring the discrepancy between a set of samples and a desired posterior distribution for Bayesian inference. Classical methods for assessing sample quality like the effective sample size are not appropriate for scalable Bayesian sampling algorithms, such as stochastic gradient Langevin dynamics, that are asymptotically biased. Instead, the gold standard is to use the kernel Stein Discrepancy (KSD), which is itself not scalable given its quadratic cost in the number of samples. The KSD and its faster extensions also typically suffer from the curse-of-dimensionality and can require extensive tuning. To address these limitations, we develop the polynomial Stein discrepancy (PSD) and an associated goodness-of-fit test. While the new test is not fully convergence-determining, we prove that it detects differences in the first $r$ moments for Gaussian targets. We empirically show that the test has higher power than its competitors in several examples, and at a lower computational cost. Finally, we demonstrate that the PSD can assist practitioners to select hyper-parameters of Bayesian sampling algorithms more efficiently than competitors.

Lay Summary:

Modern challenges in statistics and machine learning involve the ability to process large amounts of data and complex mathematical models, which can be computationally intensive. To overcome this computational challenge, approximate models are often used in practice. However, it is difficult to assess whether samples generated from approximate models are consistent with the complex model of interest (i.e. whether they are fit-for-purpose).Traditional ways of checking or measuring the quality of simulated samples often miss important issues, especially when the approximate simulation process introduces some bias or error. Some newer techniques try to fix this, but they can be slow with large datasets, struggle with complex data, and often need a lot of fine-tuning to work properly. They can also miss key differences in basic features of the samples, like the average or the spread.To address these problems, we introduce a new, faster approach that focuses directly on checking whether the important summary features of the samples—like averages and variability—match what we expect. We show through examples that this method works well across a range of common statistical tasks.