Abstract: Sparse variational Gaussian processes (GPs) construct tractable posterior approximations to GP models. At the core of these methods is the assumption that the true posterior distribution over training function values ${\bf f}$ and inducing variables ${\bf u}$ is approximated by a variational distribution that incorporates the conditional GP prior $p({\bf f} | {\bf u})$ in its factorization. While this assumption is considered as fundamental, we show that for model training we can relax it through the use of a more general variational distribution $q({\bf f} | {\bf u} )$ that depends on $N$ extra parameters, where $N$ is the number of training examples. In GP regression, we can analytically optimize the evidence lower bound over the extra parameters and express a tractable collapsed bound that is tighter than the previous bound. The new bound is also amenable to stochastic optimization and its implementation requires minor modifications to existing sparse GP code. Further, we also describe extensions to non-Gaussian likelihoods. On several datasets we demonstrate that our method can reduce bias when learning the hyperparameters and can lead to better predictive performance.

Lay Summary:

Modeling uncertainty is one of the key challenges in Machine Learning. For regression and function approximation problems, Gaussian processes (GPs) provide a Bayesian nonparametric (i.e., memory-based) framework to estimate unknown functions by providing also uncertainty estimates. However, the complexity of these models scales cubically with the number of training examples, so for large datasets exact computations are prohibitive. In this work we elaborate on scalable GP methods that construct approximations based on smaller sets of special points called inducing points. More precisely, we improve a certain type of scalable GP method based on a posterior (variational) approximation of the model.