A Handbook for Sparse Variational Gaussian Processes

In the sparse variational Gaussian process (SVGP) framework (Titsias, 2009)¹, one augments the joint distribution $p(\mathbf{y},\mathbf{f})$ with auxiliary variables $\mathbf{u}$ so that the joint becomes $$p(\mathbf{y},\mathbf{f},\mathbf{u})=p(\mathbf{y}|\mathbf{f})p(\mathbf{f},\mathbf{u}).$$ The vector $\mathbf{u}={\left[\begin{array}{c}u({\mathbf{z}}_1)\cdots u({\mathbf{z}}_M)\end{array}\right]}^{\mathrm{\top }}\in {\mathbb{R}}^M$ consists of inducing variables, the latent function values corresponding to the inducing input locations contained in the matrix $\mathbf{Z}={\left[\begin{array}{c}{\mathbf{z}}_1\cdots {\mathbf{z}}_M\end{array}\right]}^{\mathrm{\top }}\in {\mathbb{R}}^{M\times D}$.

Prior

The joint distribution of the latent function values $\mathbf{f}$, and the inducing variables $\mathbf{u}$ according to the prior is $$p(\mathbf{f},\mathbf{u})=\mathcal{N}(\left[\begin{array}{c}\mathbf{f}\\ \mathbf{u}\end{array}\right];\left[\begin{array}{c}\mathbf{0}\\ \mathbf{0}\end{array}\right],\left[\begin{array}{cc}{\mathbf{K}}_{\mathbf{ff}}& {\mathbf{K}}_{\mathbf{uf}}^{\mathrm{\top }}\\ {\mathbf{K}}_{\mathbf{uf}}& {\mathbf{K}}_{\mathbf{uu}}\end{array}\right]).$$ If we let the joint prior factorize as $$p(\mathbf{f},\mathbf{u})=p(\mathbf{f}|\mathbf{u})p(\mathbf{u}),$$ we can apply the rules of Gaussian conditioning to derive the marginal prior $p(\mathbf{u})$ and conditional prior $p(\mathbf{f}|\mathbf{u})$.

Marginal prior over inducing variables

The marginal prior over inducing variables is simply given by $$p(\mathbf{u})=\mathcal{N}(\mathbf{u}|\mathbf{0},{\mathbf{K}}_{\mathbf{uu}}).$$

Conditional prior

First, let us define the vector-valued function ${\mathit{\psi }}_{\mathbf{u}}:{\mathbb{R}}^D\to {\mathbb{R}}^M$ as $${\mathit{\psi }}_{\mathbf{u}}(\mathbf{x})\triangleq {\mathbf{K}}_{\mathbf{uu}}^{-1}{\mathbf{k}}_{\mathbf{u}}(\mathbf{x}),$$ where ${\mathbf{k}}_{\mathbf{u}}(\mathbf{x})=k_{\theta }(\mathbf{Z},\mathbf{x})$ denotes the vector of covariances between $\mathbf{x}$ and the inducing inputs $\mathbf{Z}$. Further, let $\mathbf{\Psi }\in {\mathbb{R}}^{M\times N}$ be the matrix containing values of function $\psi$ applied row-wise to the matrix of inputs $\mathbf{X}={\left[\begin{array}{c}{\mathbf{x}}_1\cdots {\mathbf{x}}_N\end{array}\right]}^{\mathrm{\top }}\in {\mathbb{R}}^{N\times D}$, $$\mathbf{\Psi }\triangleq \left[\begin{array}{c}\psi ({\mathbf{x}}_1)\cdots \psi ({\mathbf{x}}_N)\end{array}\right]={\mathbf{K}}_{\mathbf{uu}}^{-1}{\mathbf{K}}_{\mathbf{uf}}.$$ Then, we can condition the joint prior distribution on the inducing variables to give $$p(\mathbf{f}|\mathbf{u})=\mathcal{N}(\mathbf{f}|\mathbf{m},\mathbf{S}),$$ where the mean vector and covariance matrix are $$\mathbf{m}={\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{u},\text{and}\mathbf{S}={\mathbf{K}}_{\mathbf{ff}}-{\mathbf{\Psi }}^{\mathrm{\top }}{\mathbf{K}}_{\mathbf{uu}}\mathbf{\Psi }.$$

Before moving on, we briefly highlight the important quantity, $${\mathbf{Q}}_{\mathbf{ff}}\triangleq {\mathbf{\Psi }}^{\mathrm{\top }}{\mathbf{K}}_{\mathbf{uu}}\mathbf{\Psi },$$ which is sometimes referred to as the Nyström approximation of ${\mathbf{K}}_{\mathbf{ff}}$. It can be written as $${\mathbf{Q}}_{\mathbf{ff}}={\mathbf{K}}_{\mathbf{fu}}{\mathbf{K}}_{\mathbf{\text{uu}}}^{-1}{\mathbf{K}}_{\mathbf{uf}}.$$

Variational Distribution

We specify a joint variational distribution $q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})$ which factorizes as $$q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})\triangleq p(\mathbf{f}|\mathbf{u})q_{\mathit{\varphi }}(\mathbf{u}).$$ For convenience, let us specify a variational distribution that is also Gaussian, $$q_{\mathit{\varphi }}(\mathbf{u})\triangleq \mathcal{N}(\mathbf{u}|\mathbf{b},\mathbf{W}{\mathbf{W}}^{\mathrm{\top }}),$$ with variational parameters $\mathit{\varphi }=\{\mathbf{W},\mathbf{b}\}$. To obtain the corresponding marginal variational distribution over $\mathbf{f}$, we marginalize out the inducing variables $\mathbf{u}$, leading to $$q_{\mathit{\varphi }}(\mathbf{f})=\int q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})\mathrm{d}\mathbf{u}=\mathcal{N}(\mathbf{f}|\mathit{\mu },\mathbf{\Sigma }),$$ where $$\mathit{\mu }={\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{b},\text{and}\mathbf{\Sigma }={\mathbf{K}}_{\mathbf{ff}}-{\mathbf{\Psi }}^{\mathrm{\top }}({\mathbf{K}}_{\mathbf{uu}}-\mathbf{W}{\mathbf{W}}^{\mathrm{\top }})\mathbf{\Psi }.$$

Whitened parameterization

Whitening is a powerful trick for stabilizing the learning of variational parameters that works by reducing correlations in the variational distribution (Murray & Adams, 2010; Hensman et al, 2015)^{2 3}. Let $\mathbf{L}$ be the Cholesky factor of ${\mathbf{K}}_{\mathbf{uu}}$, i.e. the lower triangular matrix such that $\mathbf{L}{\mathbf{L}}^{\mathrm{\top }}={\mathbf{K}}_{\mathbf{uu}}$. Then, the whitened variational parameters are given by $$\mathbf{W}\triangleq \mathbf{L}{\mathbf{W}}^{\prime },\text{and}\mathbf{b}\triangleq \mathbf{L}{\mathbf{b}}^{\prime },$$ with free parameters $\{{\mathbf{W}}^{\prime },{\mathbf{b}}^{\prime }\}$. This leads to mean and covariance $$\mathit{\mu }={\mathbf{\Lambda }}^{\mathrm{\top }}{\mathbf{b}}^{\prime },\text{and}\mathbf{\Sigma }={\mathbf{K}}_{\mathbf{ff}}-{\mathbf{\Lambda }}^{\mathrm{\top }}({\mathbf{I}}_M-{\mathbf{W}}^{\prime }{{\mathbf{W}}^{\prime }}^{\mathrm{\top }})\mathbf{\Lambda },$$ where $$\mathbf{\Lambda }\triangleq {\mathbf{L}}^{\mathrm{\top }}\mathbf{\Psi }={\mathbf{L}}^{-1}{\mathbf{K}}_{\mathbf{uf}}.$$ Refer to Appendix I for derivations.

For an efficient and numerically stable way to compute and evaluate the variational distribution $q_{\mathit{\varphi }}(\mathbf{f})$ at an arbitrary set of inputs, see Appendix II.

Inference

Preliminaries

We seek to approximate the exact posterior $p(\mathbf{f},\mathbf{u}\mid \mathbf{y})$ by an variational distribution $q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})$. To this end, we minimize the Kullback-Leibler (KL) divergence between $q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})$ and $p(\mathbf{f},\mathbf{u}\mid \mathbf{y})$, which is given by $$\begin{array}{rl}\mathrm{KL}[q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})\mid \mid p(\mathbf{f},\mathbf{u}\mid \mathbf{y})]& ={\mathbb{E}}_{q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})}[\log \frac{q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})}{p(\mathbf{f},\mathbf{u}\mid \mathbf{y})}]\\ & =\log p(\mathbf{y})+{\mathbb{E}}_{q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})}[\log \frac{q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})}{p(\mathbf{f},\mathbf{u},\mathbf{y})}]\\ & =\log p(\mathbf{y})-\mathrm{ELBO}(\mathit{\varphi },\mathbf{Z}),\end{array}$$ where we’ve defined the evidence lower bound (ELBO) as $$\mathrm{ELBO}(\mathit{\varphi },\mathbf{Z})\triangleq {\mathbb{E}}_{q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})}[\log \frac{p(\mathbf{f},\mathbf{u},\mathbf{y})}{q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})}].$$ Notice that minimizing the KL divergence above is equivalent to maximizing the ELBO. Furthermore, the ELBO is a lower bound on the log marginal likelihood, since $$\log p(\mathbf{y})=\mathrm{ELBO}(\mathit{\varphi },\mathbf{Z})+\mathrm{KL}[q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})\mid \mid p(\mathbf{f},\mathbf{u}\mid \mathbf{y})],$$ and the KL divergence is nonnegative. Therefore, we have $\log p(\mathbf{y})\ge \mathrm{ELBO}(\mathit{\varphi },\mathbf{Z})$ with equality at $\mathrm{KL}[q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})\mid \mid p(\mathbf{f},\mathbf{u}\mid \mathbf{y})]=0\iff q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})=p(\mathbf{f},\mathbf{u}\mid \mathbf{y})$.

Let us now focus our attention on the ELBO, which can be written as $$\begin{array}{rl}\mathrm{ELBO}(\mathit{\varphi },\mathbf{Z})& =\iint \log \frac{p(\mathbf{f},\mathbf{u},\mathbf{y})}{q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})}{q}_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})\mathrm{d}\mathbf{f}\mathrm{d}\mathbf{u}\\ & =\iint \log \frac{p(\mathbf{y}|\mathbf{f})\bcancel{p(\mathbf{f}|\mathbf{u})}p(\mathbf{u})}{\bcancel{p(\mathbf{f}|\mathbf{u})}{q}_{\mathit{\varphi }}(\mathbf{u})}{q}_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})\mathrm{d}\mathbf{f}\mathrm{d}\mathbf{u}\\ & =\int \log \frac{\mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})}{q_{\mathit{\varphi }}(\mathbf{u})}{q}_{\mathit{\varphi }}(\mathbf{u})\mathrm{d}\mathbf{u},\end{array}$$ where we have made use of the previous definition $q_{\mathit{\varphi }}(\mathbf{f},\mathbf{u})=p(\mathbf{f}|\mathbf{u})q_{\mathit{\varphi }}(\mathbf{u})$ and also introduced the definition $$\mathrm{\Phi }(\mathbf{y},\mathbf{u})\triangleq \exp (\int \log p(\mathbf{y}|\mathbf{f})p(\mathbf{f}|\mathbf{u})\mathrm{d}\mathbf{f}).$$ It is straightforward to verify that the optimal variational distribution, that is, the distribution $q_{{\mathit{\varphi }}^{\star }}(\mathbf{u})$ at which the ELBO is maximized, satisfies $$q_{{\mathit{\varphi }}^{\star }}(\mathbf{u})\propto \mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u}).$$ Refer to Appendix III for details. Specifically, after normalization, we have $$q_{{\mathit{\varphi }}^{\star }}(\mathbf{u})=\frac{\mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})}{\mathcal{Z}},$$ where $\mathcal{Z}\triangleq \int \mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})\mathrm{d}\mathbf{u}$. Plugging this back into the ELBO, we get $$\mathrm{ELBO}({\mathit{\varphi }}^{\star },\mathbf{Z})=\int \log \left(\bcancel{\mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})}\frac{\mathcal{Z}}{\bcancel{\mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})}}\right)q_{\mathit{\varphi }}(\mathbf{u})\mathrm{d}\mathbf{u}=\log \mathcal{Z}.$$

Gaussian Likelihoods – Sparse Gaussian Process Regression (SGPR)

Let us assume we have a Gaussian likelihood of the form $$p(\mathbf{y}|\mathbf{f})=\mathcal{N}(\mathbf{y}|\mathbf{f},{\beta }^{-1}\mathbf{I}).$$ Then it is straightforward to show that $$\log \mathrm{\Phi }(\mathbf{y},\mathbf{u})=\log \mathcal{N}(\mathbf{y}|\mathbf{m},{\beta }^{-1}\mathbf{I})-\frac{\beta }{2}\mathrm{tr}(\mathbf{S}),$$ where $\mathbf{m}$ and $\mathbf{S}$ are defined as before, i.e. $\mathbf{m}={\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{u}$ and $\mathbf{S}={\mathbf{K}}_{\mathbf{\text{ff}}}-{\mathbf{\Psi }}^{\mathrm{\top }}{\mathbf{K}}_{\mathbf{\text{uu}}}\mathbf{\Psi }$. Refer to Appendix IV for derivations.

Now, there are a few key objects of interest. First, the optimal variational distribution $q_{{\mathit{\varphi }}^{\star }}(\mathbf{u})$, which is required to compute the predictive distribution $q_{{\mathit{\varphi }}^{\star }}(\mathbf{f})=\int p(\mathbf{f}|\mathbf{u})q_{{\mathit{\varphi }}^{\star }}(\mathbf{u})\mathrm{d}\mathbf{u}$, but which may also be of independent interest. Second, the ELBO, the objective with respect to which the inducing input locations $\mathbf{Z}$ are optimized.

The optimal variational distribution is given by $$q_{{\mathit{\varphi }}^{\star }}(\mathbf{u})=\mathcal{N}(\mathbf{u}\mid \beta {\mathbf{K}}_{\mathbf{uu}}{\mathbf{M}}^{-1}{\mathbf{K}}_{\mathbf{uf}}\mathbf{y},{\mathbf{K}}_{\mathbf{uu}}{\mathbf{M}}^{-1}{\mathbf{K}}_{\mathbf{uu}}),$$ where $$\mathbf{M}\triangleq {\mathbf{K}}_{\mathbf{uu}}+\beta {\mathbf{K}}_{\mathbf{uf}}{\mathbf{K}}_{\mathbf{fu}}.$$ This can be verified by reducing the product of two exponential-quadratic functions in $\mathrm{\Phi }(\mathbf{y},\mathbf{u})$ and $p(\mathbf{u})$ into a single exponential-quadratic function up to a constant factor, an operation also known as “completing the square”. Refer to Appendix V for complete derivations.

This leads to the predictive distribution $$\begin{array}{rl}{q}_{{\mathit{\varphi }}^{\star }}(\mathbf{f})& =\mathcal{N}(\beta {\mathbf{\Psi }}^{\mathrm{\top }}{\mathbf{K}}_{\mathbf{uu}}{\mathbf{M}}^{-1}{\mathbf{K}}_{\mathbf{uu}}\mathbf{\Psi }\mathbf{y},{\mathbf{K}}_{\mathbf{ff}}-{\mathbf{\Psi }}^{\mathrm{\top }}({\mathbf{K}}_{\mathbf{uu}}-{\mathbf{K}}_{\mathbf{uu}}{\mathbf{M}}^{-1}{\mathbf{K}}_{\mathbf{uu}})\mathbf{\Psi })\\ & =\mathcal{N}(\beta {\mathbf{K}}_{\mathbf{fu}}{\mathbf{M}}^{-1}{\mathbf{K}}_{\mathbf{uf}}\mathbf{y},{\mathbf{K}}_{\mathbf{ff}}-{\mathbf{K}}_{\mathbf{fu}}({\mathbf{K}}_{\mathbf{uu}}^{-1}-{\mathbf{M}}^{-1}){\mathbf{K}}_{\mathbf{uf}}).\end{array}$$

The ELBO is given by $$\mathrm{ELBO}({\mathit{\varphi }}^{\star },\mathbf{Z})=\log \mathcal{Z}=\log \mathcal{N}(\mathbf{0},{\mathbf{Q}}_{\mathbf{ff}}+{\beta }^{-1}\mathbf{I})-\frac{\beta }{2}\mathrm{tr}(\mathbf{S}).$$ This can be verified by applying simple rules for marginalizing Gaussians. Again, refer to Appendix VI for complete derivations. Refer to Appendix VII for a numerically efficient and robust method for computing these quantities.

Non-Gaussian Likelihoods

Recall from earlier that the ELBO is written as $$\begin{array}{rl}\mathrm{ELBO}(\mathit{\varphi },\mathbf{Z})& =\int \log \left(\frac{\mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})}{q_{\mathit{\varphi }}(\mathbf{u})}\right)q_{\mathit{\varphi }}(\mathbf{u})\mathrm{d}\mathbf{u}\\ & =\int (\log \mathrm{\Phi }(\mathbf{y},\mathbf{u})+\log \frac{p(\mathbf{u})}{q_{\mathit{\varphi }}(\mathbf{u})})q_{\mathit{\varphi }}(\mathbf{u})\mathrm{d}\mathbf{u}\\ & =\mathrm{ELL}(\mathit{\varphi },\mathbf{Z})-\mathrm{KL}[q_{\mathit{\varphi }}(\mathbf{u})\Vert p(\mathbf{u})],\end{array}$$ where we define $\mathrm{ELL}(\mathit{\varphi },\mathbf{Z})$, the expected log-likelihood (ELL), as $$\mathrm{ELL}(\mathit{\varphi },\mathbf{Z})\triangleq {\mathbb{E}}_{q_{\mathit{\varphi }}(\mathbf{u})}[\log \mathrm{\Phi }(\mathbf{y},\mathbf{u})].$$ This constitutes the first term in the ELBO, and can be written as $$\begin{array}{rl}\mathrm{ELL}(\mathit{\varphi },\mathbf{Z})& =\int \log \mathrm{\Phi }(\mathbf{y},\mathbf{u})q_{\mathit{\varphi }}(\mathbf{u})\mathrm{d}\mathbf{u}\\ & =\int (\int \log p(\mathbf{y}|\mathbf{f})p(\mathbf{f}|\mathbf{u})\mathrm{d}\mathbf{f})q_{\mathit{\varphi }}(\mathbf{u})\mathrm{d}\mathbf{u}\\ & =\int \log p(\mathbf{y}|\mathbf{f})(\int p(\mathbf{f}|\mathbf{u})q_{\mathit{\varphi }}(\mathbf{u})\mathrm{d}\mathbf{u})\mathrm{d}\mathbf{f}\\ & =\int \log p(\mathbf{y}|\mathbf{f})q(\mathbf{f})\mathrm{d}\mathbf{f}\\ & ={\mathbb{E}}_{q(\mathbf{f})}[\log p(\mathbf{y}|\mathbf{f})].\end{array}$$ While this integral is analytically intractable in general, we can nonetheless approximate it efficiently using numerical integration techniques such as Monte Carlo (MC) estimation or quadrature rules. In particular, because $q(\mathbf{f})$ is Gaussian, we can utilize simple yet effective rules such as Gauss-Hermite quadrature.

Now, the second term in the ELBO is the KL divergence between $q_{\mathit{\varphi }}(\mathbf{u})$ and $p(\mathbf{u})$, which are both multivariate Gaussians, $$\mathrm{KL}[q_{\mathit{\varphi }}(\mathbf{u})\Vert p(\mathbf{u})]=\mathrm{KL}[\mathcal{N}(\mathbf{b},\mathbf{W}{\mathbf{W}}^{\mathrm{\top }})||\mathcal{N}(\mathbf{0},{\mathbf{K}}_{\mathbf{uu}})],$$ and has a closed-form expression. In the case of the whitened parameterization, it can be simplified as $$\begin{array}{rl}\mathrm{KL}[q_{\mathit{\varphi }}(\mathbf{u})\Vert p(\mathbf{u})]& =\mathrm{KL}[\mathcal{N}({\mathbf{b}}^{\prime },{\mathbf{W}}^{\prime }{{\mathbf{W}}^{\prime }}^{\mathrm{\top }})||\mathcal{N}(\mathbf{0},{\mathbf{K}}_{\mathbf{uu}})]\\ & =\mathrm{KL}[\mathcal{N}(\mathbf{b},\mathbf{W}{\mathbf{W}}^{\mathrm{\top }})||\mathcal{N}(\mathbf{0},\mathbf{I})].\end{array}$$ This comes from the fact that $$\mathrm{KL}[\mathcal{N}(\mathbf{A}{\mathit{\mu }}_0,\mathbf{A}{\mathbf{\Sigma }}_0{\mathbf{A}}^{\mathrm{\top }})||\mathcal{N}(\mathbf{A}{\mathit{\mu }}_1,\mathbf{A}{\mathbf{\Sigma }}_1{\mathbf{A}}^{\mathrm{\top }})]=\mathrm{KL}[\mathcal{N}({\mathit{\mu }}_0,{\mathbf{\Sigma }}_0)||\mathcal{N}({\mathit{\mu }}_1,{\mathbf{\Sigma }}_1)]$$ where we set ${\mathit{\mu }}_0=\mathbf{b},{\mathbf{\Sigma }}_0=\mathbf{W}{\mathbf{W}}^{\mathrm{\top }},{\mathit{\mu }}_1=\mathbf{0},{\mathbf{\Sigma }}_1=\mathbf{I}$ and $\mathbf{A}=\mathbf{L}$ where $\mathbf{L}$ is the Cholesky factor of ${\mathbf{K}}_{\mathbf{uu}}$, i.e. the lower triangular matrix such that $\mathbf{L}{\mathbf{L}}^{\mathrm{\top }}={\mathbf{K}}_{\mathbf{uu}}$.

Large-Scale Data with Stochastic Optimization

Links and Further Readings

• Papers:
  • Forerunners: Deterministic Training Conditional (DTC; Csató & Opper, 2002⁴; Seeger, 2003⁵); Fully Independent Training Conditional (FITC; Snelson & Ghahramani, 2005⁶; Quinonero-Candela & Rasmussen, 2005⁷)
  • Inter-domain Gaussian processes: Lázaro-Gredilla & Figueiras-Vidal, 2009⁸
  • Deep Gaussian processes: Damianou & Lawrence, 2013⁹, Salimbeni et al, 2017¹⁰
  • Non-Gaussian likelihoods: Hensman et al, 2013¹¹; Dezfouli & Bonilla, 2015¹²
  • Unifying inducing-/pseudo-point approximations: Bui et al, 2017¹³
  • Orthogonal decompositions: Salimbeni et al, 2018¹⁴; Shi et al, 2020¹⁵
  • Convergence analysis: Burt et al, 2019¹⁶
  • Efficient sampling: Wilson et al, 2020¹⁷
• Technical Reports:
  • Variational Model Selection for Sparse Gaussian Process Regression by M. Titsias
• Notes:
  • On the paper: Variational Learning of Inducing Variables in Sparse Gaussian Processes (Titsias, 2009) by T. Bui and R. Turner
• Blog posts:
  • Sparse GPs: approximate the posterior, not the model by J. Hensman

Cite as:

@article{tiao2020svgp,
  title   = "{A} {H}andbook for {S}parse {V}ariational {G}aussian {P}rocesses",
  author  = "Tiao, Louis C",
  journal = "tiao.io",
  year    = "2020",
  url     = "https://tiao.io/post/sparse-variational-gaussian-processes/"
}

To receive updates on more posts like this, follow me on Twitter and GitHub!

Appendix

I

Whitened parameterization

Recall the definition $\mathbf{\Lambda }\triangleq {\mathbf{L}}^{\mathrm{\top }}\mathbf{\Psi }$. Then, the mean simplifies to $$\mathit{\mu }={\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{b}={\mathbf{\Psi }}^{\mathrm{\top }}(\mathbf{L}{\mathbf{b}}^{\prime })=({\mathbf{L}}^{\mathrm{\top }}\mathbf{\Psi }{)}^{\mathrm{\top }}{\mathbf{b}}^{\prime }={\mathbf{\Lambda }}^{\mathrm{\top }}{\mathbf{b}}^{\prime }.$$ Similarly, the covariance simplifies to $$\begin{array}{rl}\mathbf{\Sigma }& ={\mathbf{K}}_{\mathbf{ff}}-{\mathbf{\Psi }}^{\mathrm{\top }}({\mathbf{K}}_{\mathbf{uu}}-\mathbf{W}{\mathbf{W}}^{\mathrm{\top }})\mathbf{\Psi }\\ & ={\mathbf{K}}_{\mathbf{ff}}-{\mathbf{\Psi }}^{\mathrm{\top }}(\mathbf{L}{\mathbf{L}}^{\mathrm{\top }}-\mathbf{L}({\mathbf{W}}^{\prime }{{\mathbf{W}}^{\prime }}^{\mathrm{\top }}){\mathbf{L}}^{\mathrm{\top }})\mathbf{\Psi }\\ & ={\mathbf{K}}_{\mathbf{ff}}-({\mathbf{L}}^{\mathrm{\top }}\mathbf{\Psi }{)}^{\mathrm{\top }}({\mathbf{I}}_M-{\mathbf{W}}^{\prime }{{\mathbf{W}}^{\prime }}^{\mathrm{\top }})({\mathbf{L}}^{\mathrm{\top }}\mathbf{\Psi })\\ & ={\mathbf{K}}_{\mathbf{ff}}-{\mathbf{\Lambda }}^{\mathrm{\top }}({\mathbf{I}}_M-{\mathbf{W}}^{\prime }{{\mathbf{W}}^{\prime }}^{\mathrm{\top }})\mathbf{\Lambda }.\end{array}$$

II

SVGP Implementation Details

Single input index point

Here is an efficient and numerically stable way to compute $q_{\mathit{\varphi }}(f(\mathbf{x}))$ for an input $\mathbf{x}$. We take the following steps:

1. Cholesky decomposition: $\mathbf{L}\triangleq \mathrm{cholesky}({\mathbf{K}}_{\mathbf{\text{uu}}})$

   Note: $\mathcal{O}(M^3)$ complexity.

2. Solve system of linear equations: $\mathit{\lambda }(\mathbf{x})\triangleq \mathbf{L}\mathrm{\setminus }{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})$

   Note: $\mathcal{O}(M^2)$ complexity since $\mathbf{L}$ is lower triangular; $\mathit{\beta }=\mathbf{A}\mathrm{\setminus }\mathbf{x}$ denotes the vector $\mathit{\beta }$ such that $\mathbf{A}\mathit{\beta }=\mathbf{x}\iff \mathit{\beta }={\mathbf{A}}^{-1}\mathbf{x}$. Hence, $\mathit{\lambda }(\mathbf{x})={\mathbf{L}}^{-1}{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})$.

3. $s(\mathbf{x},\mathbf{x})\triangleq k_{\theta }(\mathbf{x},\mathbf{x})-{\mathit{\lambda }}^{\mathrm{\top }}(\mathbf{x})\mathit{\lambda }(\mathbf{x})$

   Note: $${\mathit{\lambda }}^{\mathrm{\top }}(\mathbf{x})\mathit{\lambda }(\mathbf{x})={\mathbf{k}}_{\mathbf{u}}^{\mathrm{\top }}(\mathbf{x}){\mathbf{L}}^{-\mathrm{\top }}{\mathbf{L}}^{-1}{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})={\mathbf{k}}_{\mathbf{u}}^{\mathrm{\top }}(\mathbf{x}){\mathbf{K}}_{\mathbf{uu}}^{-1}{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})={\mathit{\psi }}_{\mathbf{u}}^{\mathrm{\top }}(\mathbf{x}){\mathbf{K}}_{\mathbf{uu}}{\mathit{\psi }}_{\mathbf{u}}(\mathbf{x}).$$

4. For whitened parameterization:

   1. $\mu \triangleq {\mathit{\lambda }}^{\mathrm{\top }}(\mathbf{x}){\mathbf{b}}^{\prime }$

   2. ${\mathbf{v}}^{\mathrm{\top }}(\mathbf{x})\triangleq {\mathit{\lambda }}^{\mathrm{\top }}(\mathbf{x}){\mathbf{W}}^{\prime }$

      Note: ${\mathbf{v}}^{\mathrm{\top }}(\mathbf{x})\mathbf{v}(\mathbf{x})={\mathbf{k}}_{\mathbf{u}}^{\mathrm{\top }}(\mathbf{x}){\mathbf{L}}^{-\mathrm{\top }}({\mathbf{W}}^{\prime }{{\mathbf{W}}^{\prime }}^{\mathrm{\top }}){\mathbf{L}}^{-1}{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})$

   otherwise:

   1. Solve system of linear equations: ${\mathit{\psi }}_{\mathbf{u}}(\mathbf{x})\triangleq {\mathbf{L}}^{\mathrm{\top }}\mathrm{\setminus }\mathit{\lambda }(\mathbf{x})$

      Note: $\mathcal{O}(M^2)$ complexity since ${\mathbf{L}}^{\mathrm{\top }}$ is upper triangular. Further, $${\mathit{\psi }}_{\mathbf{u}}(\mathbf{x})={\mathbf{L}}^{-\mathrm{\top }}\mathit{\lambda }(\mathbf{x})={\mathbf{L}}^{-\mathrm{\top }}{\mathbf{L}}^{-1}{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})={\mathbf{K}}_{\mathbf{uu}}^{-1}{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})$$ and $${\mathit{\psi }}_{\mathbf{u}}^{\mathrm{\top }}(\mathbf{x})={\mathbf{k}}_{\mathbf{u}}^{\mathrm{\top }}(\mathbf{x}){\mathbf{K}}_{\mathbf{uu}}^{-\mathrm{\top }}={\mathbf{k}}_{\mathbf{u}}^{\mathrm{\top }}(\mathbf{x}){\mathbf{K}}_{\mathbf{uu}}^{-1}$$ since ${\mathbf{K}}_{\mathbf{uu}}$ is symmetric and nonsingular.

   2. $\mu (\mathbf{x})\triangleq {\mathit{\psi }}_{\mathbf{u}}^{\mathrm{\top }}(\mathbf{x})\mathbf{b}$

   3. ${\mathbf{v}}^{\mathrm{\top }}(\mathbf{x})\triangleq {\mathit{\psi }}_{\mathbf{u}}^{\mathrm{\top }}(\mathbf{x})\mathbf{W}$

      Note: ${\mathbf{v}}^{\mathrm{\top }}(\mathbf{x})\mathbf{v}(\mathbf{x})={\mathbf{k}}_{\mathbf{u}}^{\mathrm{\top }}(\mathbf{x}){\mathbf{K}}_{\mathbf{uu}}^{-1}(\mathbf{W}{\mathbf{W}}^{\mathrm{\top }}){\mathbf{K}}_{\mathbf{uu}}^{-1}{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})$

5. ${\sigma }^2(\mathbf{x})\triangleq s(\mathbf{x},\mathbf{x})+{\mathbf{v}}^{\mathrm{\top }}(\mathbf{x})\mathbf{v}(\mathbf{x})$

6. Return $\mathcal{N}(f(\mathbf{x});\mu (\mathbf{x}),{\sigma }^2(\mathbf{x}))$

Multiple input index points

It is simple to extend this to compute $q_{\mathit{\varphi }}(\mathbf{f})$ for an arbitary number of index points $\mathbf{X}$:

1. Cholesky decomposition: $\mathbf{L}=\mathrm{cholesky}({\mathbf{K}}_{\mathbf{\text{uu}}})$

   Note: $\mathcal{O}(M^3)$ complexity.

2. Solve system of linear equations: $\mathbf{\Lambda }=\mathbf{L}\mathrm{\setminus }{\mathbf{K}}_{\mathbf{uf}}$

   Note: $\mathcal{O}(M^2)$ complexity since $\mathbf{L}$ is lower triangular; $\mathbf{B}=\mathbf{A}\mathrm{\setminus }\mathbf{X}$ denotes the matrix $\mathbf{B}$ such that $\mathbf{A}\mathbf{B}=\mathbf{X}\iff \mathbf{B}={\mathbf{A}}^{-1}\mathbf{X}$. Hence, $\mathbf{\Lambda }={\mathbf{L}}^{-1}{\mathbf{K}}_{\mathbf{uf}}$.

3. $\mathbf{S}\triangleq {\mathbf{K}}_{\mathbf{ff}}-{\mathbf{\Lambda }}^{\mathrm{\top }}\mathbf{\Lambda }$

   Note: $${\mathbf{\Lambda }}^{\mathrm{\top }}\mathbf{\Lambda }={\mathbf{K}}_{\mathbf{fu}}{\mathbf{L}}^{-\mathrm{\top }}{\mathbf{L}}^{-1}{\mathbf{K}}_{\mathbf{uf}}={\mathbf{K}}_{\mathbf{fu}}{\mathbf{K}}_{\mathbf{\text{uu}}}^{-1}{\mathbf{K}}_{\mathbf{uf}}={\mathbf{K}}_{\mathbf{fu}}{\mathbf{K}}_{\mathbf{\text{uu}}}^{-1}({\mathbf{K}}_{\mathbf{\text{uu}}}){\mathbf{K}}_{\mathbf{\text{uu}}}^{-1}{\mathbf{K}}_{\mathbf{uf}}={\mathbf{\Psi }}^{\mathrm{\top }}{\mathbf{K}}_{\mathbf{\text{uu}}}\mathbf{\Psi }.$$

4. For whitened parameterization:

   1. $\mathit{\mu }\triangleq {\mathbf{\Lambda }}^{\mathrm{\top }}{\mathbf{b}}^{\prime }$

   2. ${\mathbf{V}}^{\mathrm{\top }}\triangleq {\mathbf{\Lambda }}^{\mathrm{\top }}{\mathbf{W}}^{\prime }$

      Note: ${\mathbf{V}}^{\mathrm{\top }}\mathbf{V}={\mathbf{K}}_{\mathbf{fu}}{\mathbf{L}}^{-\mathrm{\top }}({\mathbf{W}}^{\prime }{{\mathbf{W}}^{\prime }}^{\mathrm{\top }}){\mathbf{L}}^{-1}{\mathbf{K}}_{\mathbf{uf}}.$

   otherwise:

   1. Solve system of linear equations: $\mathbf{\Psi }={\mathbf{L}}^{\mathrm{\top }}\mathrm{\setminus }\mathbf{\Lambda }$

      Note: $\mathcal{O}(M^2)$ complexity since ${\mathbf{L}}^{\mathrm{\top }}$ is upper triangular. Further,

      $$\mathbf{\Psi }={\mathbf{L}}^{-\mathrm{\top }}\mathbf{\Lambda }={\mathbf{L}}^{-\mathrm{\top }}{\mathbf{L}}^{-1}{\mathbf{K}}_{\mathbf{uf}}=(\mathbf{L}{\mathbf{L}}^{\mathrm{\top }}{)}^{-1}{\mathbf{K}}_{\mathbf{uf}}={\mathbf{K}}_{\mathbf{uu}}^{-1}{\mathbf{K}}_{\mathbf{uf}},$$ and $${\mathbf{\Psi }}^{\mathrm{\top }}={\mathbf{K}}_{\mathbf{fu}}{\mathbf{K}}_{\mathbf{uu}}^{-\mathrm{\top }}={\mathbf{K}}_{\mathbf{fu}}{\mathbf{K}}_{\mathbf{uu}}^{-1},$$ since ${\mathbf{K}}_{\mathbf{uu}}$ is symmetric and nonsingular.

   2. $\mathit{\mu }\triangleq {\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{b}$

   3. ${\mathbf{V}}^{\mathrm{\top }}\triangleq {\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{W}$

      Note: ${\mathbf{V}}^{\mathrm{\top }}\mathbf{V}={\mathbf{K}}_{\mathbf{fu}}{\mathbf{K}}_{\mathbf{uu}}^{-1}(\mathbf{W}{\mathbf{W}}^{\mathrm{\top }}){\mathbf{K}}_{\mathbf{uu}}^{-1}{\mathbf{K}}_{\mathbf{uf}}$.

5. $\mathbf{\Sigma }\triangleq \mathbf{S}+{\mathbf{V}}^{\mathrm{\top }}\mathbf{V}$

6. Return $\mathcal{N}(\mathbf{f};\mathit{\mu },\mathbf{\Sigma })$

In TensorFlow, this looks something like:

import tensorflow as tf


def variational_predictive(Knn, Kmm, Kmn, W, b, whiten=True, jitter=1e-6):

    L = tf.linalg.cholesky(Kmm + jitter * tf.eye(m))  # L L^T = Kmm + jitter I_m
    Lambda = tf.linalg.triangular_solve(L, Kmn, lower=True)  # Lambda = L^{-1} Kmn
    S = Knn - tf.linalg.matmul(Lambda, Lambda, adjoint_a=True)  # Knn - Lambda^T Lambda
    # Phi = L^{-T} L^{-1} Kmn = Kmm^{-1} Kmn
    Phi = Lambda if whiten else tf.linalg.triangular_solve(L, Lambda, adjoint=True, lower=True)
 
    U = tf.linalg.matmul(Phi, W, adjoint_a=True)  # U = V^T = Phi^T W
 
    mu = tf.linalg.matmul(Phi, b, adjoint_a=True)  # Phi^T b
    Sigma = S + tf.linalg.matmul(U, U, adjoint_b=True)  # S + UU^T = S + V^T V
 
    return mu, Sigma

III

Optimal variational distribution (in general)

Taking the functional derivative of the ELBO wrt to $q_{\mathit{\varphi }}(\mathbf{u})$, we get $$\begin{array}{rl}\frac{\partial }{\partial q_{\mathit{\varphi }}(\mathbf{u})}\mathrm{ELBO}(\mathit{\varphi },\mathbf{Z})& =\frac{\partial }{\partial q_{\mathit{\varphi }}(\mathbf{u})}(\int \log \frac{\mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})}{q_{\mathit{\varphi }}(\mathbf{u})}{q}_{\mathit{\varphi }}(\mathbf{u})\mathrm{d}\mathbf{u})\\ & =\int \frac{\partial }{\partial q_{\mathit{\varphi }}(\mathbf{u})}(\log \frac{\mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})}{q_{\mathit{\varphi }}(\mathbf{u})}{q}_{\mathit{\varphi }}(\mathbf{u}))\mathrm{d}\mathbf{u}\\ & =\begin{array}{rl}& \int \log \frac{\mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})}{q_{\mathit{\varphi }}(\mathbf{u})}(\frac{\partial }{\partial q_{\mathit{\varphi }}(\mathbf{u})}{q}_{\mathit{\varphi }}(\mathbf{u}))+\\ & q_{\mathit{\varphi }}(\mathbf{u})(\frac{\partial }{\partial q_{\mathit{\varphi }}(\mathbf{u})}\log \frac{\mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})}{q_{\mathit{\varphi }}(\mathbf{u})})\mathrm{d}\mathbf{u}\end{array}\\ & =\int \log \frac{\mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})}{q_{\mathit{\varphi }}(\mathbf{u})}+q_{\mathit{\varphi }}(\mathbf{u})(-\frac{1}{q_{\mathit{\varphi }}(\mathbf{u})})\mathrm{d}\mathbf{u}\\ & =\int \log \mathrm{\Phi }(\mathbf{y},\mathbf{u})+\log p(\mathbf{u})-\log q_{\mathit{\varphi }}(\mathbf{u})-1\mathrm{d}\mathbf{u}.\end{array}$$ Setting this expression to zero, we have $$\begin{array}{rl}\log q_{{\mathit{\varphi }}^{\star }}(\mathbf{u})& =\log \mathrm{\Phi }(\mathbf{y},\mathbf{u})+\log p(\mathbf{u})-1\\ \Rightarrow q_{{\mathit{\varphi }}^{\star }}(\mathbf{u})& \propto \mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u}).\end{array}$$

IV

Variational lower bound (partial) for Gaussian likelihoods

To carry out this derivation, we will need to recall the following two simple identities. First, we can write the inner product between two vectors as the trace of their outer product, $${\mathbf{a}}^{\mathrm{\top }}\mathbf{b}=\mathrm{tr}(\mathbf{a}{\mathbf{b}}^{\mathrm{\top }}).$$ Second, the relationship between the auto-correlation matrix $\mathbb{E}[\mathbf{a}{\mathbf{a}}^{\mathrm{\top }}]$ and the covariance matrix, $$\begin{array}{rl}\mathrm{Cov}[\mathbf{a}]& =\mathbb{E}[\mathbf{a}{\mathbf{a}}^{\mathrm{\top }}]-\mathbb{E}[\mathbf{a}]\mathbb{E}[\mathbf{a}{]}^{\mathrm{\top }}\\ \iff \mathbb{E}[\mathbf{a}{\mathbf{a}}^{\mathrm{\top }}]& =\mathrm{Cov}[\mathbf{a}]+\mathbb{E}[\mathbf{a}]\mathbb{E}[\mathbf{a}{]}^{\mathrm{\top }}\end{array}$$ These allow us to write $$\begin{array}{rl}\log \mathrm{\Phi }(\mathbf{y},\mathbf{u})& =\int \log \mathcal{N}(\mathbf{y}|\mathbf{f},{\beta }^{-1}\mathbf{I})\mathcal{N}(\mathbf{f}|\mathbf{m},\mathbf{S})\mathrm{d}\mathbf{f}\\ & =-\frac{1}{2{\sigma }^2}\int (\mathbf{y}-\mathbf{f}{)}^{\mathrm{\top }}(\mathbf{y}-\mathbf{f})\mathcal{N}(\mathbf{f}|\mathbf{m},\mathbf{S})\mathrm{d}\mathbf{f}-\frac{N}{2}\log (2\pi {\sigma }^2)\\ & =-\frac{1}{2{\sigma }^2}\int \mathrm{tr}(\mathbf{y}{\mathbf{y}}^{\mathrm{\top }}-2\mathbf{y}{\mathbf{f}}^{\mathrm{\top }}+\mathbf{f}{\mathbf{f}}^{\mathrm{\top }})\mathcal{N}(\mathbf{f}|\mathbf{m},\mathbf{S})\mathrm{d}\mathbf{f}-\frac{N}{2}\log (2\pi {\sigma }^2)\\ & =-\frac{1}{2{\sigma }^2}\mathrm{tr}(\mathbf{y}{\mathbf{y}}^{\mathrm{\top }}-2\mathbf{y}{\mathbf{m}}^{\mathrm{\top }}+\mathbf{S}+\mathbf{m}{\mathbf{m}}^{\mathrm{\top }})-\frac{N}{2}\log (2\pi {\sigma }^2)\\ & =-\frac{1}{2{\sigma }^2}(\mathbf{y}-\mathbf{m}{)}^{\mathrm{\top }}(\mathbf{y}-\mathbf{m})-\frac{N}{2}\log (2\pi {\sigma }^2)-\frac{1}{2{\sigma }^2}\mathrm{tr}(\mathbf{S})\\ & =\log \mathcal{N}(\mathbf{y}|\mathbf{m},{\beta }^{-1}\mathbf{I})-\frac{1}{2{\sigma }^2}\mathrm{tr}(\mathbf{S}).\end{array}$$

V

Optimal variational distribution for Gaussian likelihoods

Firstly, the optimal variational distribution can be found in closed-form as $$\begin{array}{rl}{q}_{{\mathit{\varphi }}^{\star }}(\mathbf{u})& \propto \mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})\\ & \propto \mathcal{N}(\mathbf{y}\mid {\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{u},{\beta }^{-1}\mathbf{I})\mathcal{N}(\mathbf{u}\mid \mathbf{0},{\mathbf{K}}_{\mathbf{uu}})\\ & \propto \exp (-\frac{\beta }{2}(\mathbf{y}-{\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{u}{)}^{\mathrm{\top }}(\mathbf{y}-{\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{u})-\frac{1}{2}{\mathbf{u}}^{\mathrm{\top }}{\mathbf{K}}_{\mathbf{uu}}^{-1}\mathbf{u})\\ & \propto \exp (-\frac{1}{2}({\mathbf{u}}^{\mathrm{\top }}\mathbf{C}\mathbf{u}-2\beta (\mathbf{\Psi }\mathbf{y}{)}^{\mathrm{\top }}\mathbf{u})),\end{array}$$ where $$\mathbf{C}\triangleq {\mathbf{K}}_{\mathbf{uu}}^{-1}+\beta \mathbf{\Psi }{\mathbf{\Psi }}^{\mathrm{\top }}={\mathbf{K}}_{\mathbf{uu}}^{-1}({\mathbf{K}}_{\mathbf{uu}}+\beta {\mathbf{K}}_{\mathbf{uf}}{\mathbf{K}}_{\mathbf{fu}}){\mathbf{K}}_{\mathbf{uu}}^{-1}.$$ By completing the square, we get $$\begin{array}{rl}{q}_{{\mathit{\varphi }}^{\star }}(\mathbf{u})& \propto \exp (-\frac{1}{2}(\mathbf{u}-\beta {\mathbf{C}}^{-1}\mathbf{\Psi }\mathbf{y}{)}^{\mathrm{\top }}\mathbf{C}(\mathbf{u}-\beta {\mathbf{C}}^{-1}\mathbf{\Psi }\mathbf{y}))\\ & \propto \mathcal{N}(\mathbf{u}\mid \beta {\mathbf{C}}^{-1}\mathbf{\Psi }\mathbf{y},{\mathbf{C}}^{-1}).\end{array}$$ We define $$\mathbf{M}\triangleq {\mathbf{K}}_{\mathbf{uu}}+\beta {\mathbf{K}}_{\mathbf{uf}}{\mathbf{K}}_{\mathbf{fu}}$$ so that $$\mathbf{C}={\mathbf{K}}_{\mathbf{uu}}^{-1}\mathbf{M}{\mathbf{K}}_{\mathbf{uu}}^{-1},$$ which allows us to write $$q_{{\mathit{\varphi }}^{\star }}(\mathbf{u})=\mathcal{N}(\mathbf{u}\mid \beta {\mathbf{K}}_{\mathbf{uu}}{\mathbf{M}}^{-1}{\mathbf{K}}_{\mathbf{uf}}\mathbf{y},{\mathbf{K}}_{\mathbf{uu}}{\mathbf{M}}^{-1}{\mathbf{K}}_{\mathbf{uu}}).$$

VI

Variational lower bound (complete) for Gaussian likelihoods

We have $$\begin{array}{rl}\mathrm{ELBO}({\mathit{\varphi }}^{\star },\mathbf{Z})& =\log \mathcal{Z}\\ & =\log \int \mathrm{\Phi }(\mathbf{y},\mathbf{u})p(\mathbf{u})\mathrm{d}\mathbf{u}\\ & =\log [\exp (-\frac{\beta }{2}\mathrm{tr}(\mathbf{S}))\int \mathcal{N}(\mathbf{y}|{\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{u},{\beta }^{-1}\mathbf{I})p(\mathbf{u})\mathrm{d}\mathbf{u}]\\ & =\log \int \mathcal{N}(\mathbf{y}\mid {\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{u},{\beta }^{-1}\mathbf{I})\mathcal{N}(\mathbf{u}\mid \mathbf{0},{\mathbf{K}}_{\mathbf{uu}})\mathrm{d}\mathbf{u}-\frac{\beta }{2}\mathrm{tr}(\mathbf{S})\\ & =\log \mathcal{N}(\mathbf{y}\mid \mathbf{0},{\beta }^{-1}\mathbf{I}+{\mathbf{\Psi }}^{\mathrm{\top }}{\mathbf{K}}_{\mathbf{\text{uu}}}\mathbf{\Psi })-\frac{\beta }{2}\mathrm{tr}(\mathbf{S})\\ & =\log \mathcal{N}(\mathbf{y}\mid \mathbf{0},{\mathbf{Q}}_{\mathbf{ff}}+{\beta }^{-1}\mathbf{I})-\frac{\beta }{2}\mathrm{tr}(\mathbf{S}).\end{array}$$

VII

SGPR Implementation Details

Here we provide implementation details that simultaneously minimizes the computational demands while avoiding numerically unstable calculations.

The difficulty in calculating the ELBO stem from terms involving the inverse and the determinant of ${\mathbf{Q}}_{\mathbf{ff}}+{\beta }^{-1}\mathbf{I}$. More specifically, we have $$\begin{array}{rl}\mathrm{ELBO}({\mathit{\varphi }}^{\star },\mathbf{Z})& =-\frac{1}{2}(\log det({\mathbf{Q}}_{\mathbf{ff}}+{\beta }^{-1}\mathbf{I})+{\mathbf{y}}^{\mathrm{\top }}{({\mathbf{Q}}_{\mathbf{ff}}+{\beta }^{-1}\mathbf{I})}^{-1}\mathbf{y}+N\log 2\pi )\\ & -\frac{\beta }{2}\mathrm{tr}(\mathbf{S}).\end{array}$$ It turns out that many of the required terms can be expressed in terms of the symmetric positive definite matrix $$\mathbf{B}\triangleq \mathbf{U}{\mathbf{U}}^{\mathrm{\top }}+\mathbf{I},$$ where $\mathbf{U}\triangleq {\beta }^{\frac{1}{2}}\mathbf{\Lambda }$.

First, let’s tackle the inverse term. Using the Woodbury identity, we can write it as $$\begin{array}{rl}{({\mathbf{Q}}_{\mathbf{ff}}+{\beta }^{-1}\mathbf{I})}^{-1}& ={({\beta }^{-1}\mathbf{I}+{\mathbf{\Psi }}^{\mathrm{\top }}{\mathbf{K}}_{\mathbf{uu}}\mathbf{\Psi })}^{-1}\\ & =\beta \mathbf{I}-{\beta }^2{\mathbf{\Psi }}^{\mathrm{\top }}{({\mathbf{K}}_{\mathbf{uu}}^{-1}+\beta \mathbf{\Psi }{\mathbf{\Psi }}^{\mathrm{\top }})}^{-1}\mathbf{\Psi }\\ & =\beta (\mathbf{I}-\beta {\mathbf{\Psi }}^{\mathrm{\top }}{\mathbf{C}}^{-1}\mathbf{\Psi }).\end{array}$$

Recall that ${\mathbf{C}}^{-1}={\mathbf{K}}_{\mathbf{uu}}{\mathbf{M}}^{-1}{\mathbf{K}}_{\mathbf{uu}}$. We can expand $\mathbf{M}$ as $$\begin{array}{rl}\mathbf{M}& \triangleq {\mathbf{K}}_{\mathbf{uu}}+\beta {\mathbf{K}}_{\mathbf{uf}}{\mathbf{K}}_{\mathbf{fu}}\\ & =\mathbf{L}{\mathbf{L}}^{\mathrm{\top }}+\beta \mathbf{L}{\mathbf{L}}^{-1}{\mathbf{K}}_{\mathbf{uf}}{\mathbf{K}}_{\mathbf{fu}}{\mathbf{L}}^{-\mathrm{\top }}{\mathbf{L}}^{\mathrm{\top }}\\ & =\mathbf{L}(\mathbf{I}+\beta \mathbf{\Lambda }{\mathbf{\Lambda }}^{\mathrm{\top }}){\mathbf{L}}^{\mathrm{\top }}\\ & =\mathbf{L}\mathbf{B}{\mathbf{L}}^{\mathrm{\top }},\end{array}$$ so its inverse is simply $${\mathbf{M}}^{-1}={\mathbf{L}}^{-\mathrm{\top }}{\mathbf{B}}^{-1}{\mathbf{L}}^{-1}.$$ Therefore, we have $$\begin{array}{rl}{\mathbf{C}}^{-1}& ={\mathbf{K}}_{\mathbf{uu}}{\mathbf{L}}^{-\mathrm{\top }}{\mathbf{B}}^{-1}{\mathbf{L}}^{-1}{\mathbf{K}}_{\mathbf{uu}}\\ & =\mathbf{L}{\mathbf{B}}^{-1}{\mathbf{L}}^{\mathrm{\top }}\\ & =\mathbf{W}{\mathbf{W}}^{\mathrm{\top }}\end{array}$$ where $$\mathbf{W}\triangleq \mathbf{L}{\mathbf{L}}_{\mathbf{B}}^{-\mathrm{\top }}$$ and ${\mathbf{L}}_{\mathbf{B}}$ is the Cholesky factor of $\mathbf{B}$, i.e. the lower triangular matrix such that ${\mathbf{L}}_{\mathbf{B}}{\mathbf{L}}_{\mathbf{B}}^{\mathrm{\top }}=\mathbf{B}$. All in all, we now have $$\begin{array}{rl}{({\mathbf{Q}}_{\mathbf{ff}}+{\beta }^{-1}\mathbf{I})}^{-1}& =\beta (\mathbf{I}-\beta {\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{W}{\mathbf{W}}^{\mathrm{\top }}\mathbf{\Psi }),\end{array}$$ so we can compute the quadratic term in $\mathbf{y}$ as $$\begin{array}{rl}{\mathbf{y}}^{\mathrm{\top }}{({\mathbf{Q}}_{\mathbf{ff}}+{\beta }^{-1}\mathbf{I})}^{-1}\mathbf{y}& =\beta ({\mathbf{y}}^{\mathrm{\top }}\mathbf{y}-\beta {\mathbf{y}}^{\mathrm{\top }}{\mathbf{\Psi }}^{\mathrm{\top }}\mathbf{W}{\mathbf{W}}^{\mathrm{\top }}\mathbf{\Psi }\mathbf{y})\\ & =\beta {\mathbf{y}}^{\mathrm{\top }}\mathbf{y}-{\mathbf{c}}^{\mathrm{\top }}\mathbf{c},\end{array}$$ where $$\mathbf{c}\triangleq \beta {\mathbf{W}}^{\mathrm{\top }}\mathbf{\Psi }\mathbf{y}=\beta {\mathbf{L}}_{\mathbf{B}}^{-1}\mathbf{\Lambda }\mathbf{y}={\beta }^{\frac{1}{2}}{\mathbf{L}}_{\mathbf{B}}^{-1}\mathbf{U}\mathbf{y}.$$

Next, let’s address the determinant term. To this end, first note that the determinant of $\mathbf{M}$ is $$\begin{array}{rl}det\left(\mathbf{M}\right)& =det\left(\mathbf{L}\mathbf{B}{\mathbf{L}}^{\mathrm{\top }}\right)\\ & =det\left(\mathbf{L}\right)det\left(\mathbf{B}\right)det\left({\mathbf{L}}^{\mathrm{\top }}\right)\\ & =det\left({\mathbf{K}}_{\mathbf{uu}}\right)det\left(\mathbf{B}\right).\end{array}$$ Hence, the determinant of $\mathbf{C}$ is $$\begin{array}{rl}det\left(\mathbf{C}\right)& =det\left({\mathbf{K}}_{\mathbf{uu}}^{-1}\mathbf{M}{\mathbf{K}}_{\mathbf{uu}}^{-1}\right)\\ & =\frac{det\left(\mathbf{M}\right)}{det{\left({\mathbf{K}}_{\mathbf{uu}}\right)}^2}\\ & =\frac{det\left(\mathbf{B}\right)}{det\left({\mathbf{K}}_{\mathbf{uu}}\right)}.\end{array}$$ Therefore, by the matrix determinant lemma, we have $$\begin{array}{rl}det({\mathbf{Q}}_{\mathbf{ff}}+{\beta }^{-1}\mathbf{I})& =det({\beta }^{-1}\mathbf{I}+{\mathbf{\Psi }}^{\mathrm{\top }}{\mathbf{K}}_{\mathbf{uu}}\mathbf{\Psi })\\ & =det({\mathbf{K}}_{\mathbf{uu}}^{-1}+\beta \mathbf{\Psi }{\mathbf{\Psi }}^{\mathrm{\top }})det\left({\mathbf{K}}_{\mathbf{uu}}\right)det\left({\beta }^{-1}\mathbf{I}\right)\\ & =det\left(\mathbf{C}\right)det\left({\mathbf{K}}_{\mathbf{uu}}\right)det\left({\beta }^{-1}\mathbf{I}\right)\\ & =det\left(\mathbf{B}\right)det\left({\beta }^{-1}\mathbf{I}\right).\end{array}$$ We can re-use ${\mathbf{L}}_{\mathbf{B}}$ to calculate $det\left(\mathbf{B}\right)$ in linear time.