Probabilistic Machine Learning in the Age of Deep Learning: New Perspectives for Gaussian Processes, Bayesian Optimization and Beyond (PhD Thesis)

Louis Tiao
September, 2023
Preprint Full Acknowledgements

Abstract

Advances in artificial intelligence (AI) are rapidly transforming our world, with systems now matching or surpassing human capabilities in areas ranging from game-playing to scientific discovery. Much of this progress traces back to machine learning (ML), particularly deep learning and its ability to uncover meaningful patterns and representations in data. However, true intelligence in AI demands more than raw predictive power; it requires a principled approach to making decisions under uncertainty. This highlights the necessity of probabilistic ML, which offers a systematic framework for reasoning about the unknown through probability theory and Bayesian inference. Gaussian processes (GPs) stand out as a quintessential probabilistic model, offering flexibility, data efficiency, and well-calibrated uncertainty estimates. They are integral to many sequential decision-making algorithms, notably Bayesian optimisation (BO), which has emerged as an indispensable tool for optimising expensive and complex black-box objective functions. While considerable efforts have focused on improving gp scalability, performance gaps persist in practice when compared against neural networks (NNs) due in large to its lack of representation learning capabilities. This, among other natural deficiencies of GPs, has hampered the capacity of BO to address critical real-world optimisation challenges. This thesis aims to unlock the potential of deep learning within probabilistic methods and reciprocally lend probabilistic perspectives to deep learning. The contributions include improving approximations to bridge the gap between GPs and NNs, providing a new formulation of BO that seamlessly accommodates deep learning methods to tackle complex optimisation problems, as well as a probabilistic interpretation of a powerful class of deep generative models for image style transfer. By enriching the interplay between deep learning and probabilistic ML, this thesis advances the foundations of AI and facilitates the development of more capable and dependable automated decision-making systems.

Type
Thesis

The full text is available as a single PDF file

You can also find a list of contents and PDFs corresponding to each individual chapter below:

Table of Contents

  • Chapter 1: Introduction
  • Chapter 2: Background
  • Chapter 3: Orthogonally-Decoupled Sparse Gaussian Processes with Spherical Neural Network Activation Features
  • Chapter 4: Cycle-Consistent Generative Adversarial Networks as a Bayesian Approximation
  • Chapter 5: Bayesian Optimisation by Classification with Deep Learning and Beyond
  • Chapter 6: Conclusion
  • Appendix A: Numerical Methods for Improved Decoupled Sampling of Gaussian Processes
  • Bibliography

Please find Chapter 1: Introduction reproduced in full below:

Introduction

Machine Learning Probabilistic Models Gaussian Processes Variational Inference Bayesian Optimization Deep Learning
Louis Tiao
Louis Tiao
PhD Candidate (AI & Machine Learning)

Thanks for stopping by! Let’s connect – drop me a message or follow me