Hi. My name is Louis. I am a budding machine learning researcher and PhD candidate at the University of Sydney, working with Edwin Bonilla and Fabio Ramos. My main research interests lie at the intersection of Bayesian deep learning, approximate inference, and probabilistic models with intractable likelihoods.

Until 2017, I was a software engineer at NICTA (now incorporated under CSIRO as Data61) in the inference systems group, working on scalable Bayesian machine learning. I now work at Data61 on a part-time basis when I am not teaching.

Prior to that, I studied computer science at the University of New South Wales, where I had a major emphasis on algorithm design and analysis, theoretical computer science, programming language theory, artificial intelligence, machine learning, and a minor emphasis on mathematics and statistics. I undertook my final-year thesis under Aleksandar Ignjatovic, and graduated with first-class honours in 2015.

Research Experience

In the Summer-Fall of 2019, I interned as an Applied Scientist at Amazon Berlin, conducting research in the areas of AutoML and Hyperparameter Optimization for the AWS SageMaker’s Automatic Model Tuning service.

I had the good fortune of working with Matthias Seeger and Cédric Archambeau, and together, we tackled the challenges of making Multi-fidelity Bayesian Optimization asynchronously parallel.

The research developed during my internship culminated in a research paper and the release of our code as part of the open-source AutoGluon library.

Bayesian optimization (BO) is among the most effective and widely-used blackbox optimization methods. BO proposes solutions according to an explore-exploit trade-off criterion encoded in an acquisition function, many of which are derived from the posterior predictive of a probabilistic surrogate model. Prevalent among these is the expected improvement (EI). Naturally, the need to ensure analytical tractability in the model poses limitations that can ultimately hinder the efficiency and applicability of BO. In this paper, we cast the computation of EI as a binary classification problem, building on the well-known link between class probability estimation (CPE) and density ratio estimation (DRE), and the lesser-known link between density ratios and EI. By circumventing the tractability constraints imposed on the model, this reformulation provides several natural advantages, not least in scalability, increased flexibility, and greater representational capacity.

We propose a framework that lifts the capabilities of graph convolutional networks (GCNs) to scenarios where no input graph is given and increases their robustness to adversarial attacks. We formulate a joint probabilistic model that considers a prior distribution over graphs along with a GCN-based likelihood and develop a stochastic variational inference algorithm to estimate the graph posterior and the GCN parameters jointly. To address the problem of propagating gradients through latent variables drawn from discrete distributions, we use their continuous relaxations known as Concrete distributions. We show that, on real datasets, our approach can outperform state-of-the-art Bayesian and non-Bayesian graph neural network algorithms on the task of semi-supervised classification in the absence of graph data and when the network structure is subjected to adversarial perturbations.

We introduce a model-based asynchronous multi-fidelity method for hyperparameter and neural architecture search that combines the strengths of asynchronous Hyperband and Gaussian process-based Bayesian optimization. At the heart of our method is a probabilistic model that can simultaneously reason across hyperparameters and resource levels, and supports decision-making in the presence of pending evaluations. We demonstrate the effectiveness of our method on a wide range of challenging benchmarks, for tabular data, image classification and language modelling, and report substantial speed-ups over current state-of-the-art methods. Our new methods, along with asynchronous baselines, are implemented in a distributed framework which will be open sourced along with this publication.

We propose a framework that lifts the capabilities of graph convolutional networks (GCNs) to scenarios where no input graph is given and increases their robustness to adversarial attacks. We formulate a joint probabilistic model that considers a prior distribution over graphs along with a GCN-based likelihood and develop a stochastic variational inference algorithm to estimate the graph posterior and the GCN parameters jointly. To address the problem of propagating gradients through latent variables drawn from discrete distributions, we use their continuous relaxations known as Concrete distributions. We show that, on real datasets, our approach can outperform state-of-the-art Bayesian and non-Bayesian graph neural network algorithms on the task of semi-supervised classification in the absence of graph data and when the network structure is subjected to adversarial perturbations.

(2020).

Variational Inference for Graph Convolutional Networks in the Absence of Graph Data and Adversarial Settings.
In *Neural Information Processing Systems (NeurIPS) 2020 (To appear as a ***Spotlight Presentation** – Top 3%).

(2020).
(2020).
Variational Graph Convolutional Networks.
In *NeurIPS2019* Graph Representation Learning *(Accepted as ***Outstanding Contribution Talk**).

(2019).
Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference.
In *ICML2018* Theoretical Foundations and Applications of Deep Generative Models *(Accepted as ***Contributed Talk**).

(2018).
A summary of notation, identities and derivations for the sparse variational Gaussian process (SVGP) framework.

This post demonstrates how to approximate the KL divergence (in fact, any f-divergence) between implicit distributions, using density ratio estimation by probabilistic classification.

We illustrate how to build complicated probability distributions in a modular fashion using the Bijector API from TensorFlow Probability.

An in-depth practical guide to variational encoders from a probabilistic perspective.

The meshgrid function is useful for creating coordinate arrays to vectorize function evaluations over a grid. Experienced NumPy users will have noticed some discrepancy between meshgrid and the mgrid, a function that is used just as often, for exactly the same purpose.

ECCV2020 Tutorial – From HPO to NAS: Automated Deep Learning

Outstanding Contribution Talk at NeurIPS2019 Workshop on Graph Representation Learning

AutoGluon is a library for asynchronously distributed hyperparameter optimization (HPO) and neural architecture search (NAS) that implements numerous state-of-the-art methods. I was a core developer of the Gaussian process-based multi-fidelity searcher module.

Expanding the scope and applicability of variational inference to encompass implicit probabilistic models.

Aboleth is a minimalistic TensorFlow framework for scalable Bayesian deep learning and Gaussian process approximation.

Determinant is a software service that makes predictions from sparse data, and learns what data it needs to optimise its performance.

Revrand is a full-featured Python library for Bayesian generalized linear models, with random basis kernels for large-scale Gaussian process approximations.

The course has a primary focus on probabilistic machine learning methods, covering the topics of exact and approximate inference in directed and undirected probabilistic graphical models - continuous latent variable models, structured prediction models, and non-parametric models based on Gaussian processes.

This course has a major emphasis on maintaining a good balance between theory and practice. As the teaching assistant (TA) for this course, my primary responsibility was to create lab exercises that aid students in gaining hands-on experience with these methods, specifically applying them to real-world data using the most current tools and libraries. The labs were Python-based, and relied heavily on the Python scientific computing and data analysis stack ( NumPy, SciPy, Matplotlib, Seaborn, Pandas, IPython/Jupyter notebooks), and the popular machine learning libraries scikit-learn and TensorFlow.

Students were given the chance to experiment with a broad range of methods on various problems, such as Markov chain Monte Carlo (MCMC) for Bayesian logistic regression, probabilistic PCA (PPCA), factor analysis (FA) and independent component analysis (ICA) for dimensionality reduction, hidden Markov models (HMMs) for speech recognition, conditional random fields (CRFs) for named-entity recognition, and Gaussian processes (GPs) for regression and classification.