Comparing Markov Chain Monte Carlo and Variational Methods for Bayesian Inference on Mixtures of Gamma Distributions

This was my undergraduate honours thesis at the University of Western Australia, supervised by John Lau and Edward Cripps. It develops a coordinate-ascent variational inference procedure for finite mixtures of gamma distributions and compares it with a Gibbs sampler on synthetic datasets and Australian rainfall data.

The central difficulty is the expectation of the log gamma function under the variational distribution for the shape parameter. The method approximates this using Taylor and Stirling expansions. Under the conventional shape-rate parameterisation, the mean-field approximation substantially underestimates posterior variance and cannot represent the dependence between shape and rate. Reparameterising the model in terms of shape and mean largely overcomes that problem at negligible computational cost.

The thesis compares the resulting variational methods with MCMC in posterior accuracy, predictive performance, and execution time. Variational inference converges much faster and provides a closed-form posterior approximation, while the Gibbs sampler retains the stronger asymptotic guarantees and more faithfully represents posterior dependence.

Read the thesis PDF

MixGammaVI is the accompanying TensorFlow and TensorFlow Probability implementation. The repository includes examples and the source released on PyPI. It is an archival research project rather than a currently maintained statistical package.