Constructing Generative Topographic Mapping by Variational Bayes with ARD Hierarchical Prior
Graduate School of Science and Engineering, Saga University, 1 Honjo, Saga 840-8502, Japan
Generative Topographic Mapping (GTM) is a nonlinear latent variable model introduced as a data visualization technique by Bishop et al. In this paper, we focus on variational Bayesian inference in GTM. Variational Bayesian GTM, first proposed by Olier et al., uses a single regularization term and regularization parameter to avoid overfitting and therefore cannot be used to control the degree of regularization locally. To overcome this problem, we propose variational Bayesian inference with Automatic Relevance Determination (ARD) hierarchical prior for use with GTM. The proposed model uses multiple regularization parameters and therefore can be used to control the degree of regularization in local areas of data space individually. Several experiments show that GTM that we propose provides better visualization than conventional GTM approaches.
-  C. M. Bishop, M. Svensén, and C. K. I. Williams, “GTM: the generative topographic mapping,” Neural Computation, Vol.10, No.1, pp. 215-234, 1998.
-  A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” J. of the Royal Statistical Society. Series B (Methodological), Vol.39, No.1, pp. 1-38, 1977.
-  T. Kohonen, M. R. Schroeder, and T. S. Huang (Eds.), “Self-Organizing Maps,” Springer-Verlag, 2001.
-  C. M. Bishop and C. K. I. Williams, “Developments of the generative topographic mapping,” Neurocomputing, Vol.21, pp. 203-224, 1998.
-  A. Vellido, W. El-Deredy, and P. J. G. Lisboa, “Selective smoothing of the generative topographic mapping,” IEEE Trans. on Neural Networks, Vol.14, No.4, pp. 847-852, 2003.
-  D. J. C. MacKay, “Probable networks and plausible predictions – a review of practical Bayesian methods for supervised neural networks,” Network: Computation in Neural Systems, Vol.6, No.3, pp. 469-505, 1995.
-  H. Attias, “Inferring parameters and structure of latent variable models by variational bayes,” Proc. of the 15th Conf. on Uncertainty in artificial intelligence, pp. 21-30, 1999.
-  M. Beal, “Variational Algorithms for Approximate Bayesian Inference,” Ph.D. thesis, Gatsby Computational Neuroscience Unit, University College London, 2003.
-  I. Olier, J. Amengual, and A. Vellido, “Variational Bayesian Generative Topographic Mapping,” J. of Mathematical Modelling and Algorithms, Vol.7, No.4, pp. 371-387, 2008.
-  R. M. Neal, “Bayesian Learning for Neural Networks,” Springer-Verlag, New York, 1996.
-  C. Blake and C. Merz, “UCI Repository of machine learning databases,” 1998.
-  M. Sato, “Online Model Selection Based on the Variational Bayes,” Neural Computation, Vol.13, No.7, pp. 1649-1681, 2001.