Constructing Generative Topographic Mapping by Variational Bayes with ARD Hierarchical Prior

Nobuhiko Yamaguchi

doi:10.20965/jaciii.2013.p0473

single-jc.php

JACIII Vol.17 No.4 pp. 473-479

doi: 10.20965/jaciii.2013.p0473

(2013)

Paper:

Views over last 60 days: 443

Constructing Generative Topographic Mapping by Variational Bayes with ARD Hierarchical Prior

Nobuhiko Yamaguchi

Graduate School of Science and Engineering, Saga University, 1 Honjo, Saga 840-8502, Japan

Received:

January 11, 2013

Accepted:

April 2, 2013

Published:

July 20, 2013

Keywords:

generative topographic mapping, data visualization, variational Bayes, automatic relevance determination

Abstract

Generative Topographic Mapping (GTM) is a nonlinear latent variable model introduced as a data visualization technique by Bishop et al. In this paper, we focus on variational Bayesian inference in GTM. Variational Bayesian GTM, first proposed by Olier et al., uses a single regularization term and regularization parameter to avoid overfitting and therefore cannot be used to control the degree of regularization locally. To overcome this problem, we propose variational Bayesian inference with Automatic Relevance Determination (ARD) hierarchical prior for use with GTM. The proposed model uses multiple regularization parameters and therefore can be used to control the degree of regularization in local areas of data space individually. Several experiments show that GTM that we propose provides better visualization than conventional GTM approaches.

Cite this article as:

N. Yamaguchi, “Constructing Generative Topographic Mapping by Variational Bayes with ARD Hierarchical Prior,” J. Adv. Comput. Intell. Intell. Inform., Vol.17 No.4, pp. 473-479, 2013.

Data files:

References

[1] C. M. Bishop, M. Svensén, and C. K. I. Williams, “GTM: the generative topographic mapping,” Neural Computation, Vol.10, No.1, pp. 215-234, 1998.
[2] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” J. of the Royal Statistical Society. Series B (Methodological), Vol.39, No.1, pp. 1-38, 1977.
[3] T. Kohonen, M. R. Schroeder, and T. S. Huang (Eds.), “Self-Organizing Maps,” Springer-Verlag, 2001.
[4] C. M. Bishop and C. K. I. Williams, “Developments of the generative topographic mapping,” Neurocomputing, Vol.21, pp. 203-224, 1998.
[5] A. Vellido, W. El-Deredy, and P. J. G. Lisboa, “Selective smoothing of the generative topographic mapping,” IEEE Trans. on Neural Networks, Vol.14, No.4, pp. 847-852, 2003.
[6] D. J. C. MacKay, “Probable networks and plausible predictions – a review of practical Bayesian methods for supervised neural networks,” Network: Computation in Neural Systems, Vol.6, No.3, pp. 469-505, 1995.
[7] H. Attias, “Inferring parameters and structure of latent variable models by variational bayes,” Proc. of the 15th Conf. on Uncertainty in artificial intelligence, pp. 21-30, 1999.
[8] M. Beal, “Variational Algorithms for Approximate Bayesian Inference,” Ph.D. thesis, Gatsby Computational Neuroscience Unit, University College London, 2003.
[9] I. Olier, J. Amengual, and A. Vellido, “Variational Bayesian Generative Topographic Mapping,” J. of Mathematical Modelling and Algorithms, Vol.7, No.4, pp. 371-387, 2008.
[10] R. M. Neal, “Bayesian Learning for Neural Networks,” Springer-Verlag, New York, 1996.
[11] C. Blake and C. Merz, “UCI Repository of machine learning databases,” 1998.
[12] M. Sato, “Online Model Selection Based on the Variational Bayes,” Neural Computation, Vol.13, No.7, pp. 1649-1681, 2001.

This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.

[1] [1] C. M. Bishop, M. Svensén, and C. K. I. Williams, “GTM: the generative topographic mapping,” Neural Computation, Vol.10, No.1, pp. 215-234, 1998.

[2] [2] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” J. of the Royal Statistical Society. Series B (Methodological), Vol.39, No.1, pp. 1-38, 1977.

[3] [3] T. Kohonen, M. R. Schroeder, and T. S. Huang (Eds.), “Self-Organizing Maps,” Springer-Verlag, 2001.

[4] [4] C. M. Bishop and C. K. I. Williams, “Developments of the generative topographic mapping,” Neurocomputing, Vol.21, pp. 203-224, 1998.

[5] [5] A. Vellido, W. El-Deredy, and P. J. G. Lisboa, “Selective smoothing of the generative topographic mapping,” IEEE Trans. on Neural Networks, Vol.14, No.4, pp. 847-852, 2003.

[6] [6] D. J. C. MacKay, “Probable networks and plausible predictions – a review of practical Bayesian methods for supervised neural networks,” Network: Computation in Neural Systems, Vol.6, No.3, pp. 469-505, 1995.

[7] [7] H. Attias, “Inferring parameters and structure of latent variable models by variational bayes,” Proc. of the 15th Conf. on Uncertainty in artificial intelligence, pp. 21-30, 1999.

[8] [8] M. Beal, “Variational Algorithms for Approximate Bayesian Inference,” Ph.D. thesis, Gatsby Computational Neuroscience Unit, University College London, 2003.

[9] [9] I. Olier, J. Amengual, and A. Vellido, “Variational Bayesian Generative Topographic Mapping,” J. of Mathematical Modelling and Algorithms, Vol.7, No.4, pp. 371-387, 2008.

[10] [10] R. M. Neal, “Bayesian Learning for Neural Networks,” Springer-Verlag, New York, 1996.

[11] [11] C. Blake and C. Merz, “UCI Repository of machine learning databases,” 1998.

[12] [12] M. Sato, “Online Model Selection Based on the Variational Bayes,” Neural Computation, Vol.13, No.7, pp. 1649-1681, 2001.