Adaptive Critic Design with Local Gaussian Process Models
Wei Wang, Xin Chen†, and Jianxin He
School of Automation, China University of Geosciences
Wuhan 430074, China
In this paper, local Gaussian process (GP) approximation is introduced to build the critic network of adaptive dynamic programming (ADP). The sample data are partitioned into local regions, and for each region, an individual GP model is utilized. The nearest local model is used to predict a given state-action point. With the two-phase value iteration method for a Gaussian-kernel (GK)-based critic network which realizes the update of the hyper-parameters and value functions simultaneously, fast value function approximation can be achieved. Combining this critic network with an actor network, we present a local GK-based ADP approach. Simulations were carried out to demonstrate the feasibility of the proposed approach.
-  R. S. Sutton and A. G. Barto, “Reinforcement learning: An introduction,” Cambridge, MA, USA: MIT Press, 1998.
-  L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey,” J. of Artificial Intelligence Research, Vol.4, pp. 237-285, 1996.
-  W. B. Powell, “Approximate Dynamic Programming: Solving the Curses of Dimensionality,” New York, USA: Wiley, 2007.
-  F. Y. Wang, H. Zhang, and D. Liu, “Adaptive dynamic programming: an introduction,” Computational Intelligence Magazine, IEEE, Vol.4, No.2, pp. 39-47, 2009.
-  F. L. Lewis and K. G. Vamvoudakis, “Reinforcement learning for partially observable dynamic processes: Adaptive dynamic programming using measured output data,” Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Trans. on., Vol.41, No.1, pp, 14-25, 2011.
-  D. R. Liu and Q. L. Wei, “Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems,” IEEE Trans. on., Neural Networks and Learning Systems, Vol.25, No.3, pp. 621-634, 2014.
-  X. Chen, P. H. Xie, Y. H. Xiong, et al., “Two-Phase Iteration for Value Function Approximation and Hyperparameter Optimization in Gaussian-Kernel-Based Adaptive Critic Design,” Mathematical Problems in Engineering, Vol.2015, 2015.
-  V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, et al., “Playing Atari with deep reinforcement learning,” Technical report. Deepmind Technologies, arXiv: 1312.5602 [cs.LG], 2013.
-  A. A. Jamshidi and B. P. Warren, “A recursive local polynomial approximation method using dirichlet clouds and radial basis functions,” SIAM J. on Scientific Computation, 2013.
-  X. N. Zhong, H. B. He, H. Zhang, and Z. Wang, “A neural network based online learning and control approach for Markov jump systems,” Neurocomputing, Vol.149, pp. 116-123, 2015.
-  X. Xu, L. Zuo, and Z. Huang, “Reinforcement learning algorithms with function approximation: recent advances and applications,” Information Sciences, Vol.261, pp. 1-31, 2014.
-  D. Ormoneit and S. Sen, “Kernel-Based Reinforcement Learning,” Machine Learning, Vol.49, No.2-3, pp. 161-178, 2002.
-  N. Jong and P. Stone, “Kernel-Based Models for Reinforcement Learning,” ICML Workshop on Kernel Machines and Reinforcement Learning, Pittsburgh, PA, USA, 2006.
-  C. E. Rasmussen and C. K. I. Williams, “Gaussian Processes for Machine Learning,” The MIT Press, 2006.
-  R. B. Gramacy and D. W. Apley, “Local Gaussian process approximation for large computer experiments,” J. of Computational and Graphical Statistics, Vol.24, No.2, pp. 561-578, 2015.
-  E. Snelson and Z. Ghahramani, “Local and global sparse Gaussian process approximations,” InAISTATS, Vol.11, pp. 524-531, 2007.