Estimation of Different Reward Functions Latent in Trajectory Data

Masaharu Saito; Sachiyo Arai

doi:10.20965/jaciii.2024.p0403

single-jc.php

« previous

JACIII Vol.28 No.2 pp. 403-412

doi: 10.20965/jaciii.2024.p0403

(2024)

Research Paper:

Views over last 60 days: 49

Estimation of Different Reward Functions Latent in Trajectory Data

Masaharu Saito and Sachiyo Arai

Department of Urban Environment Systems, Graduate School of Science and Engineering, Chiba University
1-33 Yayoi-cho, Inage-ku, Chiba 263-8522, Japan

Received:

June 4, 2023

Accepted:

January 30, 2024

Published:

March 20, 2024

Keywords:

inverse reinforcement learning, intention estimation, real-world application

Abstract

In recent years, inverse reinforcement learning has attracted attention as a method for estimating the intention of actions using the trajectories of various action-taking agents, including human flow data. In the context of reinforcement learning, “intention” refers to a reward function. Conventional inverse reinforcement learning assumes that all trajectories are generated from policies learned under a single reward function. However, it is natural to assume that people in a human flow act according to multiple policies. In this study, we introduce an expectation-maximization algorithm to inverse reinforcement learning, and propose a method to estimate different reward functions from the trajectories of human flow. The effectiveness of the proposed method was evaluated through a computer experiment based on human flow data collected from subjects around airport gates.

Cite this article as:

M. Saito and S. Arai, “Estimation of Different Reward Functions Latent in Trajectory Data,” J. Adv. Comput. Intell. Intell. Inform., Vol.28 No.2, pp. 403-412, 2024.

Data files:

References

[1] K. M. Kitani, B. D. Ziebart, J. A. Bagnell, and M. Hebert, “Activity forecasting,” European Conf. on Computer Vision, pp. 201-214, 2012. https://doi.org/10.1007/978-3-642-33765-9_15
[2] S. Yamaguchi, H. Naoki, M. Ikeda, Y. Tsukada, S. Nakano, I. Mori, and S. Ishii, “Identification of animal behavioral strategies by inverse reinforcement learning,” PLoS Computational Biology, Vol.14, No.5, Article No.e1006122, 2018. https://doi.org/10.1371/journal.pcbi.1006122
[3] T. Hirakawa, T. Yamashita, T. Tamaki, H. Fujiyoshi, Y. Umezu, I. Takeuchi, S. Matsumoto, and K. Yoda, “Can AI predict animal movements? Filling gaps in animal trajectories using inverse reinforcement learning,” Ecosphere, Vol.9, No.10, Article No.e02447, 2018. https://doi.org/10.1002/ecs2.2447
[4] M. Babes, V. Marivate, K. Subramanian, and M. L. Littman, “Apprenticeship learning about multiple intentions,” Proc. of the 28th Int. Conf. on Machine Learning (ICML-11), pp. 897-904, 2011.
[5] S. Russell, “Learning agents for uncertain environments,” Proc. of the 11th Annual Conf. on Computational Learning Theory, pp. 101-103, 1998. https://doi.org/10.1145/279943.279964
[6] A. Y. Ng and S. J. Russell, “Algorithms for inverse reinforcement learning,” Proc. of the 17th Int. Conf. on Machine Learning (ICML 2000), Vol.1, pp. 663-670, 2000.
[7] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey, “Maximum entropy inverse reinforcement learning,” Proc. of the 23rd AAAI Conf. on Artificial Intelligence, Vol.8, pp. 1433-1438, Chicago, IL, USA, 2008.
[8] K. Muelling, A. Boularias, B. Mohler, B. Schölkopf, and J. Peters, “Learning strategies in table tennis using inverse reinforcement learning,” Biological Cybernetics, Vol.108, No.5, pp. 603-619, 2014. https://doi.org/10.1007/s00422-014-0599-1
[9] A. Boularias, J. Kober, and J. Peters, “Relative entropy inverse reinforcement learning,” Proc. of the 14th Int. Conf. on Artificial Intelligence and Statistics, pp. 182-189, 2011.
[10] K. Dvijotham and E. Todorov, “Inverse optimal control with linearly-solvable MDPs,” Proc. of the 27th Int. Conf. on Machine Learning (ICML-10), pp. 335-342, 2010.
[11] P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” Proc. of the 21st Int. Conf. on Machine Learning, 2004. https://doi.org/10.1145/1015330.1015430
[12] D. Ramachandran and E. Amir, “Bayesian Inverse Reinforcement Learning,” Proc. of the 20th Int. Joint Conf. on Artifical Intelligence (IJCAI 2007), Vol.7, pp. 2586-2591, 2007.
[13] M. Wulfmeier, P. Ondruska, and I. Posner, “Maximum entropy deep inverse reinforcement learning,” arXiv:1507.04888, 2015. https://doi.org/10.48550/arXiv.1507.04888
[14] J. Choi and K.-E. Kim, “Map inference for bayesian inverse reinforcement learning,” Advances in Neural Information Processing Systems, Vol.24, 2011.
[15] E. Todorov, “Linearly-solvable Markov decision problems,” Advances in Neural Information Processing Systems, Vol.19, 2006.
[16] E. Uchibe, “Model-free deep inverse reinforcement learning by logistic regression,” Neural Processing Letters, Vol.47, No.3, pp. 891-905, 2018. https://doi.org/10.1007/s11063-017-9702-7
[17] J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adversarial inverse reinforcement learning,” arXiv:1710.11248, 2017. https://doi.org/10.48550/arXiv.1710.11248
[18] J. Choi and K.-E. Kim, “Nonparametric Bayesian inverse reinforcement learning for multiple reward functions,” Advances in Neural Information Processing Systems, Vol.25, 2012.
[19] “Haneda Airport Map.” https://tokyo-haneda.com/en/floor/index.html [Accessed June 4, 2023]
[20] A. Mimata and S. Arai, “Detection of Suspicious Behaviors based on Intention Inferred from Human Trajectories Using Inverse Reinforcement Learning,” The 34th Annual Conf. of the Japanese Society for Artificial Intelligence, 2J6GS203, 2020 (in Japanese). https://doi.org/10.11517/pjsai.JSAI2020.0_2J6GS203

This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.

[1] [1] K. M. Kitani, B. D. Ziebart, J. A. Bagnell, and M. Hebert, “Activity forecasting,” European Conf. on Computer Vision, pp. 201-214, 2012. https://doi.org/10.1007/978-3-642-33765-9_15

[2] [2] S. Yamaguchi, H. Naoki, M. Ikeda, Y. Tsukada, S. Nakano, I. Mori, and S. Ishii, “Identification of animal behavioral strategies by inverse reinforcement learning,” PLoS Computational Biology, Vol.14, No.5, Article No.e1006122, 2018. https://doi.org/10.1371/journal.pcbi.1006122

[3] [3] T. Hirakawa, T. Yamashita, T. Tamaki, H. Fujiyoshi, Y. Umezu, I. Takeuchi, S. Matsumoto, and K. Yoda, “Can AI predict animal movements? Filling gaps in animal trajectories using inverse reinforcement learning,” Ecosphere, Vol.9, No.10, Article No.e02447, 2018. https://doi.org/10.1002/ecs2.2447

[4] [4] M. Babes, V. Marivate, K. Subramanian, and M. L. Littman, “Apprenticeship learning about multiple intentions,” Proc. of the 28th Int. Conf. on Machine Learning (ICML-11), pp. 897-904, 2011.

[5] [5] S. Russell, “Learning agents for uncertain environments,” Proc. of the 11th Annual Conf. on Computational Learning Theory, pp. 101-103, 1998. https://doi.org/10.1145/279943.279964

[6] [6] A. Y. Ng and S. J. Russell, “Algorithms for inverse reinforcement learning,” Proc. of the 17th Int. Conf. on Machine Learning (ICML 2000), Vol.1, pp. 663-670, 2000.

[7] [7] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey, “Maximum entropy inverse reinforcement learning,” Proc. of the 23rd AAAI Conf. on Artificial Intelligence, Vol.8, pp. 1433-1438, Chicago, IL, USA, 2008.

[8] [8] K. Muelling, A. Boularias, B. Mohler, B. Schölkopf, and J. Peters, “Learning strategies in table tennis using inverse reinforcement learning,” Biological Cybernetics, Vol.108, No.5, pp. 603-619, 2014. https://doi.org/10.1007/s00422-014-0599-1

[9] [9] A. Boularias, J. Kober, and J. Peters, “Relative entropy inverse reinforcement learning,” Proc. of the 14th Int. Conf. on Artificial Intelligence and Statistics, pp. 182-189, 2011.

[10] [10] K. Dvijotham and E. Todorov, “Inverse optimal control with linearly-solvable MDPs,” Proc. of the 27th Int. Conf. on Machine Learning (ICML-10), pp. 335-342, 2010.

[11] [11] P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” Proc. of the 21st Int. Conf. on Machine Learning, 2004. https://doi.org/10.1145/1015330.1015430

[12] [12] D. Ramachandran and E. Amir, “Bayesian Inverse Reinforcement Learning,” Proc. of the 20th Int. Joint Conf. on Artifical Intelligence (IJCAI 2007), Vol.7, pp. 2586-2591, 2007.

[13] [13] M. Wulfmeier, P. Ondruska, and I. Posner, “Maximum entropy deep inverse reinforcement learning,” arXiv:1507.04888, 2015. https://doi.org/10.48550/arXiv.1507.04888

[14] [14] J. Choi and K.-E. Kim, “Map inference for bayesian inverse reinforcement learning,” Advances in Neural Information Processing Systems, Vol.24, 2011.

[15] [15] E. Todorov, “Linearly-solvable Markov decision problems,” Advances in Neural Information Processing Systems, Vol.19, 2006.

[16] [16] E. Uchibe, “Model-free deep inverse reinforcement learning by logistic regression,” Neural Processing Letters, Vol.47, No.3, pp. 891-905, 2018. https://doi.org/10.1007/s11063-017-9702-7

[17] [17] J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adversarial inverse reinforcement learning,” arXiv:1710.11248, 2017. https://doi.org/10.48550/arXiv.1710.11248

[18] [18] J. Choi and K.-E. Kim, “Nonparametric Bayesian inverse reinforcement learning for multiple reward functions,” Advances in Neural Information Processing Systems, Vol.25, 2012.

[19] [19] “Haneda Airport Map.” https://tokyo-haneda.com/en/floor/index.html [Accessed June 4, 2023]

[20] [20] A. Mimata and S. Arai, “Detection of Suspicious Behaviors based on Intention Inferred from Human Trajectories Using Inverse Reinforcement Learning,” The 34th Annual Conf. of the Japanese Society for Artificial Intelligence, 2J6GS203, 2020 (in Japanese). https://doi.org/10.11517/pjsai.JSAI2020.0_2J6GS203