This paper focuses on policy selection and scheduling of sensors and attackers in cyber-physical systems (CPSs) with multiple sensors under denial-of-service (DoS) attacks. DoS attacks have caused enormous disruption to the regular operation of CPSs, and it is necessary to assess this damage. The state estimation of the CPSs plays a vital role in providing real-time information about their operational status and ensuring accurate prediction and assessment of their security. For a multi-sensor CPS, this paper is different from utilizing robust control methods to characterize the state of the system against DoS attacks, but rather positively analyzes the optimal policy selection of the sensors and the attackers through dynamic programming ideology. To optimize the strategies of both sides, game theory is employed as a means to study the dynamic interaction that occurs between the sensors and the attackers. During the policy iterative optimization process, the sensors and attackers dynamically learn and adjust strategies by incorporating reinforcement learning. In order to explore more state information, the restriction on the set of states is relaxed, i.e., the transfer of states is not limited compulsorily. Meanwhile, the complexity of the proposed algorithm is decreased by introducing a penalty in the reward function. Finally, simulation results show that the proposed algorithm can effectively optimize policy selection and scheduling for CPSs with multiple sensors.

1. [1] W. Duo, M. Zhou, and A. Abusorrah, “A survey of cyber attacks on cyber physical systems: Recent advances and challenges,” IEEE/CAA J. of Automatica Sinica, Vol.9, No.5, pp. 784-800, 2022. https://doi.org/10.1109/JAS.2022.105548
2. [2] H. Zhang, B. Liu, and H. Wu, “Smart grid cyber-physical attack and defense: A review,” IEEE Access, Vol.9, pp. 29641-29659, 2021. https://doi.org/10.1109/ACCESS.2021.3058628
3. [3] I. Ortega-Fernandez and F. Liberati, “A review of denial of service attack and mitigation in the smart grid using reinforcement learning,” Energies, Vol.16, No.2, Article No.635, 2023. https://doi.org/10.3390/en16020635
4. [4] G. Cabour, É. Ledoux, and S. Bassetto, “A work-centered approach for cyber-physical-social system design: Applications in aerospace industrial inspection,” arXiv:2101.05385, 2021. https://doi.org/10.48550/arXiv.2101.05385
5. [5] T. Zhang et al., “Data-driven based cruise control of connected and automated vehicles under cyber-physical system framework,” IEEE Trans. on Intelligent Transportation Systems, Vol.22, No.10, pp. 6307-6319, 2021. https://doi.org/10.1109/TITS.2020.2991223
6. [6] A. Razaque et al., “A mobility-aware human-centric cyber–physical system for efficient and secure smart healthcare,” IEEE Internet of Things J., Vol.9, No.22, pp. 22434-22452, 2022. https://doi.org/10.1109/JIOT.2021.3140090
7. [7] S. Kim, K.-J. Park, and C. Lu, “A survey on network security for cyber–physical systems: From threats to resilient design,” IEEE Communications Surveys & Tutorials, Vol.24, No.3, pp. 1534-1573, 2022. https://doi.org/10.1109/COMST.2022.3187531
8. [8] L. Cao et al., “A survey of network attacks on cyber-physical systems,” IEEE Access, Vol.8, pp. 44219-44227, 2020. https://doi.org/10.1109/ACCESS.2020.2977423
9. [9] Z. Yu et al., “A survey on cyber–physical systems security,” IEEE Internet of Things J., Vol.10, No.24, pp. 21670-21686, 2023. https://doi.org/10.1109/JIOT.2023.3289625
10. [10] Z. Han et al., “Secure state estimation for event-triggered cyber-physical systems against deception attacks,” J. of the Franklin Institute, Vol.359, No.18, pp. 11155-11185, 2022. https://doi.org/10.1016/j.jfranklin.2022.10.049
11. [11] J. Guo et al., “System identification with binary-valued observations under data tampering attacks,” IEEE Trans. on Automatic Control, Vol.66, No.8, pp. 3825-3832, 2021. https://doi.org/10.1109/TAC.2020.3029325
12. [12] J. Guo et al., “Identification of FIR systems with binary-valued observations against data tampering attacks,” IEEE Trans. on Systems, Man, and Cybernetics: Systems, Vol.53, No.9, pp. 5861-5873, 2023. https://doi.org/10.1109/TSMC.2023.3276352
13. [13] H. Yuan et al., “Resilient strategy design for cyber-physical system under active eavesdropping attack,” J. of the Franklin Institute, Vol.358, No.10, pp. 5281-5304, 2021. https://doi.org/10.1016/j.jfranklin.2021.04.049
14. [14] Q. Sun, K. Zhang, and Y. Shi, “Resilient model predictive control of cyber–physical systems under DoS attacks,” IEEE Trans. on Industrial Informatics, Vol.16, No.7, pp. 4920-4927, 2020. https://doi.org/10.1109/TII.2019.2963294
15. [15] T. Li et al., “Active security control approach against DoS attacks in cyber-physical systems,” IEEE Trans. on Automatic Control, Vol.66, No.9, pp. 4303-4310, 2021. https://doi.org/10.1109/TAC.2020.3032598
16. [16] Z. Li et al., “Event-based fixed-time secure cooperative control for nonlinear cyber-physical systems under denial-of-service attacks,” IEEE Trans. on Control of Network Systems, Vol.10, No.3, pp. 1520-1530, 2023. https://doi.org/10.1109/TCNS.2022.3233927
17. [17] A. Wang et al., “Secure adaptive event-triggered control for cyber–physical power systems under denial-of-service attacks,” IEEE Trans. on Cybernetics, Vol.54, No.3, pp. 1722-1733, 2024. https://doi.org/10.1109/TCYB.2023.3241179
18. [18] Z. Jin et al., “Secure state estimation of cyber-physical system under cyber attacks: Q-learning vs. SARSA,” Electronics, Vol.11, No.19, Article No.3161, 2022. https://doi.org/10.3390/electronics11193161
19. [19] Y. Hu, X. Zhang, and X. Lin, “Covariance intersection based event-triggered distributed state estimation under channel independent DoS attacks,” Neurocomputing, Vol.570, Article No.127139, 2024. https://doi.org/10.1016/j.neucom.2023.127139
20. [20] Y. Liu et al., “Distributed resilient state estimation for nonlinear systems against sensor attacks,” IEEE Trans. on Industrial Informatics, Vol.20, No.3, pp. 3639-3649, 2024. https://doi.org/10.1109/TII.2023.3308337
21. [21] Y. Liu and G.-H. Yang, “Event-triggered distributed state estimation for cyber-physical systems under DoS attacks,” IEEE Trans. on Cybernetics, Vol.52, No.5, pp. 3620-3631, 2022. https://doi.org/10.1109/TCYB.2020.3015507
22. [22] Y.-C. Sun and G.-H. Yang, “Event-triggered distributed state estimation for multiagent systems under DoS attacks,” IEEE Trans. on Cybernetics, Vol.52, No.7, pp. 6901-6910, 2022. https://doi.org/10.1109/TCYB.2020.3034456
23. [23] Y. Liu and G.-H. Yang, “Resilient event-triggered distributed state estimation for nonlinear systems against DoS attacks,” IEEE Trans. on Cybernetics, Vol.52, No.9, pp. 9076-9089, 2022.
24. [24] H. Ge et al., “A game theory based optimal allocation strategy for defense resources of smart grid under cyber-attack,” Information Sciences, Vol.652, Article No.119759, 2024. https://doi.org/10.1016/j.ins.2023.119759
25. [25] Y. Li et al., “Stochastic detection against deception attacks in CPS: Performance evaluation and game-theoretic analysis,” Automatica, Vol.144, Article No.110461, 2022. https://doi.org/10.1016/j.automatica.2022.110461
26. [26] Z. Jin et al., “Security state estimation for cyber-physical systems against DoS attacks via reinforcement learning and game theory,” Actuators, Vol.11, No.7, Article No.192, 2022. https://doi.org/10.3390/act11070192
27. [27] X.-F. Wang et al., “Robust distributed Nash equilibrium seeking for games under attacks and communication delays,” IEEE Trans. on Automatic Control, Vol.67, No.9, pp. 4892-4899, 2022. https://doi.org/10.1109/TAC.2022.3164984
28. [28] Z. Zhu et al., “Transfer learning in deep reinforcement learning: A survey,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.45, No.11, pp. 13344-13362, 2023. https://doi.org/10.1109/TPAMI.2023.3292075
29. [29] J. Hao et al., “Exploration in deep reinforcement learning: From single-agent to multiagent domain,” IEEE Trans. on Neural Networks and Learning Systems, 2023. https://doi.org/10.1109/TNNLS.2023.3236361
30. [30] C. Li et al., “Deep reinforcement learning in smart manufacturing: A review and prospects,” CIRP J. of Manufacturing Science and Technology, Vol.40, pp. 75-101, 2023. https://doi.org/10.1016/j.cirpj.2022.11.003
31. [31] T. T. Nguyen and V. J. Reddi, “Deep reinforcement learning for cyber security,” IEEE Trans. on Neural Networks and Learning Systems, Vol.34, No.8, pp. 3779-3795, 2023. https://doi.org/10.1109/TNNLS.2021.3121870
32. [32] A. Uprety and D. B. Rawat, “Reinforcement learning for IoT security: A comprehensive survey,” IEEE Internet of Things J., Vol.8, No.11, pp. 8693-8706, 2021. https://doi.org/10.1109/JIOT.2020.3040957
33. [33] Z. Jin et al., “Reinforcement learning-based policy selection of multi-sensor cyber physical systems under DoS attacks,” Proc. of the 8th Int. Workshop on Advanced Computational Intelligence and Intelligent Informatics (IWACIII 2023), Part 1, pp. 298-309, 2023. https://doi.org/10.1007/978-981-99-7590-7_24
34. [34] P. Dai et al., “Distributed reinforcement learning for cyber-physical system with multiple remote state estimation under DoS attacker,” IEEE Trans. on Network Science and Engineering, Vol.7, No.4, pp. 3212-3222, 2020. https://doi.org/10.1109/TNSE.2020.3018871
35. [35] J. F. Nash, Jr., “Equilibrium points in n-person games,” Proc. of the National Academy of Sciences, Vol.36, No.1, pp. 48-49, 1950. https://doi.org/10.1073/pnas.36.1.48
36. [36] S. Russell and P. Norvig, “Artificial Intelligence: A Modern Approach,” 3rd Edition, Pearson Education, Inc., 2010.