JRM Vol.35 No.2 pp. 501-509
doi: 10.20965/jrm.2023.p0501


Vision-Based Robot Arm Control Interface for Retrieving Objects from the Floor

Laijun Yang*, Ryota Sakamoto** ORCID Icon, Norihiko Kato*, and Ken’ichi Yano*

*Graduate School of Engineering, Mie University
1577 Kurimamachiya-cho, Tsu, Mie 514-8507, Japan

**Mie University Hospital
2-174 Edobashi, Tsu, Mie 514-0001, Japan

September 1, 2022
January 15, 2023
April 20, 2023
welfare robot arm, vision-based interface, remote control, cervical cord injury, eye in hand system

Approximately half of the patients with spinal cord injuries in Japan have a cervical spinal cord injury. Owing to the trunk dysfunction, patients with high-level spinal cord injuries have particular difficulty when searching for or picking up objects from the floor. Recently, welfare robot arms have been developed to help such individuals increase self-reliance. In this study, we propose an operating system that includes an eye-in-hand system with a touchscreen interface for grasping objects from the floor and delivering them to the individual. In the proposed method, the visual information of the target object is shown on a touchscreen interface. The patient specifies the target position for the robot arm by drawing a line on the target object on the interface. We conducted an experiment to compare the proposed interface with an on-screen joystick to demonstrate the proposed system’s efficiency and its ability to reduce physical burden. The results show that the proposed method is both quicker to use and effectively reduces the physical burden on the user compared to the conventional method.

The usage of proposed interface

The usage of proposed interface

Cite this article as:
L. Yang, R. Sakamoto, N. Kato, and K. Yano, “Vision-Based Robot Arm Control Interface for Retrieving Objects from the Floor,” J. Robot. Mechatron., Vol.35 No.2, pp. 501-509, 2023.
Data files:
  1. [1] Cabinet Office, “Annual Report on Government Measures for Persons with Disabilities,” p. 213, 2021 (in Japanese).
  2. [2] Cabinet Office, “Annual Report on Government Measures for Persons with Disabilities,” p. 1, 2011 (in Japanese).
  3. [3] H. Sakai, T. Ueta, T. Maeda, and K. Shiba, “Survey for Spinal Cord Injuries in Fukuoka,” Sogo Rehabilitation, Vol.36, No.10, pp. 969-972, 2008 (in Japanese).
  4. [4] K. Shibazaki, “National Spinal Cord Injury Registration Statistics 2002.1–12,” The J. of the Japan Medical Society of Spinal Cord Lesion, Vol.18, pp. 271-274, 2005 (in Japanese).
  5. [5] F. Tajima, Y. Kamijo, and Y. Nishimura, “Basis of Rehabilitation in Patients with Spinal Cord Injury,” Spinal Surgery, Vol.30, No.1, pp. 58-67, 2016 (in Japanese).
  6. [6] K. Nagata, Y. Wakita, N. Yamanobe, and N. Ando, “Development of RT-Component for Assistive Robotic Arm and Man-Machine Interface Using OpenRTM-aist,” Trans. of the Japan Society of Mechanical Engineers, Series C, Vol.76, No.766, pp. 1419-1426, 2010 (in Japanese).
  7. [7] V. Maheu, P. S. Archambault, J. Frappier, and F. Routhier, “Evaluation of the JACO Robotic Arm: Clinico-Economic Study for Powered Wheelchair Users with Upper-Extremity Disabilities,” 2011 IEEE Int. Conf. on Rehabilitation Robotics, 2011.
  8. [8] W. Yoon, Y. Wakita, and N. Yamanobe, “Self-Reliance Support by RAPUDA, a Robot Arm that Supports the Lives of People with Disabilities in the Upper Limbs,” J. of the Society of Biomechanisms, Vol.37, No.2, pp. 100-104, 2013 (in Japanese).
  9. [9] J. Vogel et al., “An Ecosystem for Heterogeneous Robotic Assistants in Caregiving: Core Functionalities and Use Cases,” IEEE Robotics & Automation Magazine, Vol.28, No.3, pp. 12-28, 2021.
  10. [10] J. Yamaguchi et al., “Clinical Evaluation of Assistive Robotic Arm: Efficacy Evaluation Using Psychological Estimation Scale,” Proc. of the 59th Annual Conf. of JSSD, 2012 (in Japanese).
  11. [11] C.-S. Chung et al., “Performance Evaluation of a Mobile Touchscreen Interface for Assistive Robotic Manipulators: A Pilot Study,” Top Spinal Cord Injury Rehabilitation, Vol.23, No.2, pp. 131-139, 2017.
  12. [12] National Rehabilitation Center for Persons with Disabilities, “Electric Wheelchair Compatibility & Operation Practice Manual,” p. 8, 2006 (in Japanese).
  13. [13] T. Matsumaru, A. I. Septiana, and K. Horiuchi, “Three-Dimensional Aerial Image Interface, 3DAII,” J. Robot. Mechatron., Vol.31, No.5, pp. 657-670, 2019.
  14. [14] A. Namiki, “Multi-Eye Vision Robot,” J. of the Robotics Society of Japan, Vol.40, No.5, pp. 387-392, 2022 (in Japanese).
  15. [15] S. Hashimoto, A. Ishida, M. Imani, and T. Igarashi, “TouchMe: An Augmented Reality Interface for Remote Robot Control,” J. Robot. Mechatron., Vol.25, No.3, pp. 529-537, 2013.
  16. [16] B. J. F. Driessen, T. K. T. Kate, F. Liefhebber, A. H. G. Versluis, and J. A. v. Woerden, “Collaborative Control of the Manus Manipulator,” Universal Access in the Information Society, Vol.4, No.2, pp. 165-173, 2005.
  17. [17] P. Hebert et al., “Supervised Remote Robot with Guided Autonomy and Teleoperation (SURROGATE): A Framework for Whole-Body Manipulation,” 2015 IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 5509-5516, 2015.
  18. [18] J. Soares, A. Vale, and R. Ventura, “A Multi-Purpose Rescue Vehicle and a Human-Robot Interface Architecture for Remote Assistance in ITER,” Fusion Engineering and Design, Vols.98-99, pp. 1656-1659, 2015.
  19. [19] K. M. Tsui, D.-J. Kim, A. Behal, D. Kontak, and H. A. Yanco, “‘I want that’: Human-in-the-Loop Control of a Wheelchair-Mounted Robotic Arm,” Applied Bionics and Biomechanics, Vol.8, No.1, pp. 127-147, 2011.
  20. [20] H. Fujii, K. Sugimoto, A. Yamashita, and H. Asama, “Online Half Diminished-Reality Imaging Using Multiple RGB-D Sensors for Remote Control Robot,” J. of the Japan Society for Precision Engineering, Vol.81, No.12, pp. 1185-1192, 2015 (in Japanese).
  21. [21] H. Jiang, T. Zhang, J. P. Wachs, and B. S. Duerstock, “Enhanced Control of a Wheelchair-Mounted Robotic Manipulator Using 3-D Vision and Multimodal Interaction,” Computer Vision and Image Understanding, Vol.149, pp. 21-31, 2016.
  22. [22] D. Ding, B. Styler, C.-S. Chung, and A. Hourtet, “Development of a Vision-Guided Shared-Control System for Assistive Robotic Manipulators,” Sensors, Vol.22, No.12, Article No.4351, 2022.
  23. [23] H. Tanaka et al., “Visual Marker System for Autonomous Object Handling by Assistive Robotic Arm,” J. Robot. Mechatron., Vol.23, No.4, pp. 484-493, 2011.
  24. [24] Y. Miyawaki et al., “Interface for Operating Electric Wheelchair Without Gazing Operation Screen,” The Proc. of 68th Conf. of Tokai Branch, Article No.519, 2019 (in Japanese).

*This site is desgined based on HTML5 and CSS3 for modern browsers, e.g. Chrome, Firefox, Safari, Edge, Opera.

Last updated on Apr. 22, 2024