Top > JRM > Transparency in Human-Machine Mutual Action

single-rb.php

JRM Vol.33 No.5 pp. 987-1003
doi: 10.20965/jrm.2021.p0987
(2021)

Review:

Views over last 60 days: 948

Transparency in Human-Machine Mutual Action

Hiroto Saito*1, Arata Horie*2, Azumi Maekawa*1, Seito Matsubara*3, Sohei Wakisaka*1, Zendai Kashino*1, Shunichi Kasahara*1,*4, and Masahiko Inami*1

*1Information Somatics Lab, Research Center for Advanced Science and Technology, The University of Tokyo
4-6-1 Komaba, Meguro-ku, Tokyo 153-8505, Japan

*2Graduate School of Engineering, The University of Tokyo
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan

*3Graduate School of Information Science and Technology, The University of Tokyo
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan

*4Sony Computer Science Laboratories, Inc.
3-14-13 Higashigotanda, Shinagawa-ku, Tokyo 141-0022, Japan

Received:
April 4, 2021
Accepted:
June 14, 2021
Published:
October 20, 2021
Creative Commons License
Keywords:
human-computer integration, transparency, human-machine mutual action, human augmentation
Abstract

Recent advances in human-computer integration (HInt) have focused on the development of human-machine systems, where both human and machine autonomously act upon each other. However, a key challenge in designing such systems is augmenting the user’s physical abilities while maintaining their sense of self-attribution. This challenge is particularly prevalent when both human and machine are capable of acting upon each other, thereby creating a human-machine mutual action (HMMA) system. To address this challenge, we present a design framework that is based on the concept of transparency. We define transparency in HInt as the degree to which users can self-attribute an experience when machines intervene in the users’ action. Using this framework, we form a set of design guidelines and an approach for designing HMMA systems. By using transparency as our focus, we aim to provide a design approach for not only achieving human-machine fusion into a single agent, but also controlling the degrees of fusion at will. This study also highlights the effectiveness of our design approach through an analysis of existing studies that developed HMMA systems. Further development of our design approach is discussed, and future prospects for HInt and HMMA system designs are presented.

Transparency-based design framework for HInt

Transparency-based design framework for HInt

Cite this article as:
H. Saito, A. Horie, A. Maekawa, S. Matsubara, S. Wakisaka, Z. Kashino, S. Kasahara, and M. Inami, “Transparency in Human-Machine Mutual Action,” J. Robot. Mechatron., Vol.33 No.5, pp. 987-1003, 2021.
Data files:
References
  1. [1] F. F. Mueller, P. Lopes, P. Strohmeier, W. Ju, C. Seim, M. Weigel, S. Nanayakkara, M. Obrist, Z. Li, J. Delfa, J. Nishida, E. M. Gerber, D. Svanaes, J. Grudin, S. Greuter, K. Kunze, T. Erickson, S. Greenspan, M. Inami, J. Marshall, H. Reiterer, K. Wolf, J. Meyer, T. Schiphorst, D. Wang, and P. Maes, “Next Steps for Human-Computer Integration,” Proc. of the 2020 CHI Conf. on Human Factors in Computing Systems (CHI’20), pp. 1-15, 2020.
  2. [2] G. Gourmelen, A. Verhulst, B. Navarro, T. Sasaki, G. Gowrishankar, and M. Inami, “Co-Limbs: An Intuitive Collaborative Control for Wearable Robotic Arms,” SIGGRAPH Asia 2019 Emerging Technologies (SA’19), pp. 9-10, 2019.
  3. [3] Y. Hasegawa, Y. Mikami, K. Watanabe, and Y. Sankai, “Five-fingered assistive hand with mechanical compliance of human finger,” Proc. of 2008 IEEE Int. Conf. on Robotics and Automation, pp. 718-724, 2008.
  4. [4] P. Heo, G. M. Gu, S.-J. Lee, K. Rhee, and J. Kim, “Current Hand Exoskeleton Technologies for Rehabilitation and Assistive engineering,” Int. J. of Precision Engineering and Manufacturing, Vol.13, Issue 5, pp. 807-824, 2012.
  5. [5] A. Maekawa, S. Takahashi, M. Y. Saraiji, S. Wakisaka, H. Iwata, and M. Inami, “Naviarm: Augmenting the Learning of Motor Skills Using a Backpack-Type Robotic Arm System,” Proc. of the 10th Augmented Human Int. Conf. 2019 (AH 2019), 38, 2019.
  6. [6] P. Lopes, A. Ion, W. Mueller, D. Hoffmann, P. Jonell, and P. Baudisch, “Proprioceptive Interaction,” Proc. of the 33rd Annual ACM Conf. on Human Factors in Computing Systems (CHI’15), pp. 939-948, 2015.
  7. [7] P. Lopes, D. Yüksel, F. Guimbretière, and P. Baudisch, “Muscle-plotter: An Interactive System based on Electrical Muscle Stimulation that Produces Spatial Output,” Proc. of the 29th Annual Symp. on User Interface Software and Technology (UIST’16), pp. 207-217, 2016.
  8. [8] S. Kasahara, K. Konno, R. Owaki, T. Nishi, A. Takeshita, T. Ito, S. Kasuga, and J. Ushiba, “Malleable Embodiment: Changing Sense of Embodiment by Spatial-Temporal Deformation of Virtual Human Body,” Proc. of the 2017 CHI Conf. on Human Factors in Computing Systems (CHI’17), pp. 6438-6448, 2017.
  9. [9] M. Gonzalez-Franco, B. Cohn, E. Ofek, D. Burin, and A. Maselli, “The Self-Avatar Follower Effect in Virtual Reality,” Proc. of 2020 IEEE Conf. on Virtual Reality and 3D User Interfaces (VR 2020), pp. 18-25, 2020.
  10. [10] W. Wen, Y. Kuroki, and H. Asama, “The Sense of Agency in Driving Automation,” Front. Psychol., Vol.10, 2691, 2019.
  11. [11] B. A. Nardi (Ed.), “Context and consciousness: Activity theory and human-computer interaction,” The MIT Press, 1996.
  12. [12] D. A. Lawrence, “Stability and transparency in bilateral teleoperation,” IEEE Trans. on Robotics and Automation, Vol.9, No.5, pp. 624-637, 1993.
  13. [13] B. Hannaford, “A design framework for teleoperators with kinesthetic feedback,” IEEE Trans. on Robotics and Automation, Vol.5, No.4, pp. 426-434, 1989.
  14. [14] J. E. Colgate, “Robust impedance shaping telemanipulation,” Vol.9, No.4, pp. 374-384, 1993.
  15. [15] T. H. Massie, J. K. Salisbury et al., “The phantom haptic interface: A device for probing virtual objects,” Proc. of the ASME Winter Annual Meeting, Symp. on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Vol.55, pp. 295-300, 1994.
  16. [16] G. Millet, S. Haliyo, S. Regnier, and V. Hayward, “The ultimate haptic device: first step,” Proc. of the 3rd Joint EuroHaptics Conf. and Symp. on Haptic Interfaces for Virtual Environment and Teleoperator Systems (World Haptics 2009), pp. 273-278, 2009.
  17. [17] A. Mohand-Ousaid, G. Millet, S. Régnier, S. Haliyo, and V. Hayward, “Haptic interface transparency achieved through viscous coupling,” The Int. J. of Robotics Research, Vol.31, Issue 3, pp. 319-329, 2012.
  18. [18] H. Hoshino, R. Hirata, T. Maeda, and S. Tachi, “A construction method of virtual haptic space,” Proc. of the 4th Int. Conf. on Artificail Reality and Tele-Existence (ICAT’94), pp. 131-138, 1994.
  19. [19] W. A. McNeely, “Robotic graphics: a new approach to force feedback for virtual reality,” Proc. of IEEE Virtual Reality Annual Int. Symp., pp. 336-341, 1993.
  20. [20] A. Zenner and A. Kruger, “Shifty: A Weight-Shifting Dynamic Passive Haptic Proxy to Enhance Object Perception in Virtual Reality,” Proc. of IEEE Trans. Vis. Comput. Graph., Vol.23, No.4, pp. 1285-1294, 2017.
  21. [21] P. Haggard, “Conscious intention and motor cognition,” Trends Cogn. Sci., Vol.9, Issue 6, pp. 290-295, 2005.
  22. [22] P. Haggard, “Sense of agency in the human brain,” Nat. Rev. Neurosci., Vol.18, Issue 4, pp. 196-207, 2017.
  23. [23] M. Tsakiris, S. Schütz-Bosbach, and S. Gallagher, “On agency and body-ownership: Phenomenological and neurocognitive reflections,” Conscious. Cogn., Vol.16, Issue 3, pp. 645-660, 2007.
  24. [24] D. M. Wolpert and Z. Ghahramani, “Computational principles of movement neuroscience,” Nat. Neurosci., Vol.3, Issue 11, pp. 1212-1217, 2000.
  25. [25] M. Wolpert, “Computational approaches to motor control,” Trends in Cognitive Sciences, Vol.1, Issue 6, pp. 209-216, 1997.
  26. [26] D. M. Wolpert, Z. Ghahramani, and M. I. Jordan, “An internal model for sensorimotor integration,” Science, Vol.269, Issue 5232, pp. 1880-1882, 1995.
  27. [27] M. Kawato, “Internal models for motor control and trajectory planning,” Curr. Opin. Neurobiol., Vol.9, Issue 6, pp. 718-727, 1999.
  28. [28] T. Asai, “Know thy agency in predictive coding: Meta-monitoring over forward modeling,” Conscious. Cogn., Vol.51, pp. 82-99, 2017.
  29. [29] R. Ohata, T. Asai, H. Kadota, H. Shigemasu, K. Ogawa, and H. Imamizu, “Sense of Agency Beyond Sensorimotor Process: Decoding Self-Other Action Attribution in the Human Brain,” Cereb. Cortex, Vol.30, Issue 7, pp. 4076-4091, 2020.
  30. [30] H. Imamizu, S. Miyauchi, T. Tamada, Y. Sasaki, R. Takino, B. Pütz, T. Yoshioka, and M. Kawato, “Human cerebellar activity reflecting an acquired internal model of a new tool,” Nature, Vol.403, 6766, pp. 192-195, 2000.
  31. [31] B. Mehta and S. Schaal, “Forward models in visuomotor control,” J. Neurophysiol., Vol.88, Issue 2, pp. 942-953, 2002.
  32. [32] J. Kluzik, J. Diedrichsen, R. Shadmehr, and A. J. Bastian, “Reach adaptation: what determines whether we learn an internal model of the tool or adapt the model of our arm?,” J. of Neurophysiology, Vol.100, Issue 3, pp. 1455-1464, 2008.
  33. [33] S. J. Blakemore, D. M. Wolpert, and C. D. Frith, “Abnormalities in the awareness of action,” Trends Cogn. Sci., Vol.6, Issue 6, pp. 237-242, 2002.
  34. [34] C. D. Frith, S. Blakemore, and D. M. Wolpert, “Explaining the symptoms of schizophrenia: abnormalities in the awareness of action,” Brain Res. Rev., Vol.31, Issues 2-3, pp. 357-363, 2000.
  35. [35] M. Synofzik, G. Vosgerau, and A. Newen, “Beyond the comparator model: A multifactorial two-step account of agency,” Conscious. Cogn., Vol.17, Issue 1, pp. 219-239, 2008.
  36. [36] M. Synofzik, G. Vosgerau, and M. Voss, “The experience of agency: an interplay between prediction and postdiction,” Front. Psychol., Vol.4, 127, 2013.
  37. [37] P. Haggard and M. Tsakiris, “The Experience of Agency: Feelings, Judgments, and Responsibility,” Curr. Dir. Psychol. Sci., Vol.18, Issue 4, pp. 242-246, 2009.
  38. [38] J. W. Moore, D. Middleton, P. Haggard, and P. C. Fletcher, “Exploring implicit and explicit aspects of sense of agency,” Conscious. Cogn., Vol.21, Issue 4, pp. 1748-1753, 2012.
  39. [39] S. Gallagher, “Multiple aspects in the sense of agency,” New Ideas Psychol., Vol.30, Issue 1, pp. 15-31, 2012.
  40. [40] D. M. Wolpert and J. R. Flanagan, “Motor prediction,” Curr. Biol., Vol.11, Issue 18, pp. R729-R732, 2001.
  41. [41] S. J. Blakemore, D. M. Wolpert, and C. D. Frith, “Central cancellation of self-produced tickle sensation,” Nat. Neurosci., Vol.1, No.7, pp. 635-640, 1998.
  42. [42] S. J. Blakemore, C. D. Frith, and D. M. Wolpert, “Spatio-temporal prediction modulates the perception of self-produced stimuli,” J. Cogn. Neurosci., Vol.11, Issue 5, pp. 551-559, 1999.
  43. [43] S. J. Blakemore, D. Wolpert, and C. Frith, “Why can’t you tickle yourself?,” Neuroreport, Vol.11, Issue 11, pp. R11-R16, 2000.
  44. [44] M. Tsakiris, P. Haggard, N. Franck, N. Mainy, and A. Sirigu, “A specific role for efferent information in self-recognition,” Cognition, Vol.96, Issue 3, pp. 215-231, 2005.
  45. [45] S. Gallagher, “Philosophical conceptions of the self: implications for cognitive science,” Trends Cogn. Sci., Vol.4, Issue 1, pp. 14-21, 2000.
  46. [46] D. M. Wegner, “The mind’s best trick: how we experience conscious will,” Trends Cogn. Sci., Vol.7, No.2, pp. 65-69, 2003.
  47. [47] D. M. Wegner, B. Sparrow, and L. Winerman, “Vicarious agency: experiencing control over the movements of others,” J. Pers. Soc. Psychol., Vol.86, No.6, pp. 838-848, 2004.
  48. [48] S. Kasahara, K. Takada, and J. Nishida, “Preserving Agency During Electrical Muscle Stimulation Training Speeds up Reaction Time Directly After Removing EMS,” Proc. of 2021 CHI Conf. on Human Factors in Computing Systems (CHI’21), 194, 2021.
  49. [49] K. Matsumiya, “Awareness of voluntary action, rather than body ownership, improves motor control,” Sci. Rep., Vol.11, No.1, 418, 2021.
  50. [50] A. Bandura, “Exercise of Human Agency Through Collective Efficacy,” Curr. Dir. Psychol. Sci., Vol.9, Issue 3, pp. 75-78, 2000.
  51. [51] D. Coyle, J. Moore, P. O. Kristensson, P. Fletcher, and A. Blackwell, “I did that! Measuring users’ experience of agency in their own actions,” Proc. of the SIGCHI Conf. on Human Factors in Computing Systems (CHI’12), pp. 2025-2034, 2012.
  52. [52] T. Asai, “Feedback control of one’s own action: Self-other sensory attribution in motor control,” Conscious. Cogn., Vol.38, pp. 118-129, 2015.
  53. [53] T. Asai, “Self is “other,” other is “self”: poor self-other discriminability explains schizotypal twisted agency judgment,” Psychiatry Res., Vol.246, pp. 593-600, 2016.
  54. [54] N. David, S. Skoruppa, A. Gulberti, J. Schultz, and A. K. Engel, “The Sense of Agency Is More Sensitive to Manipulations of Outcome than Movement-Related Feedback Irrespective of Sensory Modality,” PLoS ONE, Vol.11, No.8, e0161156, 2016.
  55. [55] F. Steinicke, G. Bruder, J. Jerald, H. Frenz, and M. Lappe, “Estimation of detection thresholds for redirected walking techniques,” IEEE Trans. Vis. Comput. Graph., Vol.16, No.1, pp. 17-27, 2010.
  56. [56] R. Arakawa and H. Yakura, “Mindless Attractor: A False-Positive Resistant Intervention for Drawing Attention Using Auditory Perturbation,” Proc. of 2021 CHI Conf. on Human Factors in Computing Systems (CHI’21), 99, 2021.
  57. [57] S. Kasahara, J. Nishida, and P. Lopes, “Preemptive Action: Accelerating Human Reaction using Electrical Muscle Stimulation Without Compromising Agency,” Proc. of 2019 CHI Conf. on Human Factors in Computing Systems (CHI’19), 643, 2019.
  58. [58] S. Imaizumi and T. Asai, “Dissociation of agency and body ownership following visuomotor temporal recalibration,” Front. Integr. Neurosci., Vol.9, 35, 2015.
  59. [59] S. Matsubara, S. Wakisaka, K. Aoyama, K. Seaborn, A. Hiyama, and M. Inami, “Perceptual simultaneity and its modulation during EMG-triggered motion induction with electrical muscle stimulation,” PLoS ONE, Vol.15, No.8, e0236497, 2020.
  60. [60] T. Honda, N. Hagura, T. Yoshioka, and H. Imamizu, “Imposed visual feedback delay of an action changes mass perception based on the sensory prediction error,” Front. Psychol., Vol.4, 760, 2013.
  61. [61] H. Galvan Debarba, R. Boulic, R. Salomon, O. Blanke, and B. Herbelin, “Self-attribution of distorted reaching movements in immersive virtual reality,” Comput. Graph., Vol.76, pp. 142-152, 2018.
  62. [62] H. Imamizu, T. Kuroda, S. Miyauchi, T. Yoshioka, and M. Kawato, “Modular organization of internal models of tools in the human cerebellum,” Proc. of the National Academy of Sciences, Vol.100, No.9, pp. 5461-5466, 2003.
  63. [63] J. Kluzik, J. Diedrichsen, R. Shadmehr, and A. J. Bastian, “Reach adaptation: what determines whether we learn an internal model of the tool or adapt the model of our arm?,” J. of Neurophysiology, Vol.100, Issue 3, pp. 1455-1464, 2008.
  64. [64] A. Maekawa, S. Matsubara, A. Hiyama, and M. Inami, “PickHits: hitting experience generation with throwing motion via a handheld mechanical device,” Proc. of ACM SIGGRAPH 2019 Emerging Technologies (SIGGRAPH’19), 20, 2019.
  65. [65] A. Maekawa, S. Matsubara, S. Wakisaka, D. Uriu, A. Hiyama, and M. Inami, “Dynamic Motor Skill Synthesis with Human-Machine Mutual Actuation,” Proc. of 2020 CHI Conf. on Human Factors in Computing Systems (CHI’20), 576, 2020.
  66. [66] A. Maekawa, S. Kasahara, H. Saito, D. Uriu, G. Ganesh, and M. Inami, “The Tight Game: Implicit Force Intervention in Inter-personal Physical Interactions on Playing Tug of War,” Proc. of ACM SIGGRAPH 2020 Emerging Technologies (SIGGRAPH’20), 10, 2020.
  67. [67] C. M. Harris and D. M. Wolpert, “Signal-dependent noise determines motor planning,” Nature, Vol.394, Issue 6695, pp. 780-784, 1998.
  68. [68] R. Lammfromm and D. Gopher, “Transfer of skill from a virtual reality trainer to real juggling,” BIO Web of Conferences, Vol.1, 00054, 2011.
  69. [69] K. Friston, “The free-energy principle: a rough guide to the brain?,” Trends Cogn. Sci., Vol.13, Issue 7, pp. 293-301, 2009.
  70. [70] K. J. Friston, J. Daunizeau, J. Kilner, and S. J. Kiebel, “Action and behavior: a free-energy formulation,” Biol. Cybern., Vol.102, Issue 3, pp. 227-260, 2010.

Creative Commons License  This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.

*This site is desgined based on HTML5 and CSS3 for modern browsers, e.g. Chrome, Firefox, Safari, Edge, Opera.

Last updated on Apr. 19, 2024