JRM Vol.14 No.5 pp. 479-489
doi: 10.20965/jrm.2002.p0479


Real-time Auditory and Visual Multiple-speaker Tracking For Human-robot Interaction

Kazuhiro Nakadai*, Ken-ichi Hidai**, Hiroshi G. Okuno*,***, Hiroshi Mizoguchi**** and Hiroaki Kitano*,*****

*Kitano Symbiotic Systems Project, ERATO, Japan Science and Technology Corp. Mansion 31 Suite 6A, 6-31-15, Jingumae, Shibuya-ku, Tokyo, 150-0001 Japan

**Digital Creatures Laboratory, Sony Corp.

***Graduate School of Informatics, Kyoto University

****Department of Mechanical Engineering, Tokyo University of Science

*****Sony Computer Science Laboratories, Inc.

April 1, 2002
July 8, 2002
October 20, 2002
active audition, audio-visual integration, speaker tracking, human-robot interaction
This paper addresses real-time multiple speaker tracking because it is essential in robot perception and human-robot social interaction. The difficulty lies in treating a mixture of sounds, occlusion (some speakers are hidden) and real-time processing. Our approach consists of three components: (1) the extraction of the direction of each speaker by using interaural phase difference and interaural intensity difference, (2) the resolution of each speakers direction by multimodal integration of audition, vision and motion with canceling inevitable motor noises in motion in case of an unseen or silent speaker, and (3) the distributed implementation to three PCs connected by TCP/IP network to attain real-time processing. As a result, we attain robust real-time speaker tracking with 200 ms delay in a non-anechoic room, even when multiple speakers exist and the tracking person is visually occluded. In addition, the feasibility of social interaction is shown through application of our technique to a receptionist robot and a companion robot at a party.
Cite this article as:
K. Nakadai, K. Hidai, H. Okuno, H. Mizoguchi, and H. Kitano, “Real-time Auditory and Visual Multiple-speaker Tracking For Human-robot Interaction,” J. Robot. Mechatron., Vol.14 No.5, pp. 479-489, 2002.
Data files:

*This site is desgined based on HTML5 and CSS3 for modern browsers, e.g. Chrome, Firefox, Safari, Edge, Opera.

Last updated on Jul. 12, 2024