A 3D Point-of-Intention Estimation Method Using Multimodal Fusion of Hand Pointing, Eye Gaze and Depth Sensing for Collaborative Robots

Journal article

Authors/Editors

KOSIN CHAMNONGTHAI

Strategic Research Themes

Digital Transformation (Strategic Research Themes)

Publication Details

Author list: Yeamkuan S., Chamnongthai K., Pichitwong W.

Publisher: Institute of Electrical and Electronics Engineers

Publication year: 2021

Journal: IEEE Sensors Journal (1530-437X)

Volume number: 22

Issue number: 3

ISSN: 1530-437X

eISSN: 1558-1748

URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85121387918&doi=10.1109%2fJSEN.2021.3133471&partnerID=40&md5=a857f8a446972d91b547f84173ff418f

Languages: English-Great Britain (EN-GB)

View in Web of Science | View on publisher site | View citing articles in Web of Science

Abstract

Hand pointing psychologically expressing intention has been fused with eye gaze to assist in detecting the point of intention (POI). Ideally, a POI detection approach using a pair of hand-pointing and eye-gaze rays may achieve precise POI positioning in 3-dimensional (3D) space; however, an object sometimes occludes the detection process in real applications. This paper proposes a method of 3D POI estimation using multimodal fusion of hand pointing, eye gaze and depth sensing for collaborative robots. In this method, depth sensors are used to assist in the fusion of the original hand-pointing and eye-gaze rays to determine the POI based on a volume of interest (VOI) to find 3D POI. At real production sites where humans work with collaborative robots, workpieces may sometimes occlude the depth sensors, resulting in blind views. This problem is solved by increasing the number of depth sensors and arranging them with overlapping view ranges to cover otherwise blind views due to occlusion. The number of depth sensors is determined based on the largest possible obstacle size determined by users in advance. Experiments performed by 10 participants confirm the effectiveness of POI estimation in the presence of obstacles; specifically, with our proposed method, we measured 3D POIs at 50 cm, 70 cm, 75 cm, 80 cm, 85 cm, 90 cm, and 95 cm with average distance errors of 0.82%, 0.68%, 0.68%, 0.72%, 0.76%, 0.87%, and 0.81%, respectively, compared with the conventional POI estimation method based on the fusion of hand pointing and eye gaze. IEEE

Keywords

3D POG, eye gaze, hand pointing, multimodal, point cloud, point of intention, Robot sensing systems, Sensor phenomena and characterization, Three-dimensional displays