A 3D Point-of-Intention Estimation Method Using Multimodal Fusion of Hand Pointing, Eye Gaze and Depth Sensing for Collaborative Robots

บทความในวารสาร

ผู้เขียน/บรรณาธิการ

กลุ่มสาขาการวิจัยเชิงกลยุทธ์

รายละเอียดสำหรับงานพิมพ์

รายชื่อผู้แต่ง: Yeamkuan S., Chamnongthai K., Pichitwong W.

ผู้เผยแพร่: Institute of Electrical and Electronics Engineers

ปีที่เผยแพร่ (ค.ศ.): 2021

วารสาร: IEEE Sensors Journal (1530-437X)

Volume number: 22

Issue number: 3

นอก: 1530-437X

eISSN: 1558-1748

URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85121387918&doi=10.1109%2fJSEN.2021.3133471&partnerID=40&md5=a857f8a446972d91b547f84173ff418f

ภาษา: English-Great Britain (EN-GB)

ดูในเว็บของวิทยาศาสตร์ | ดูบนเว็บไซต์ของสำนักพิมพ์ | บทความในเว็บของวิทยาศาสตร์

บทคัดย่อ

Hand pointing psychologically expressing intention has been fused with eye gaze to assist in detecting the point of intention (POI). Ideally, a POI detection approach using a pair of hand-pointing and eye-gaze rays may achieve precise POI positioning in 3-dimensional (3D) space; however, an object sometimes occludes the detection process in real applications. This paper proposes a method of 3D POI estimation using multimodal fusion of hand pointing, eye gaze and depth sensing for collaborative robots. In this method, depth sensors are used to assist in the fusion of the original hand-pointing and eye-gaze rays to determine the POI based on a volume of interest (VOI) to find 3D POI. At real production sites where humans work with collaborative robots, workpieces may sometimes occlude the depth sensors, resulting in blind views. This problem is solved by increasing the number of depth sensors and arranging them with overlapping view ranges to cover otherwise blind views due to occlusion. The number of depth sensors is determined based on the largest possible obstacle size determined by users in advance. Experiments performed by 10 participants confirm the effectiveness of POI estimation in the presence of obstacles; specifically, with our proposed method, we measured 3D POIs at 50 cm, 70 cm, 75 cm, 80 cm, 85 cm, 90 cm, and 95 cm with average distance errors of 0.82%, 0.68%, 0.68%, 0.72%, 0.76%, 0.87%, and 0.81%, respectively, compared with the conventional POI estimation method based on the fusion of hand pointing and eye gaze. IEEE

คำสำคัญ

3D POG, eye gaze, hand pointing, multimodal, point cloud, point of intention, Robot sensing systems, Sensor phenomena and characterization, Three-dimensional displays