American Sign Language Words Recognition Using Spatiooral Prosodic and Angle Features: A Sequential Learning Approach

Journal article

Authors/Editors

KOSIN CHAMNONGTHAI

Strategic Research Themes

Digital Transformation (Strategic Research Themes)

Publication Details

Author list: Abdullahi S.B., Chamnongthai K.

Publisher: Institute of Electrical and Electronics Engineers

Publication year: 2022

Journal: IEEE Access (2169-3536)

Volume number: 10

Start page: 15911

End page: 15923

Number of pages: 13

ISSN: 2169-3536

eISSN: 2169-3536

URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85124231010&doi=10.1109%2fACCESS.2022.3148132&partnerID=40&md5=22b6ffd5e4da72a80af183663d65ec6c

Languages: English-Great Britain (EN-GB)

View in Web of Science | View on publisher site | View citing articles in Web of Science

Abstract

Most of the available American Sign Language (ASL) words share similar characteristics. These characteristics are usually during sign trajectory which yields similarity issues and hinders ubiquitous application. However, recognition of similar ASL words confused translation algorithms, which lead to misclassification. In this paper, based on fast fisher vector (FFV) and bi-directional Long-Short Term memory (Bi-LSTM) method, a large database of dynamic sign words recognition algorithm called bidirectional long-short term memory-fast fisher vector (FFV-Bi-LSTM) is designed. This algorithm is designed to train 3D hand skeletal information of motion and orientation angle features learned from the leap motion controller (LMC). Each bulk features in the 3D video frame is concatenated together and represented as an high-dimensional vector using FFV encoding. Evaluation results demonstrate that the FFV-Bi-LSTM algorithm is suitable for accurately recognizing dynamic ASL words on basis of prosodic and angle cues. Furthermore, comparison results demonstrate that FFV-Bi-LSTM can provide better recognition accuracy of 98.6% and 91.002% for randomly selected ASL dictionary and 10 pairs of similar ASL words, in leave-one-subject-out cross-validation on the constructed dataset. The performance of our FFV-Bi-LSTM is further evaluated on ASL data set, leap motion dynamic hand gestures data set (LMDHG), and Semaphoric hand gestures contained in the Shape Retrieval Contest (SHREC) dataset. We improve the accuracy of the ASL data set, LMDHG, and SHREC data sets by 2%, 2%, and 3.19% respectively. © 2013 IEEE.

Keywords

Fast fisher vector, Leap motion controller, Orientation angles, Spatiooral sequence, Ubiquitous computing