American Sign Language Fingerspelling Recognition in the Wild with Iterative Language Model Construction

Journal article

Authors/Editors

WUTTIPONG KUMWILAISAK

Strategic Research Themes

Digital Transformation (Strategic Research Themes)

Publication Details

Author list: Kumwilaisak, Wuttipong; Pannattee, Peerawat; Hansakunbuntheung, Chatchawarn; Thatphithakkul, Nattanun;

Publisher: Now Publishers

Publication year: 2022

Volume number: 11

Issue number: 1

ISSN: 2048-7703

eISSN: 2048-7703

URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85135408571&doi=10.1561%2f116.00000003&partnerID=40&md5=4ffb67e094c6f479029be04de3be1106

Languages: English-Great Britain (EN-GB)

View in Web of Science | View on publisher site | View citing articles in Web of Science

Abstract

This paper proposes a novel method to improve the accuracy of the American Sign Language fingerspelling recognition. Video sequences from the training set of the “ChicagoFSWild” dataset are first utilized for training a deep neural network of weakly supervised learning to generate frame labels from a sequence label automatically. The network of weakly supervised learning contains the AlexNet and the LSTM. This trained network generates a collection of frame-labeled images from the training video sequences that have Levenshtein distance between the predicted sequence and the sequence label equal to zero. The negative and positive pairs of all fingerspelling gestures are randomly formed from the collected image set. These pairs are adopted to train the Siamese network of the ResNet-50 and the projection function to produce efficient feature representations. The trained Resnet-50 and the projection function are concatenated with the bidirectional LSTM, a fully connected layer, and a softmax layer to form a deep neural network for the American Sign Language fingerspelling recognition. With the training video sequences, video frames corresponding to the video sequences that have Levenshtein distance between the predicted sequence and the sequence label equal to zero are added to the collected image set. The updated collected image set is used to train the Siamese network. The training process, from training the Siamese network to the update of the collected image set, is iterated until the image recognition performance is not further enhanced. The experimental results from the “ChicagoFSWild” dataset show that the proposed method surpasses the existing works in terms of the character error rate. © 2022 W. Kumwilaisak, P. Pannattee, C. Hansakunbuntheung and N. Thatphithakkul.

Keywords

iterative training