Bengali Word Detection from Lip Movements Using Mask RCNN and Generalized Linear Model

Abstract

Speech processing with the help of lip detection and lip reading is anadvancing field. For this, we need proper algorithms and techniques to detectlips and movements of lips perfectly. Lip detection and configuration are themost important parts of speech recognition. In this paper, we focus ondetecting the lip segment properly. Mask R-CNN (Regional ConvolutionalNeural Network) performs object detection and instance segmentation pervideo frame to detect the lip segment. The process of mask R-CNN addsonly a small overhead to Faster R-CNN and is quite simple to train, runningat 5 frames per second. The Mask R-CNN involves keypoint detection whichhelps to extract the location of the lip landmarks pixel by pixel. Once the lipregion is extracted and the landmarks are highlighted, we observe how thelip landmarks change as the object's lips move over time to each Bengaliword. The keypoint changes that are observed during each millisecond arethen the landmarks used to train the GLM (Generalized Linear Model). Inaddition, we compare the performance of GLM with Naive Bayes, LogisticRegression, and Decision Tree. The GLM has exhibited the highest 91.8%accuracy, whereas the Naive Bayes, Logistic Regression, and Decision Treeshow the accuracy of 87.1%, 38.3%, and 82.2%, respectively.