Dynamic Hand Gesture Recognition Through Three Stream 3D CNN and Bidirectional Gated Recurrent Unit Using Multimodal Fusion

Abstract

People in the world have been applying Long Short-term memory network (LSTM) and Recurrent Neural Network (RNN) for processing sequential information like multimedia video and image data. In the last decade, many researchers had tried work in hand gesture recognition but applied less focus on advanced feature extraction, gathering motion information, fusion scheme, and temporal feature extraction. In this proposed research paper, we have implemented a hand gesture recognition method by three streams 3D convolutional neural network for feature extraction and deep bidirectional gated recurrent network for temporal recognition with advanced feature extraction. First, we have converted all the video to a fixed number of 35-frames. After that, we have calculated an optical flow from RGB frames for finding proper motion information. Then 3D convolutional neural network is used for space-temporal feature extraction. These features are blended using various fusion scheme and passed to the bidirectional neural network for merging features back and forth at different timestamp t. This bidirectional gated recurrent unit (GRU) has multiple layers per timestep and it gives a higher learning rate than a single directional recurrent unit. Our proposed method gets a classification accuracy of 80.5% on the standard VIVA dataset. This dataset provides hand gesture for a touchless interface in vehicles for automatic controlling various device.