Open Access Open Access  Restricted Access Subscription or Fee Access

Dynamic Hand Gesture Recognition Through Three Stream 3D CNN and Bidirectional Gated Recurrent Unit Using Multimodal Fusion

Sunil A. Patel, Ramji M. Makwana

Abstract



People in the world have been applying Long Short-term memory network (LSTM) and Recurrent Neural Network (RNN) for processing sequential information like multimedia video and image data. In the last decade, many researchers had tried work in hand gesture recognition but applied less focus on advanced feature extraction, gathering motion information, fusion scheme, and temporal feature extraction. In this proposed research paper, we have implemented a hand gesture recognition method by three streams 3D convolutional neural network for feature extraction and deep bidirectional gated recurrent network for temporal recognition with advanced feature extraction. First, we have converted all the video to a fixed number of 35-frames. After that, we have calculated an optical flow from RGB frames for finding proper motion information. Then 3D convolutional neural network is used for space-temporal feature extraction. These features are blended using various fusion scheme and passed to the bidirectional neural network for merging features back and forth at different timestamp t. This bidirectional gated recurrent unit (GRU) has multiple layers per timestep and it gives a higher learning rate than a single directional recurrent unit. Our proposed method gets a classification accuracy of 80.5% on the standard VIVA dataset. This dataset provides hand gesture for a touchless interface in vehicles for automatic controlling various device.

Keywords


Convolutional neural network, Bidirectionnel gated recurrent unit, Hand gesture, VIVA ,3D CNN

Full Text:

PDF


Disclaimer/Regarding indexing issue:

We have provided the online access of all issues and papers to the indexing agencies (as given on journal web site). It’s depend on indexing agencies when, how and what manner they can index or not. Hence, we like to inform that on the basis of earlier indexing, we can’t predict the today or future indexing policy of third party (i.e. indexing agencies) as they have right to discontinue any journal at any time without prior information to the journal. So, please neither sends any question nor expects any answer from us on the behalf of third party i.e. indexing agencies.Hence, we will not issue any certificate or letter for indexing issue. Our role is just to provide the online access to them. So we do properly this and one can visit indexing agencies website to get the authentic information.