Multimodal Fusion CRNN

Official PyTorch implementation of A Deep Learning-based Multimodal Depth-Aware Dynamic Hand Gesture Recognition System.

Setup

pip install -r requirements.txt

Training

python train_DHG.py --conf <path/to/config.yaml>

python train_SHREC.py --conf <path/to/config.yaml>

See the sample configs for config examples.

Tracking with W&B

We use Weights & Biases for tracking experiments. You would need to make a free account, and then get an api token to use in your config. Here is a snapshot of the experiment panel, for example:

Tutorials

• DHG-14/28: Training & Evaluation on Kaggle

Grayscale Variation

Original 16-bit depth image:

Depth quantized 8-bit gvar image:

Citation

@misc{mahmud2021deep,
      title   = {A Deep Learning-based Multimodal Depth-Aware Dynamic Hand Gesture Recognition System}, 
      author  = {Hasan Mahmud and Mashrur M. Morshed and Md. Kamrul Hasan},
      year    = {2021},
      eprint  = {2107.02543},
      archivePrefix = {arXiv},
      primaryClass = {cs.CV}
}