Unified Framework for Real-world Skeleton Action Recognition

SSTA-PRS: Selective Spatio-Temporal Aggregation Based Pose Refinement System | Project page
UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition | Project page
ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting | Project page

Evironment

-- Python3 with PyTorch version >=Pytorch0.4.

Data Preparation

• mkdir data

• Posetics: please contact us (di.yang@inria.fr) for Data Request.

• Toyota Smarthome: download the raw data (skeleton-v2.0 refined by SSTA-PRS).

• Penn Action: download the raw skeleton data.

• For other datasets: NTU-RGB+D/Skeleton-Kinetics.

• Put them under the data directory:

   -data\
    -smarthome_raw\
       -smarthome_skeletons\
        - ... .json
          ... .json
          ...               
    -pennAction_raw\
       -skeletons\
        - ... .json
          ...
    -posetics_raw\
       -posetics_train_label.json
       -posetics_val_label.json  
       -posetics_train\
        - ... .json
          ...        
       -posetics_val\
        - ... .json
          ... 
    -nturgbd_raw\
       -samples_with_missing_skeletons.txt
       -nturgb+d_skeletons\
        - ... .skeleton
        - ...
    -kinetics_raw\             
           ...
    -...
  

• Preprocess the data with

  cd data_gen
  python smarthome_gendata.py
  python penn_gendata.py
         ...
  

• Generate the bone data with:

  python gen_bone_data.py

Pre-training on Posetics

python run_unik.py --config ./config/posetics/train_joint.yaml

Pre-trained model is now avalable here. Move it to

./weights/

Training (Fine-tuining) & Testing

Change the config file depending on what you want (e.g., for Smarthome).

python run_unik.py --config ./config/smarthome-cross-subject/train_joint.yaml

python run_unik.py --config ./config/smarthome-cross-subject/train_bone.yaml

To ensemble the results of joints and bones, run test firstly to generate the scores of the softmax layer.

python run_unik.py --config ./config/smarthome-cross-subject/test_joint.yaml

python run_unik.py --config ./config/smarthome-cross-subject/test_bone.yaml

Then combine the generated scores with:

python ensemble.py --datasets smarthome/xsub

For evaluation on Smarthome:

• Cross-subject:

  python evaluation-cs.py runs/smarthome/smarthome_cs_unik_test_joint_right.txt 31
  

• Cross-view:

  python evaluation-cv.py runs/smarthome/smarthome_cv2_unik_test_joint_right.txt 19
  python evaluation-cv.py runs/smarthome/smarthome_cv1_unik_test_joint_right.txt 19
  

Citation

If you find this code useful for your research, please consider citing our paper:

@article{yang2021unik,
      title={UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition}, 
      author={Di Yang and Yaohui Wang and Antitza Dantcheva and Lorenzo Garattoni and Gianpiero Francesca and Francois Bremond},
      year={2021},
      journal={BMVC}
}

@article{yang2022via,
      title={ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting}, 
      author={Di Yang and Yaohui Wang and Antitza Dantcheva and Lorenzo Garattoni and Gianpiero Francesca and Francois Bremond},
      year={2022},
      journal={arXiv preprint arXiv:2209.00065}
}