DexTrack: Dexterous Manipulation Tracking

Website | Videos | OpenReview | arXiv

Implementation of our work DexTrack, capable of serving as an effective tool to 1) create dexterous robot hand-object manipulation demonstrations by mimicking kinematic references, and 2) develop a versatile tracking controller that can accomplish diverse manipulation tasks.

demo_video_vxx.mp4

Getting Started

Installation

Create a vritual environment with python 3.8.0:

conda create -n dextrack python=3.8.0
conda activate dextrack

Download the Isaac Gym Preview 4 release from the website.

cd isaacgym/python
pip install -e .

Install torch_cluster:

cd DexTrack/whls
pip install torch_cluster-1.6.3+pt24cu121-cp38-cp38-linux_x86_64.whl

Install other dependencies:

pip install rl_games transforms3d matplotlib omegaconf hydra-core trimesh mujoco tqdm

Data

Download the retargeted data from this link. Extract .zip files in the folder isaacgymenvs/data.

Download the first part of object files from this link. Extract this file in the folder assets.

Download the second part of object files from this link. Extract this file in the folder assets/rsc.

Download the third part of object files from this link. Extract this file in the folder assets.

(Optional) Download checkpoints from this link. Extract it in the folder isaacgymenvs/.

File structure:

isaacgymenvs/
  data/
    GRAB_Tracking_PK_reduced_300/
    GRAB_Tracking_PK_LEAP_OFFSET_0d4_0d5_warm_v2_v2urdf/
    modified_kinematics_data_leap_wfranka_v15urdf/
    TACO_Tracking_PK_reduced/
  ckpts/
    ...
assets/
  datasetv4.1/
  meshdatav3_scaled/
  rsc/
    objs/

Usage

This repository includes RL environments, along with the training and evaluation procedures, for the dexterous manipulation tracking problem.

We support 1) a fly Allegro hand (an Allegro hand with 6 global translational and rotational DoFs) and 2) a LEAP hand mounted on a Franka Panda arm. Two control strategies (action spaces) are implemented:

• Cumulated residual positional targets with kinematic bias;
• Relative positional targets. The original implementation uses the first action space.

DexTrack include two levels of tracking:

• Single trajectory tracking: The goal is to train a trajectory-specific policy to follow a single manipulation trajectory.
• Multiple trajectories tracking: The goal is to train a generalizable tracking policy capable of mimicking multiple manipulation trajectories and generalizing to unseen sequences.

Below, we will outline the training and evaluation processes for both settings.

Please ensure that you are running these commands in the DexTrack/isaacgymenvs folder.

Sincle trajectory tracking (fly Allegro hand)

GRAB Dataset

To train a single trajectory tracker for a sequence retargeted from the GRAB dataset using the cumulative residual action space, run the following code:

bash scripts/run_tracking_headless_grab_single.sh <GPU_ID> <SEQ_NAME>

Please replace <GPU_ID> with the index of the GPU you wish to use. Only single-GPU training is supported. For other arguments, replace <SEQ_NAME> with the name of the sequence you wish to track. Checkpoints will be saved in the ./logs folder.

Once you have obtained a checkpoint with a satisfactory reward, run the following code to evaluate it. A display is required if HEADLESS is set to False.

bash scripts/run_tracking_headless_grab_single_test.sh <GPU_ID> <SEQ_NAME> <CKPT> <HEADLESS>

To train a single-trajectory tracker for a sequence retargeted from the GRAB dataset using the relative positional action space, run the following code:

bash scripts/run_tracking_headless_grab_single_ctlv2.sh <GPU_ID> <SEQ_NAME>

Similarly, after you've obtained a good checkpoint, run the following code to evaluate it.

bash scripts/run_tracking_headless_grab_single_test_ctlv2.sh <GPU_ID> <SEQ_NAME> <CKPT> <HEADLESS>

Below, we provide several examples.

The following videos illustrate their corresponding input (kinematic references retargeted from human-object manipulation trajectories) and output (tracking results) that can be achieved.

	Cube	Duck	Flute
Kinematic References
Tracking Result

Please refer to the following instructions to reproduce the above tracking examples.

Case 1: Cubesmall inspect

To track the cubesmall_inspect trajectory from subject s2 on GPU 0, whose corresponding sequence name is ori_grab_s2_cubesmall_inspect_1, please run:

bash scripts/run_tracking_headless_grab_single.sh 0 ori_grab_s2_cubesmall_inspect_1

This sequence can be tracked pretty well using quite short time. We can reach a reward more than 150 at epoch 50 using 22000 parallel environments.

Our pre-trained weights can be downloaded form . Following instructions stated above and extract these files in the folder ./ckpts, you can run the test code using our trained policy for this sequence using the following command:

bash scripts/run_tracking_headless_grab_single_test.sh 0 ori_grab_s2_cubesmall_inspect_1 ./ckpts/s2_cubesmall_inspect_ckpt.pth False 

Case 2: Duck inspect

To track the duck_inspect trajectory from subject s2 on GPU 0, whose corresponding sequence name is ori_grab_s2_duck_inspect_1, please run:

bash scripts/run_tracking_headless_grab_single.sh 0 ori_grab_s2_duck_inspect_1

This sequence can also be tracked pretty well after training for a short time. We can reach a reward more than 150 at epoch 100 using 22000 parallel environments.

Similarly, our pretrained policy for this sequence can be evaluated using the following command:

bash scripts/run_tracking_headless_grab_single_test.sh 0 ori_grab_s2_duck_inspect_1 ./ckpts/s2_duck_inspect_ckpt.pth False

Case 3: Flute pass

To track the flute_pass trajectory from subject s2 on GPU 0, whose corresponding sequence name is ori_grab_s2_flute_pass_1, please run:

bash scripts/run_tracking_headless_grab_single.sh 0 ori_grab_s2_flute_pass_1

Our pretrained policy for this sequence can be evaluated using the following command:

bash scripts/run_tracking_headless_grab_single_test.sh 0 ori_grab_s2_flute_pass_1 ./ckpts/s2_flute_pass_ckpt.pth False

TACO Dataset

For sequences retargeted from TACO dataset, to track a trajectory with tag <TAG> using the cumulative residual action space, run the following code:

bash scripts/run_tracking_headless_taco_single.sh <GPU_ID> <TAG>

Similarly, after you've obtained a satisfactory checkpoint, run the following code to evaluate it.

bash scripts/run_tracking_headless_taco_single_test.sh <GPU_ID> <TAG> <CKPT> <HEADLESS>

Below, we provide several examples.

The following videos illustrate their corresponding input (kinematic references retargeted from human-object manipulation trajectories) and output (tracking results) that can be achieved.

	Shovel	Ladle	Soap
Kinematic References
Tracking Result

Case 1: Tool-using sequence (shovel) from TACO

To track the tool using sequence from TACO dataset tagged with taco_20231104_169 on GPU 0, run the following code

bash scripts/run_tracking_headless_taco_single.sh 0 taco_20231104_169

In our test, we can get nice result after 300 epochs training. To evaluate our pretrained policy, run the following comamnd:

bash scripts/run_tracking_headless_taco_single_test.sh 0 taco_20231104_169 ./ckpts/taco_1_ckpt.pth False 

Case 2: Tool-using sequence (ladle) from TACO

To track the tool using sequence from TACO dataset tagged with taco_20231104_186 on GPU 0, run the following code

bash scripts/run_tracking_headless_taco_single.sh 0 taco_20231104_186

To evaluate our pretrained policy, run the following comamnd:

bash scripts/run_tracking_headless_taco_single_test.sh 0 taco_20231104_186 ./ckpts/taco_2_ckpt.pth False

Case 3: Tool-using sequence (soap) from TACO

To track the tool using sequence from TACO dataset tagged with taco_20231103_073 on GPU 0, run the following code

bash scripts/run_tracking_headless_taco_single.sh 0 taco_20231103_073

To evaluate our pretrained policy, run the following comamnd:

bash scripts/run_tracking_headless_taco_single_test.sh 0 taco_20231103_073 ./ckpts/taco_3_ckpt.pth False

Multiple trajectory tracking (fly Allegro hand)

GRAB Dataset

To train a multiple trajectories tracker for sequences retargeted from the GRAB dataset using the cumulative residual action space, run the following code:

bash scripts/run_tracking_headless_grab_multiple.sh <GPU_ID> <SUBJ_NM> <SEQ_TAG_LIST>

Please replace <GPU_ID> with the index of the card you wish to run the code on. <SEQ_TAG_LIST> is the file specifying trajectories to track.

For instance, to train a neural controller that can track all manipulation trajectories with the object duck, run the following command:

bash scripts/run_tracking_headless_grab_multiple.sh 0 '' ../assets/inst_tag_list_obj_duck.npy

For all trajectories in the GRAB's training split (trajectories from s2 to s10), run:

bash scripts/run_tracking_headless_grab_multiple.sh 0 '' ''

Run the following command to evaluate a checkpoint:

bash scripts/run_tracking_headless_grab_multiple_test.sh <GPU_ID> <SEQ_NM> <CKPT> <HEADLESS> 

Similarly, to train a multiple trajectory tracker for a sequence retargeted from GRAB dataset using the relative target action space, run the following code:

bash scripts/run_tracking_headless_grab_multiple_ctlv2.sh <GPU_ID> <SUBJ_NM> <SEQ_TAG_LIST>

Running the following command for evaluation:

bash scripts/run_tracking_headless_grab_multiple_test_ctlv2.sh <GPU_ID> <TAG> <CKPT> <HEADLESS> 

Single trajectory tracking (LEAP hand with Franka arm)

To train a tracking policy for a sequence retargeted from the GRAB dataset using the relative positional action space, run the following code:

bash scripts/run_tracking_headless_grab_single_wfranka.sh <GPU_ID> <TAG>

For testing:

 bash scripts/run_tracking_headless_grab_single_wfranka_test.sh <GPU_ID> <TAG> <CKPT> <HEADLESS>

In additional to trajectories contained in the original GRAB dataset, we've synthesized more trajectories with in-hand reorientations. Please refer to data/modified_kinematics_data_leap_wfranka_v15urdf for their kinematic motions (the in-hand reorienntation stage contains only object motion variations with hand pose fixed). To train a tracking policy for a synthesized trajectory, run the following code:

bash scripts/run_tracking_headless_grab_single_syntraj_wfranka.sh <GPU_ID> <TAG> <SAMPLE_ID>

<SAMPLE_ID> should be replaced by a integer ranging from 0 to 99. For test:

bash scripts/run_tracking_headless_grab_single_syntraj_wfranka.sh <GPU_ID> <TAG> <SAMPLE_ID> <CKPT> <HEADLESS>

Below, we give several examples.

Their corresponding input and output are illustrated in the following videos:

	Elephant	Hammer	Watch
Kinematic References
Tracking Result

Case 1: Elephant

For training, run the following code

bash scripts/run_tracking_headless_grab_single_wfranka.sh 0 ori_grab_s2_elephant_inspect_1

To evaluate our pretrained policy, run the following comamnd:

bash scripts/run_tracking_headless_grab_single_wfranka_test.sh 0 ori_grab_s2_elephant_inspect_1 ./ckpts/elephant_inspect_wfranka_ckpt.pth False

Case 2: Hammer

For training, run the following code

bash scripts/run_tracking_headless_grab_single_syntraj_wfranka.sh 0 ori_grab_s2_hammer_use_2 6

To evaluate our pretrained policy, run the following comamnd:

bash scripts/run_tracking_headless_grab_single_syntraj_wfranka.sh 0 ori_grab_s2_hammer_use_2 6 ./ckpts/hammer_reorient_sample_6_ckpt.pth False

Case 3: Watch

For training, run the following code

bash scripts/run_tracking_headless_grab_single_wfranka.sh 0 s1_watch_set_2

To evaluate our pretrained policy, run the following comamnd:

bash scripts/run_tracking_headless_grab_single_wfranka_test.sh 0 ori_grab_s1_watch_set_2 ./ckpts/watch_set_ckpt.pth False 

Multiple trajectories tracking (LEAP with Franka arm)

Similar to the fly hand setting, run the following command to train a tracking controller for sequences retargeted from the GRAB dataset:

bash scripts/run_tracking_headless_grab_multiple_wfranka.sh <GPU_ID> <SEQ_TAG_LIST>

For evaluation:

bash scripts/run_tracking_headless_grab_multiple_wfranka_test.sh <GPU_ID> <TAG> <CKPT> <HEADLESS>

Notice: In addition to the single and multiple trajectory tracking processes included above, DexTrack incorporates two key components that make the specialist-generalist iterative training framework work: 1) homotopy optimization for enhancing single trajectory tracking (applicable only to policies using the cumulative residual action space), and 2) a combination of IL and RL to improve the generalist tracker. However, these components require significant human effort and cannot easily be condensed into a single script. Besides, the corresponding code and scripts are too messy to be cleaned within an acceptable time frame. As a result, we do not currently plan to release them publicly.

Contact

Please contact xymeow7@gmail.com or create a github issue if you have any questions.

Acknowledgments

This code is standing on the shoulders of giants. We want to thank the following contributors that our code is based on: IsaacGymEnvs and UniDexGrasp.

License

See LICENSE.