CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning

Jiange Yang, Yansong Shi, Haoyi Zhu, Mingyu Liu, Kaijing Ma, Yating Wang, Gangshan Wu, Tong He, Limin Wang

Overview

Training latent motion IDM.

Prepare the training video data by following the list format used in idm/mask_extracted_40k.txt, idm/droid_success_40k.txt, and idm/samv_40k.txt.

Example command (uses 4 GPUs):

cd idm
CUDA_VISIBLE_DEVICES=0,1,2,3 NUM_PROCESSES=4 MAIN_PROCESS_PORT=18897 bash train.sh configs/train_como_default.json

Extracting latent motion using trained IDM as pseudo action labels.

Example: Download the LIBERO dataset from ATM first. Extract latent motion from ATM LIBERO-10 data

cd idm
python process.py \
  --model-path /path/to/your/model.pt \
  --data-folder /path/to/ATM/libero/libero_10/*/images/*/ \
  --output-file latent_motion_our \
  --model-config-path configs/process_default.json 

Training and evaluating the unified CoMo policy.

The policy stage jointly trains on robot demonstrations and human-video latent motion labels. The preprocessed HDF5 files should follow the ATM/LIBERO layout under a suite directory such as:

<COMO_LIBERO_DATA_ROOT>/
  libero_10/
    <task_name>_demo/
      demo_0.hdf5
      env_meta.json
      images/

For each task, CoMo uses 50 demos:

• demo_0 to demo_9: robot demos with 7-D actions
• demo_10 to demo_49: human-video demos whose root/actions are replaced by 256-D latent motion

By default the latent-motion files are expected at:

<COMO_LIBERO_DATA_ROOT>/latent_motion/latent_motion_our/<suite>/<task_name>_demo/demo_*.hdf5

--latent-action-tag controls the subdirectory name in that path. --latent-action-root can be used to point to a different latent-motion directory directly.

If you need to preprocess raw LIBERO demonstrations into the ATM/LIBERO layout:

cd DP
python scripts/preprocess_libero.py \
  --root /path/to/raw/libero \
  --save /path/to/atm_libero \
  --suite libero_10

Train the policy on a target LIBERO suite (example: libero_10):

cd DP
export COMO_LIBERO_DATA_ROOT=/path/to/atm_libero
python scripts/train_libero_policy.py --suite libero_10

Or pass the data root explicitly:

cd DP
python scripts/train_libero_policy.py \
  --suite libero_10 \
  --data-root /path/to/atm_libero

If the latent-motion files are stored elsewhere:

cd DP
python scripts/train_libero_policy.py \
  --suite libero_10 \
  --data-root /path/to/atm_libero \
  --latent-action-root /path/to/latent_motion_our

If you only need to switch the latent tag:

cd DP
python scripts/train_libero_policy.py \
  --suite libero_10 \
  --data-root /path/to/atm_libero \
  --latent-action-tag latent_motion_our

Evaluate a trained policy checkpoint directory:

cd DP
export COMO_LIBERO_DATA_ROOT=/path/to/atm_libero
python scripts/eval_libero_policy.py \
  --suite libero_object \
  --exp-dir /path/to/DP/results/policy/<your_experiment_dir>

Citation

@article{yang2025learning,
  title={Como: Learning continuous latent motion from internet videos for scalable robot learning},
  author={Yang, Jiange and Shi, Yansong and Zhu, Haoyi and Liu, Mingyu and Ma, Kaijing and Wang, Yating and Wu, Gangshan and He, Tong and Wang, Limin},
  journal={arXiv preprint arXiv:2505.17006},
  year={2025}
}

Contact

jiangeyang.jgy@gmail.com

Acknowledgements

We thank the authors and open-source communities of the following projects:

• Moto-GPT
• RoboticsDiffusionTransformer
• LAPA
• deoxys_control
• ATM