LeRobot + MuJoCo: ACT Pipeline

Original Work Credit:
This repository is based on lerobot-mujoco-tutorial by Jeongeun Park. This fork focuses on the ACT (Action-Chunking Transformer) pipeline for the EE5108 mini-project.

Collect demonstration data in MuJoCo, train an ACT policy, and deploy it in simulation. Single-task pick-and-place (SO-101 arm, blue block → bin).

LeRobot SO-101 arm simulation in Mujoco

Installation

Python 3.12 recommended. Docker is the simplest option.

Docker (recommended)

Two images:

Runtime (Dockerfile.runtime): data collection + policy deployment (smaller, no Jupyter).
Train/dev (Dockerfile): same as runtime + Jupyter for local training.

Build from lerobot_mujoco_sim/:

docker build -t lerobot_mujoco_sim:runtime -f Dockerfile.runtime .
docker build -t lerobot_mujoco_sim:train -f Dockerfile .

Run (GPU):

docker run --rm -it --gpus all -v "$PWD:/workspace" lerobot_mujoco_sim:runtime

Inside the container: cd /workspace/lerobot_mujoco_sim (or /workspace if you mounted the repo root).

With docker-compose:

docker compose up -d lerobot-runtime
docker compose exec lerobot-runtime bash
cd /workspace/lerobot_mujoco_sim

SO-101 / SO-100 Arm Assets

Vendored: third_party/SO-ARM100
MuJoCo assets: asset/so_arm100/SO101/, asset/so_arm100/SO100/
Scene used for ACT: asset/scene_so101_y.xml (SO-101, blue block → bin)

EE5108 Mini-Project Workflow

End-to-end flow: capture data → upload → train ACT (e.g. Colab) → deploy in MuJoCo.

1) Build and start runtime

From repo root on host:

docker build -t lerobot_mujoco_sim:runtime -f Dockerfile.runtime .
docker run --rm -it --gpus all -v "$PWD:/workspace" lerobot_mujoco_sim:runtime

Then inside container: cd /workspace/lerobot_mujoco_sim.

2) Capture data (SO-101)

Manual teleop (recommended first time):

python scripts/collect/manual_collect_data.py \
  --env-robot-profile so101 \
  --num-demo 20 \
  --repo-name <your-dataset-repo> \
  --root data/demo_data_so101 \
  --offline-local-only

Scripted batch (Mink FSM, no teleop):

python scripts/collect/batch_collect_data.py \
  --env-robot-profile so101 \
  --num-demo 200 \
  --repo-name <your-dataset-repo> \
  --root data/demo_data_so101 \
  --offline-local-only

Both write a LeRobot-style dataset under data/demo_data_so101 (or a _fresh_* variant if the directory already exists and --offline-local-only is set).

3) Upload dataset to Hugging Face

You'll need a Huggingface account and an access token to create a dataset. Login (one of):

python -c "from huggingface_hub import login; login(token='hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')"

Upload:

python scripts/hf/upload_hf.py \
  --folder data/demo_data_so101 \
  --repo-id <your-hf-username>/<your-dataset-repo> \
  --repo-type dataset \
  --create-if-missing \
  --message "Upload SO101 dataset from demo_data_so101"

Use --private for a private repo.

4) Train ACT

Use Google Colab (recommended) and run the notebook notebooks/EE5108_training_act.ipynb.
In the notebook, set dataset.repo_id to <your-hf-username>/<your-dataset-repo> (the dataset you uploaded in step 3).
The notebook trains an ACT checkpoint and writes deploy_metadata.json into the checkpoint directory for easy deployment.

5) Deploy in MuJoCo

Copy the trained checkpoint (e.g. into checkpoints/act_y/), then:

python scripts/deploy/deploy_act.py --checkpoint checkpoints/act_y

Config-file driven deploy (recommended):

python scripts/deploy/deploy_act.py --config configs/deploy_act.yaml

CPU-only:

python scripts/deploy/deploy_act.py --checkpoint checkpoints/act_y --device cpu

Project Structure

lerobot_mujoco_sim/
├── README.md
├── Dockerfile              # Train/dev image (Jupyter + deps)
├── Dockerfile.runtime      # Runtime image (collect + deploy)
├── docker-compose.yml
├── asset/
│   ├── scene_so101_y.xml   # SO-101 pick-and-place scene
│   ├── so_arm100/          # SO-101/SO-100 MuJoCo assets
│   └── tabletop/
├── configs/
│   ├── collect_batch.yaml
│   ├── collect_manual.yaml
│   └── collect_data.yaml
├── scripts/
│   ├── collect/            # manual_collect_data.py, batch_collect_data.py
│   ├── deploy/             # deploy_act.py
│   ├── train/              # train_act.py
│   ├── visualize/          # visualize_data.py
│   └── hf/                 # upload_hf.py
├── mujoco_env/             # MuJoCo environment
├── controllers/            # Mink FSM for batch collection
├── so101/                  # SO-101 kinematics
└── third_party/SO-ARM100/

Data and checkpoints are local (e.g. data/, checkpoints/) and typically not committed.

Collect Data

Default task: pick blue block, place in bin. Scene: asset/scene_so101_y.xml.

Manual collection (keyboard)

WASD + R/F + arrows: joint control (see Keyboard controls).
SPACE: toggle gripper. Z: reset and discard current episode.
Recording starts on first movement; finish a successful pick-and-place to save the episode.

Config and YAML:

python scripts/collect/manual_collect_data.py --config configs/collect_manual.yaml
python scripts/collect/batch_collect_data.py --config configs/collect_batch.yaml --num-demo 100

Keyboard controls (SO-101)

Key	Action
A / D	Shoulder pan
W / S	Shoulder lift
R / F	Elbow flex
↑ / ↓	Wrist flex
← / →	Wrist roll
SPACE	Toggle gripper
Z	Reset & discard episode

Scripted batch

batch_collect_data.py uses a Mink-based FSM to generate demonstrations without teleop. Same dataset format as manual collection.

Playback Data

Replay saved episodes in MuJoCo:

python scripts/visualize/visualize_data.py

Or use the notebook notebooks/2.visualize_data.ipynb.

Train ACT

Recommended (students): run the Colab notebook notebooks/EE5108_training_act.ipynb.

Minimal local training example (if you already have the environment set up):

python scripts/train/train_act.py

Checkpoint is written to checkpoints/act_y/ (including deploy_metadata.json for deployment). For full training (e.g. Colab), use the LeRobot pipeline with your Hugging Face dataset.

Deploy ACT

Run the policy in the same SO-101 MuJoCo scene using a trained ACT checkpoint.

python scripts/deploy/deploy_act.py --checkpoint checkpoints/act_y

scripts/deploy/deploy_act.py builds the ACT policy from your checkpoint’s config.json, then loads dataset normalization stats (needed to denormalize the action outputs correctly).

The dataset stats come from one of these places:

If checkpoints/<your-checkpoint>/deploy_metadata.json exists, deploy_act.py uses that directly.
Otherwise, it looks for local dataset metadata under --dataset-root.
It expects meta/info.json (includes features).
It expects meta/stats.json (normalization statistics).
If those local files are missing, it downloads only meta/info.json and meta/stats.json from the Hugging Face dataset repo given by --dataset-repo-id.

Notes / gotchas:

This repo’s deployment script is MuJoCo sim-only. Do not use lerobot-record for this workflow (it targets the physical robot and requires a hardware --robot.port).
Make sure --dataset-repo-id points to a Hugging Face dataset repo (uploaded with --repo-type dataset), not a model/policy repo.

Common command (local dataset stats):

python scripts/deploy/deploy_act.py \
  --checkpoint checkpoints/act_y \
  --dataset-root data/demo_data_so101

Using the YAML config

Spawn bounds and runtime defaults for deploy live in configs/deploy_act.yaml.

python scripts/deploy/deploy_act.py --config configs/deploy_act.yaml

CLI flags still override YAML values. Options include: --xml-path, --device cpu, --seed, --spawn-x-min, etc. See --help.

Important flags (most used)

Run python scripts/deploy/deploy_act.py --help for the full list, but these are the key ones:

--checkpoint: checkpoint folder containing at least config.json (and optionally deploy_metadata.json)
--dataset-root: local dataset folder (default is set in the YAML). Expected to contain meta/info.json and meta/stats.json.
--dataset-repo-id: Hugging Face dataset repo id used only if local meta/ files aren’t present
--xml-path: MuJoCo scene XML

If your dataset is private

Set HF_TOKEN in the environment, or (when using the provided docker-compose runtime) place a token in token.txt at the repo root so the container can export it.

Arm Profiles & Config

Profiles: so101 (default), so100, omy. Defaults come from configs/collect_data.yaml (e.g. env_robot_profile: so101, xml_path: ./asset/scene_so101_y.xml). Override at runtime:

python scripts/collect/manual_collect_data.py --env-robot-profile so100

Spawn bounds and scene settings are in the same config.

Upload to Hugging Face

After collecting data:

huggingface-cli login   # or set HF_TOKEN
python scripts/hf/upload_hf.py \
  --folder data/demo_data_so101 \
  --repo-id <your-hf-username>/<your-dataset-repo> \
  --repo-type dataset \
  --create-if-missing

If using docker-compose with HF_HUB_OFFLINE=1, unset it for uploads.

Acknowledgements

Original tutorial: lerobot-mujoco-tutorial by Jeongeun Park.
SO-ARM assets: TheRobotStudio/SO-ARM100.
LeRobot: huggingface/lerobot.
MuJoCo parser inspiration: yet-another-mujoco-tutorial.

License

See LICENSE. Third-party components (e.g. SO-ARM100, LeRobot) have their own licenses.

Navigation Menu

Repository files navigation

LeRobot + MuJoCo: ACT Pipeline

Table of Contents

Installation

Docker (recommended)

SO-101 / SO-100 Arm Assets

EE5108 Mini-Project Workflow

1) Build and start runtime

2) Capture data (SO-101)

3) Upload dataset to Hugging Face

4) Train ACT

5) Deploy in MuJoCo

Project Structure

Collect Data

Manual collection (keyboard)

Keyboard controls (SO-101)

Scripted batch

Playback Data

Train ACT

Deploy ACT

Using the YAML config

Important flags (most used)

If your dataset is private

Arm Profiles & Config

Upload to Hugging Face

Acknowledgements

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 32 Commits
asset		asset
configs		configs
controllers		controllers
media		media
mujoco_env		mujoco_env
notebooks		notebooks
scripts		scripts
so101		so101
test_results		test_results
third_party/SO-ARM100		third_party/SO-ARM100
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
Dockerfile.runtime		Dockerfile.runtime
INSTALL_RUNTIME_MACOS.md		INSTALL_RUNTIME_MACOS.md
INSTALL_RUNTIME_NO_DOCKER.md		INSTALL_RUNTIME_NO_DOCKER.md
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
test_checkpoint.py		test_checkpoint.py
train_act_local.py		train_act_local.py

Folders and files

Latest commit

History

Repository files navigation

LeRobot + MuJoCo: ACT Pipeline

Table of Contents

Installation

Docker (recommended)

SO-101 / SO-100 Arm Assets

EE5108 Mini-Project Workflow

1) Build and start runtime

2) Capture data (SO-101)

3) Upload dataset to Hugging Face

4) Train ACT

5) Deploy in MuJoCo

Project Structure

Collect Data

Manual collection (keyboard)

Keyboard controls (SO-101)

Scripted batch

Playback Data

Train ACT

Deploy ACT

Using the YAML config

Important flags (most used)

If your dataset is private

Arm Profiles & Config

Upload to Hugging Face

Acknowledgements

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages