Adds galbot stack cube tasks, with left_arm_gripper and right_arm_suction, using RMPFlow controller #3210
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- add surfacegripper actions - add new task envs with UR10 + suction cups
…dundant parallel_gripper task
- add galbot robot config - add RMPFlowAction and RMPFlowActionCfg - add motion_policy_configs for both galbot_left_arm_gripper and galbot_right_arm_suction - add galbot stack_rmp_rel_env_cfg - add galbot_stack_rmp_abs/rel_mimic_env - add mdp.observations/terminations/events for galbot: support both suction_cup and parallel_gripper, get obs_in_base_frame - fix a bug (eef_name) in franka_stack_ik_rel_mimic_env.py - add new device_name in se3_spacemouse.py
…the RobotCfg back to default - move surface_gripper_actions out of binary_joint_actions to a standalone file
rebeccazhang0707
requested review from
Mayankm96,
jtigue-bdai,
kellyguo11,
ooctipus,
pascal-roth and
peterd-NV
as code owners
…eractiveScene class
…miniation function cannot be reached
Signed-off-by: rebeccazhang0707 <168459200+rebeccazhang0707@users.noreply.github.com>
…ed() terminiation function cannot be reached" This reverts commit 6edd402.
- env.scene.surface_grippers is a dict
6 tasks
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
kellyguo11
reviewed
Sep 3, 2025
| # FIXME: temp uploaded to nucleus folder, remove this line in public release and use above ISAACLAB_NUCLEUS_DIR | ||
| ISAACLAB_NUCLEUS_DIR = "https://isaac-dev.ov.nvidia.com/omni/web3/omniverse://isaac-dev.ov.nvidia.com/Isaac/IsaacLab" | ||
|
|
||
| ISAACLAB_NUCLEUS_RMPFLOW_DIR = os.path.join(ISAACLAB_NUCLEUS_DIR, "Controllers", "RmpFlowAssets") |
There was a problem hiding this comment.
once we are ready with this folder, we can ping Sheikh to push these changes publicly
|
@kellyguo11 updated the docs and visualized the collision_spheres in RmpFlowAssets yaml file for your kind review: Each arm can do collision-free motion generation and teleoperation. 
|
kellyguo11
reviewed
Sep 4, 2025
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
kellyguo11
reviewed
Sep 5, 2025
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
kellyguo11
approved these changes
Sep 5, 2025
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
kellyguo11
added a commit
that referenced
this pull request
Sep 10, 2025
# Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Adds two place tasks and mimic tasks with Agibot A2D humanoid, using RMPFlow controller: - add agibot robot config in agibot.py and .usd asset - add motion_policy_configs and .urdf for rmpflow controller - add new task cfg: place_toy2box_rmp_rel_env_cfg, and place_upright_mug_rmp_rel_env_cfg - add new mimic task cfg: agibot_place_toy2box_mimic_env_cfg, and agibot_place_upright_mug_mimic_env_cfg - add new mimic task: in pick_place_mimic_env.py - add new subtasks in mdp.observations/terminations: object_grasped, object_placed_upright, object_a_is_into_b Notes: This PR relies on PR (#3210) for RmpFlowAction support. You can test the whole gr00t-mimic workflow by: 1. Record Demos ``` ./isaaclab.sh -p scripts/tools/record_demos.py \ --dataset_file datasets/recorded_demos_agibot_right_arm_rel.hdf5 \ --task Isaac-Place-Toy2Box-Agibot-Right-Arm-RmpFlow-v0 \ --teleop_device spacemouse \ --num_demos 1 ``` 2. Replay Demos ``` ./isaaclab.sh -p scripts/tools/replay_demos.py \ --dataset_file datasets/recorded_demos_agibot_right_arm_rel.hdf5 \ --task Isaac-Place-Toy2Box-Agibot-Right-Arm-RmpFlow-v0 \ --num_envs 1 ``` 3. Annotate Demos ``` ./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/annotate_demos.py \ --input_file datasets/recorded_demos_agibot_right_arm_rel.hdf5 \ --output_file datasets/annotated_demos_agibot_right_arm_rel.hdf5 \ --task Isaac-Place-Toy2Box-Agibot-Right-Arm-RmpFlow-Rel-Mimic-v0 \ --auto ``` 4. Generate Demos ``` ./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \ --input_file datasets/annotated_demos_agibot_right_arm_rel.hdf5 \ --output_file datasets/generated_demos_agibot_right_arm_rel.hdf5 \ --task Isaac-Place-Toy2Box-Agibot-Right-Arm-RmpFlow-Rel-Mimic-v0 \ --num_envs 16 \ --generation_num_trials 10 ``` <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - New feature (non-breaking change which adds functionality) ## Screenshot <img width="1401" height="567" alt="environments_agibot" src="https://github.com/user-attachments/assets/bc95e83a-e334-4dfc-a65c-3d1d97be0c4b" /> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Signed-off-by: rebeccazhang0707 <168459200+rebeccazhang0707@users.noreply.github.com> Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com>
kellyguo11
added a commit
that referenced
this pull request
Sep 10, 2025
# Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Adds two place tasks and mimic tasks with Agibot A2D humanoid, using RMPFlow controller: - add agibot robot config in agibot.py and .usd asset - add motion_policy_configs and .urdf for rmpflow controller - add new task cfg: place_toy2box_rmp_rel_env_cfg, and place_upright_mug_rmp_rel_env_cfg - add new mimic task cfg: agibot_place_toy2box_mimic_env_cfg, and agibot_place_upright_mug_mimic_env_cfg - add new mimic task: in pick_place_mimic_env.py - add new subtasks in mdp.observations/terminations: object_grasped, object_placed_upright, object_a_is_into_b Notes: This PR relies on PR (#3210) for RmpFlowAction support. You can test the whole gr00t-mimic workflow by: 1. Record Demos ``` ./isaaclab.sh -p scripts/tools/record_demos.py \ --dataset_file datasets/recorded_demos_agibot_right_arm_rel.hdf5 \ --task Isaac-Place-Toy2Box-Agibot-Right-Arm-RmpFlow-v0 \ --teleop_device spacemouse \ --num_demos 1 ``` 2. Replay Demos ``` ./isaaclab.sh -p scripts/tools/replay_demos.py \ --dataset_file datasets/recorded_demos_agibot_right_arm_rel.hdf5 \ --task Isaac-Place-Toy2Box-Agibot-Right-Arm-RmpFlow-v0 \ --num_envs 1 ``` 3. Annotate Demos ``` ./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/annotate_demos.py \ --input_file datasets/recorded_demos_agibot_right_arm_rel.hdf5 \ --output_file datasets/annotated_demos_agibot_right_arm_rel.hdf5 \ --task Isaac-Place-Toy2Box-Agibot-Right-Arm-RmpFlow-Rel-Mimic-v0 \ --auto ``` 4. Generate Demos ``` ./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \ --input_file datasets/annotated_demos_agibot_right_arm_rel.hdf5 \ --output_file datasets/generated_demos_agibot_right_arm_rel.hdf5 \ --task Isaac-Place-Toy2Box-Agibot-Right-Arm-RmpFlow-Rel-Mimic-v0 \ --num_envs 16 \ --generation_num_trials 10 ``` <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - New feature (non-breaking change which adds functionality) ## Screenshot <img width="1401" height="567" alt="environments_agibot" src="https://github.com/user-attachments/assets/bc95e83a-e334-4dfc-a65c-3d1d97be0c4b" /> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Signed-off-by: rebeccazhang0707 <168459200+rebeccazhang0707@users.noreply.github.com> Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com>
ooctipus
pushed a commit
to ooctipus/IsaacLab
that referenced
this pull request
Sep 20, 2025
# Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Adds two place tasks and mimic tasks with Agibot A2D humanoid, using RMPFlow controller: - add agibot robot config in agibot.py and .usd asset - add motion_policy_configs and .urdf for rmpflow controller - add new task cfg: place_toy2box_rmp_rel_env_cfg, and place_upright_mug_rmp_rel_env_cfg - add new mimic task cfg: agibot_place_toy2box_mimic_env_cfg, and agibot_place_upright_mug_mimic_env_cfg - add new mimic task: in pick_place_mimic_env.py - add new subtasks in mdp.observations/terminations: object_grasped, object_placed_upright, object_a_is_into_b Notes: This PR relies on PR (isaac-sim#3210) for RmpFlowAction support. You can test the whole gr00t-mimic workflow by: 1. Record Demos ``` ./isaaclab.sh -p scripts/tools/record_demos.py \ --dataset_file datasets/recorded_demos_agibot_right_arm_rel.hdf5 \ --task Isaac-Place-Toy2Box-Agibot-Right-Arm-RmpFlow-v0 \ --teleop_device spacemouse \ --num_demos 1 ``` 2. Replay Demos ``` ./isaaclab.sh -p scripts/tools/replay_demos.py \ --dataset_file datasets/recorded_demos_agibot_right_arm_rel.hdf5 \ --task Isaac-Place-Toy2Box-Agibot-Right-Arm-RmpFlow-v0 \ --num_envs 1 ``` 3. Annotate Demos ``` ./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/annotate_demos.py \ --input_file datasets/recorded_demos_agibot_right_arm_rel.hdf5 \ --output_file datasets/annotated_demos_agibot_right_arm_rel.hdf5 \ --task Isaac-Place-Toy2Box-Agibot-Right-Arm-RmpFlow-Rel-Mimic-v0 \ --auto ``` 4. Generate Demos ``` ./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \ --input_file datasets/annotated_demos_agibot_right_arm_rel.hdf5 \ --output_file datasets/generated_demos_agibot_right_arm_rel.hdf5 \ --task Isaac-Place-Toy2Box-Agibot-Right-Arm-RmpFlow-Rel-Mimic-v0 \ --num_envs 16 \ --generation_num_trials 10 ``` <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - New feature (non-breaking change which adds functionality) ## Screenshot <img width="1401" height="567" alt="environments_agibot" src="https://github.com/user-attachments/assets/bc95e83a-e334-4dfc-a65c-3d1d97be0c4b" /> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Signed-off-by: rebeccazhang0707 <168459200+rebeccazhang0707@users.noreply.github.com> Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds galbot_stack_cube tasks and mimic tasks, using RMPFlow controller:
Notes: This PR relies on PR (#3174) for surface gripper support in manager-based workflow.
You can test the whole gr00t-mimic workflow by:
./isaaclab.sh -p scripts/tools/record_demos.py --task Isaac-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow-v0 --teleop_device spacemouse --num_demos 1 --device cpu --dataset_file datasets/recorded_demos_galbot_suction_rel.hdf5./isaaclab.sh -p scripts/tools/replay_demos.py --task Isaac-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow-v0 --num_envs 1 --device cpu --dataset_file datasets/recorded_demos_galbot_suction_rel.hdf5./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/annotate_demos.py --task Isaac-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow-Rel-Mimic-v0 --auto --device cpu --input_file datasets/recorded_demos_galbot_suction_rel.hdf5 --output_file datasets/annotated_demos_galbot_suction_rel.hdf5./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py --task Isaac-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow-Rel-Mimic-v0 --num_envs 16 --device cpu --generation_num_trials 10 --input_file datasets/annotated_demos_galbot_suction_rel.hdf5 --output_file datasets/generated_demos_galbot_suction_rel.hdf5Type of change
Screenshot
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there