Tags: isaac-sim/IsaacLab
Tags
docs: add NCCL troubleshooting notes for multi-GPU training (#5195) ## Description Adds a troubleshooting note to the multi-GPU training docs for Linux systems where distributed training may fail with `CUDA error: an illegal memory access was encountered` reported by `ProcessGroupNCCL`. This PR is documentation-only. It does not change the default distributed training behavior in IsaacLab or `rsl_rl`. The note documents NCCL environment-variable workarounds that were observed to restore stability on some affected systems: - `NCCL_SHM_DISABLE=1` - `NCCL_IB_DISABLE=1` - `NCCL_ALGO=Ring` The motivation for this change is to provide an official troubleshooting path for users who hit NCCL transport/algo issues on specific Linux multi-GPU setups. In our local reproduction, the failure was not caused by IsaacLab task logic itself, but occurred in the distributed training stack when using NCCL with humanoid locomotion workloads. Dependencies: none. Refs #4011 Refs #2756 ## Type of change - Documentation update ## Screenshots N/A ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there ## Context Local reproduction environment: - Ubuntu 22.04.5 - RTX 5090 x2 - Isaac Sim / IsaacLab multi-GPU training - official distributed minimal reproduction with `Isaac-Velocity-Flat-G1-v0` Observed behavior: - the default distributed launch failed with NCCL illegal memory access - `NCCL_SHM_DISABLE=1` was sufficient to make the official dual-GPU minimal reproduction pass - `NCCL_SHM_DISABLE=1 NCCL_IB_DISABLE=1 NCCL_ALGO=Ring` also restored stability in a longer validation run This PR documents those workarounds without changing defaults, since the NCCL transport/algo selection is handled below the IsaacLab task layer. --------- Signed-off-by: bxwang <bixiong.wang@x-humanoid.com> Signed-off-by: bixiong wang <wangbx02@126.com> Co-authored-by: bxwang <bixiong.wang@x-humanoid.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Antoine RICHARD <antoiner@nvidia.com>
docs: add NCCL troubleshooting notes for multi-GPU training (#5195) ## Description Adds a troubleshooting note to the multi-GPU training docs for Linux systems where distributed training may fail with `CUDA error: an illegal memory access was encountered` reported by `ProcessGroupNCCL`. This PR is documentation-only. It does not change the default distributed training behavior in IsaacLab or `rsl_rl`. The note documents NCCL environment-variable workarounds that were observed to restore stability on some affected systems: - `NCCL_SHM_DISABLE=1` - `NCCL_IB_DISABLE=1` - `NCCL_ALGO=Ring` The motivation for this change is to provide an official troubleshooting path for users who hit NCCL transport/algo issues on specific Linux multi-GPU setups. In our local reproduction, the failure was not caused by IsaacLab task logic itself, but occurred in the distributed training stack when using NCCL with humanoid locomotion workloads. Dependencies: none. Refs #4011 Refs #2756 ## Type of change - Documentation update ## Screenshots N/A ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there ## Context Local reproduction environment: - Ubuntu 22.04.5 - RTX 5090 x2 - Isaac Sim / IsaacLab multi-GPU training - official distributed minimal reproduction with `Isaac-Velocity-Flat-G1-v0` Observed behavior: - the default distributed launch failed with NCCL illegal memory access - `NCCL_SHM_DISABLE=1` was sufficient to make the official dual-GPU minimal reproduction pass - `NCCL_SHM_DISABLE=1 NCCL_IB_DISABLE=1 NCCL_ALGO=Ring` also restored stability in a longer validation run This PR documents those workarounds without changing defaults, since the NCCL transport/algo selection is handled below the IsaacLab task layer. --------- Signed-off-by: bxwang <bixiong.wang@x-humanoid.com> Signed-off-by: bixiong wang <wangbx02@126.com> Co-authored-by: bxwang <bixiong.wang@x-humanoid.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Antoine RICHARD <antoiner@nvidia.com>
Clarifies kit-less installation in docs (#5045) # Description Clarifies kit-less installation in docs and move it to its own page to be consistent with other installation methods. Hides Newton sim2sim and sim2real documentation as those examples are not yet available. ## Type of change - Documentation update ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
Bumps version to v2.3.2 (#4399) # Description Updates documentation to prepare for v2.3.2 release. ## Type of change - Documentation update ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Pascal Roth <57946385+pascal-roth@users.noreply.github.com>
Updates starlette to 0.49.1 due to security vulnerability (#4133) # Description Updates starlette to 0.49.1 due to security vulnerability in the previously used 0.45.3. ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
Adds matplotlib-inline to license exceptions (#3853) matplotlib-inline has BSD-3 license, so it's ok for us to include it. We already have the license file specified for it. - Documentation update - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
Updates version and release notes for v2.2.1 (#3296) # Description Updates version of the framework for 2.2.1 patch release. ## Type of change - This change requires a documentation update ## Checklist - [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: Kelly Guo <kellyg@nvidia.com>
PreviousNext