Adds training benchmark unit tests with input config #2503
Merged
+772
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
matthewtrepte
requested review from
Mayankm96,
jsmith-bdai and
kellyguo11
as code owners
kellyguo11
reviewed
May 16, 2025
Contributor
kellyguo11
left a comment
kellyguo11
left a comment
There was a problem hiding this comment.
Thanks for putting this together! Based on Alexander's concerns right now, let's add the regression tests to TESTS_TO_SKIP in test_settings.py for now and work with him to create a new job for these that can be manually triggered.
kellyguo11
changed the title
fan-ziqi
pushed a commit
to fan-ziqi/IsaacLab
that referenced
this pull request
Jun 5, 2025
# Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Add unit test for training and evaluating environments across all workflows using an input config. The config determines which environments to train, how long to train, and training KPI thresholds. Every training + evaluation is one pytest, which fails if any of the thresholds aren't met. A KPI json can be created to summarize the trainings. This json also functions as a KPI payload which will be uploaded to a Grafana Dashboard every week using OSMO CI, using the full mode in the config. Can be used to track improvements and regressions. Dashboard - [LINK](https://grafana.nvidia.com/d/cejwy2w46gtmob/24b94e54-6b8b-50a4-969a-121dc573a34c?orgId=105) <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) - New feature (non-breaking change which adds functionality) - Breaking change (fix or feature that would cause existing functionality to not work as expected) - This change requires a documentation update ## Screenshots Please attach before and after screenshots of the change if applicable. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Co-authored-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyguo123@hotmail.com>
yrh012
pushed a commit
to aica-technology/isaac-lab
that referenced
this pull request
Jun 16, 2025
# Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Add unit test for training and evaluating environments across all workflows using an input config. The config determines which environments to train, how long to train, and training KPI thresholds. Every training + evaluation is one pytest, which fails if any of the thresholds aren't met. A KPI json can be created to summarize the trainings. This json also functions as a KPI payload which will be uploaded to a Grafana Dashboard every week using OSMO CI, using the full mode in the config. Can be used to track improvements and regressions. Dashboard - [LINK](https://grafana.nvidia.com/d/cejwy2w46gtmob/24b94e54-6b8b-50a4-969a-121dc573a34c?orgId=105) <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) - New feature (non-breaking change which adds functionality) - Breaking change (fix or feature that would cause existing functionality to not work as expected) - This change requires a documentation update ## Screenshots Please attach before and after screenshots of the change if applicable. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Co-authored-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyguo123@hotmail.com>
harry-wf-cheung
pushed a commit
to harry-wf-cheung/IsaacLab-Harry
that referenced
this pull request
Jul 30, 2025
# Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Add unit test for training and evaluating environments across all workflows using an input config. The config determines which environments to train, how long to train, and training KPI thresholds. Every training + evaluation is one pytest, which fails if any of the thresholds aren't met. A KPI json can be created to summarize the trainings. This json also functions as a KPI payload which will be uploaded to a Grafana Dashboard every week using OSMO CI, using the full mode in the config. Can be used to track improvements and regressions. Dashboard - [LINK](https://grafana.nvidia.com/d/cejwy2w46gtmob/24b94e54-6b8b-50a4-969a-121dc573a34c?orgId=105) <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) - New feature (non-breaking change which adds functionality) - Breaking change (fix or feature that would cause existing functionality to not work as expected) - This change requires a documentation update ## Screenshots Please attach before and after screenshots of the change if applicable. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Co-authored-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyguo123@hotmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add unit test for training and evaluating environments across all workflows using an input config.
The config determines which environments to train, how long to train, and training KPI thresholds.
Every training + evaluation is one pytest, which fails if any of the thresholds aren't met.
A KPI json can be created to summarize the trainings. This json also functions as a KPI payload which will be uploaded to a Grafana Dashboard every week using OSMO CI, using the full mode in the config. Can be used to track improvements and regressions.
Dashboard - LINK
Type of change
Screenshots
Please attach before and after screenshots of the change if applicable.
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there