Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci/tests: cleaning standalone script #19141

Merged
merged 20 commits into from
Dec 13, 2023
Merged

ci/tests: cleaning standalone script #19141

merged 20 commits into from
Dec 13, 2023

Conversation

Borda
Copy link
Member

@Borda Borda commented Dec 12, 2023

What does this PR do?

simplify & unify standalone scripts for maintenance

Before submitting
  • Was this discussed/agreed via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist
  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

📚 Documentation preview 📚: https://pytorch-lightning--19141.org.readthedocs.build/en/19141/

cc @carmocca @Borda @justusschock @awaelchli

@github-actions github-actions bot added fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package package labels Dec 12, 2023
@github-actions github-actions bot added the ci Continuous Integration label Dec 12, 2023
@Borda Borda changed the title tests: cleaning standalone script tests: cleaning standalone script +tpu Dec 12, 2023
@Borda Borda marked this pull request as ready for review December 12, 2023 16:02
Copy link
Contributor

github-actions bot commented Dec 12, 2023

⛈️ Required checks status: Has failure 🔴

Warning
This job will need to be re-run to merge your PR. If you do not have write access to the repository, you can ask Lightning-AI/lai-frameworks to re-run it. If you push a new commit, all of CI will re-trigger.

Groups summary

🔴 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, lightning, 3.8, 1.12, oldest) success
pl-cpu (macOS-11, lightning, 3.9, 1.12) success
pl-cpu (macOS-11, lightning, 3.10, 1.13) success
pl-cpu (macOS-11, lightning, 3.10, 2.0) success
pl-cpu (macOS-11, lightning, 3.10, 2.1) success
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.12, oldest) success
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12) success
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0) success
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.1) success
pl-cpu (windows-2022, lightning, 3.8, 1.12, oldest) success
pl-cpu (windows-2022, lightning, 3.9, 1.12) success
pl-cpu (windows-2022, lightning, 3.10, 1.13) success
pl-cpu (windows-2022, lightning, 3.10, 2.0) success
pl-cpu (windows-2022, lightning, 3.10, 2.1) success
pl-cpu (macOS-11, pytorch, 3.8, 1.13) success
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13) success
pl-cpu (windows-2022, pytorch, 3.8, 1.13) success
pl-cpu (macOS-12, pytorch, 3.11, 2.0) failure
pl-cpu (macOS-12, pytorch, 3.11, 2.1) success
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0) success
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.1) success
pl-cpu (windows-2022, pytorch, 3.11, 2.0) success
pl-cpu (windows-2022, pytorch, 3.11, 2.1) success

These checks are required after the changes to src/lightning/pytorch/utilities/testing/_runif.py, tests/tests_pytorch/conftest.py, tests/tests_pytorch/profilers/test_profiler.py, tests/tests_pytorch/run_standalone_tasks.sh, tests/tests_pytorch/run_tpu_tests.sh, pyproject.toml.

🟢 pytorch_lightning: Azure GPU
Check ID Status
pytorch-lightning (GPUs) (testing Lightning | latest) success
pytorch-lightning (GPUs) (testing PyTorch | latest) success

These checks are required after the changes to .azure/gpu-tests-pytorch.yml, src/lightning/pytorch/utilities/testing/_runif.py, tests/tests_pytorch/conftest.py, tests/tests_pytorch/profilers/test_profiler.py, tests/tests_pytorch/run_standalone_tasks.sh, tests/tests_pytorch/run_tpu_tests.sh, pyproject.toml.

🟢 pytorch_lightning: Benchmarks
Check ID Status
lightning.Benchmarks success

These checks are required after the changes to src/lightning/pytorch/utilities/testing/_runif.py.

🔴 pytorch-lightning: TPU workflow
Check ID Status
test-on-tpus (pytorch, xrt, v4-8) failure
test-on-tpus (pytorch, pjrt, v4-8) failure

These checks are required after the changes to tests/tests_pytorch/run_tpu_tests.sh.

🟢 fabric: Docs
Check ID Status
docs-make (fabric, doctest) success
docs-make (fabric, html) success

These checks are required after the changes to pyproject.toml.

🟢 pytorch_lightning: Docs
Check ID Status
docs-make (pytorch, doctest) success
docs-make (pytorch, html) success

These checks are required after the changes to src/lightning/pytorch/utilities/testing/_runif.py, pyproject.toml.

🔴 lightning_data: CPU workflow
Check ID Status
data-cpu (macOS-11, lightning, 3.10, 2.1) failure
data-cpu (ubuntu-20.04, lightning, 3.10, 2.1) success
data-cpu (windows-2022, lightning, 3.10, 2.1) success

These checks are required after the changes to pyproject.toml.

🟢 lightning_fabric: CPU workflow
Check ID Status
fabric-cpu (macOS-11, lightning, 3.8, 1.12, oldest) success
fabric-cpu (macOS-11, lightning, 3.9, 1.12) success
fabric-cpu (macOS-11, lightning, 3.10, 1.13) success
fabric-cpu (macOS-11, lightning, 3.10, 2.0) success
fabric-cpu (macOS-11, lightning, 3.11, 2.1) success
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.12, oldest) success
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.12) success
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0) success
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.1) success
fabric-cpu (windows-2022, lightning, 3.8, 1.12, oldest) success
fabric-cpu (windows-2022, lightning, 3.9, 1.12) success
fabric-cpu (windows-2022, lightning, 3.10, 1.13) success
fabric-cpu (windows-2022, lightning, 3.10, 2.0) success
fabric-cpu (windows-2022, lightning, 3.11, 2.1) success
fabric-cpu (macOS-11, fabric, 3.8, 1.13) success
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13) success
fabric-cpu (windows-2022, fabric, 3.8, 1.13) success
fabric-cpu (macOS-12, fabric, 3.11, 2.0) success
fabric-cpu (macOS-12, fabric, 3.11, 2.1) success
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.0) success
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.1) success
fabric-cpu (windows-2022, fabric, 3.11, 2.0) success
fabric-cpu (windows-2022, fabric, 3.11, 2.1) success

These checks are required after the changes to tests/tests_fabric/conftest.py, tests/tests_fabric/run_standalone_tests.sh, tests/tests_fabric/run_tpu_tests.sh, pyproject.toml.

🟢 lightning_fabric: Azure GPU
Check ID Status
lightning-fabric (GPUs) (testing Fabric | latest) success
lightning-fabric (GPUs) (testing Lightning | latest) success

These checks are required after the changes to .azure/gpu-tests-fabric.yml, tests/tests_fabric/run_standalone_tests.sh, tests/tests_fabric/conftest.py, tests/tests_fabric/run_tpu_tests.sh, pyproject.toml.

🔴 lightning_fabric: TPU workflow
Check ID Status
test-on-tpus (fabric, xrt, v4-8) failure
test-on-tpus (pytorch, pjrt, v4-8) failure

These checks are required after the changes to tests/tests_fabric/run_tpu_tests.sh.

🟢 lightning_app: Docs
Check ID Status
docs-make (app, doctest) success
docs-make (app, html) success

These checks are required after the changes to pyproject.toml.

🟢 mypy
Check ID Status
mypy success

These checks are required after the changes to src/lightning/pytorch/utilities/testing/_runif.py, pyproject.toml.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.8) success
install-pkg (ubuntu-22.04, app, 3.11) success
install-pkg (ubuntu-22.04, fabric, 3.8) success
install-pkg (ubuntu-22.04, fabric, 3.11) success
install-pkg (ubuntu-22.04, pytorch, 3.8) success
install-pkg (ubuntu-22.04, pytorch, 3.11) success
install-pkg (ubuntu-22.04, lightning, 3.8) success
install-pkg (ubuntu-22.04, lightning, 3.11) success
install-pkg (ubuntu-22.04, notset, 3.8) success
install-pkg (ubuntu-22.04, notset, 3.11) success
install-pkg (macOS-12, app, 3.8) success
install-pkg (macOS-12, app, 3.11) success
install-pkg (macOS-12, fabric, 3.8) success
install-pkg (macOS-12, fabric, 3.11) success
install-pkg (macOS-12, pytorch, 3.8) success
install-pkg (macOS-12, pytorch, 3.11) success
install-pkg (macOS-12, lightning, 3.8) success
install-pkg (macOS-12, lightning, 3.11) success
install-pkg (macOS-12, notset, 3.8) success
install-pkg (macOS-12, notset, 3.11) success
install-pkg (windows-2022, app, 3.8) success
install-pkg (windows-2022, app, 3.11) success
install-pkg (windows-2022, fabric, 3.8) success
install-pkg (windows-2022, fabric, 3.11) success
install-pkg (windows-2022, pytorch, 3.8) success
install-pkg (windows-2022, pytorch, 3.11) success
install-pkg (windows-2022, lightning, 3.8) success
install-pkg (windows-2022, lightning, 3.11) success
install-pkg (windows-2022, notset, 3.8) success
install-pkg (windows-2022, notset, 3.11) success

These checks are required after the changes to src/lightning/pytorch/utilities/testing/_runif.py.


Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

Copy link

codecov bot commented Dec 12, 2023

Codecov Report

Merging #19141 (304d8a2) into master (2e77862) will decrease coverage by 16%.
The diff coverage is 100%.

Additional details and impacted files
@@            Coverage Diff            @@
##           master   #19141     +/-   ##
=========================================
- Coverage      81%      66%    -16%     
=========================================
  Files         443      438      -5     
  Lines       36760    36662     -98     
=========================================
- Hits        29854    24084   -5770     
- Misses       6906    12578   +5672     

echo "Running profilers/test_profiler.py::test_pytorch_profiler_nested_emit_nvtx"
nvprof --profile-from-start off -o trace_name.prof -- python -m coverage run --source lightning.pytorch --append -m pytest --no-header profilers/test_profiler.py::test_pytorch_profiler_nested_emit_nvtx
fi
#can_run_nvprof=$(python -c "import torch; print(torch.cuda.is_available() and torch.cuda.get_device_capability()[0] < 8)")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note this was not run in any our CI

pyproject.toml Outdated Show resolved Hide resolved
@Borda Borda changed the title tests: cleaning standalone script +tpu tests: cleaning standalone script Dec 12, 2023
@Borda Borda added the run TPU label Dec 12, 2023
@Borda Borda changed the title tests: cleaning standalone script ci/tests: cleaning standalone script Dec 13, 2023
@carmocca carmocca mentioned this pull request Dec 13, 2023
@lantiga lantiga merged commit 2a8789e into master Dec 13, 2023
118 of 125 checks passed
@lantiga lantiga deleted the tests/markr-stanalone branch December 13, 2023 19:27
Borda added a commit that referenced this pull request Dec 19, 2023
* tests: cleaning standalone script

* switch

* from tests

* -m

* collect

* array

* tests_fabric/

* ..

* path prefix

* pl

* cleaning

* test_pytorch_profiler_nested_emit_nvtx

* Apply suggestions from code review

* Apply suggestions from code review

* todo

(cherry picked from commit 2a8789e)
lantiga pushed a commit that referenced this pull request Dec 20, 2023
* tests: cleaning standalone script

* switch

* from tests

* -m

* collect

* array

* tests_fabric/

* ..

* path prefix

* pl

* cleaning

* test_pytorch_profiler_nested_emit_nvtx

* Apply suggestions from code review

* Apply suggestions from code review

* todo

(cherry picked from commit 2a8789e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Continuous Integration fabric lightning.fabric.Fabric package pl Generic label for PyTorch Lightning package run TPU
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants