Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracking issue: Speed up CI #3033

Open
5 of 14 tasks
jprochazk opened this issue Aug 17, 2023 · 1 comment
Open
5 of 14 tasks

Tracking issue: Speed up CI #3033

jprochazk opened this issue Aug 17, 2023 · 1 comment
Labels
🧑‍💻 dev experience developer experience (excluding CI) 📉 performance Optimization, memory use, etc 🚢 CI 🎄 tracking issue issue that tracks a bunch of subissues
Milestone
Spring Cleaning

Comments

@jprochazk
Copy link
Member

jprochazk commented Aug 17, 2023 •
edited by emilk
Loading

Speed up the wait time for each PR

  • CI: Cache everything in cloud storage #2762
  • Split demo.rerun.io to demo and test #2566
  • Speed up web demo build (cache datasets)
  • Parallelize example .rrd file generation (?)
  • Attempt to build wheel + WASM in build web demo job instead of waiting for those two to finish separately (and reusing their artifacts).
  • Better job sequencing
    - Cargo rebuilds a lot, even with cache during the Rust lints job
  • Make sure we only build the web Wasm once
  • Do less on each PR, moving more to main and nightly
    • Only run cargo test on main
  • Only build some things when needed
    • Only build web viewer if Rust has changed
  • Try without https://github.com/Swatinem/rust-cache
    • Remember to set CARGO_INCREMENTAL self
  • Replace "Install [python] dependencies" with pixi (takes 2 minutes on the "Wheel build" step)
@jprochazk jprochazk added the 🧑‍💻 dev experience developer experience (excluding CI) label Aug 17, 2023
@jprochazk jprochazk mentioned this issue Aug 17, 2023
Open
6 tasks
@emilk emilk added 🎄 tracking issue issue that tracks a bunch of subissues 🚢 CI 📉 performance Optimization, memory use, etc labels Apr 10, 2024
@emilk
Copy link
Member

emilk commented Apr 10, 2024

Maybe some tricks can be learned from the CI section of
https://matklad.github.io/2021/09/04/fast-rust-builds.html

For instance, it suggest setting CARGO_INCREMENTAL: 0 in our env to disable incremental compilation, which should shrink the /target folder and maybe improve caching.

A lot of the tricks in #5332 also apply to this

@emilk emilk mentioned this issue Apr 10, 2024
Closed
@emilk emilk self-assigned this Apr 10, 2024
@Wumpf Wumpf mentioned this issue Apr 10, 2024
Merged
5 tasks
emilk pushed a commit that referenced this issue Apr 10, 2024
@Wumpf
Fix python lints, make python lint ci conditional (#5897)
bc47678 
### What

* related to #3033

--

* Turned out most of the python lints issues I had locally (and did not
see on ci) are about opencv-python missmatches. The way I fixed it
should work fine with either though now!
* Various python lints on ci now use pixi and run only when py files
change.
* update pixi version used on ci

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using newly built examples:
[rerun.io/viewer](https://rerun.io/viewer/pr/5897)
* Using examples from latest `main` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/5897?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/5897?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG
* [x] If applicable, add a new check to the [release
checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)!

- [PR Build Summary](https://build.rerun.io/pr/5897)
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)
This was referenced Apr 11, 2024
Merged
Merged
emilk added a commit that referenced this issue Apr 11, 2024
@emilk
Speed up CI: Skip a few steps (#5901)
60cf846 
* Part of #3033


### What
For PRs, skip:
* Building a notebook (moved to `nightly`)
* `cargo test`
* `roundtrips.py`
* `compare_snippet_output.py`

### Effect
Total duration: 17m59s -> 12m39s
Billable time: 1h15m -> 1h9m

### Checklist
* [x] A box was cheked

- [PR Build Summary](https://build.rerun.io/pr/5901)
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)
This was referenced Apr 11, 2024
Merged
Merged
teh-cmc added a commit that referenced this issue Apr 11, 2024
@emilk @teh-cmc
Update to Rust 1.76 (#5908)
eecfdb4 
* Part of #3033

---------

Co-authored-by: Clement Rey <cr.rey.clement@gmail.com>
@emilk emilk assigned emilk and unassigned emilk Apr 11, 2024
emilk added a commit that referenced this issue Apr 11, 2024
@emilk
Don't build examples on each PR (#5910)
6ad2766 
* Part of #3033

### What
It takes almost 3 minutes per PR.

Pro:
* I don't remember it ever catching a bug
* I usually test the latest examples from `main` or `nightly` anyway

Con:
* No example size diff comparisons per PR
* This breaks the <https://www.rerun.io/viewer/pr/5910> url, unless I
also update the default manifest 🤔

Total duration: 12m -> 8m
Billable time: 51m -> 43m

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using default manifest (nightly):
[rerun.io/viewer](https://rerun.io/viewer/pr/5910)
* Using examples from latest `main` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/5910?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/5910?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG
* [x] If applicable, add a new check to the [release
checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)!

- [PR Build Summary](https://build.rerun.io/pr/5910)
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)
@emilk emilk assigned emilk and unassigned emilk Apr 12, 2024
@emilk emilk added this to the Spring Cleaning milestone Apr 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🧑‍💻 dev experience developer experience (excluding CI) 📉 performance Optimization, memory use, etc 🚢 CI 🎄 tracking issue issue that tracks a bunch of subissues
Projects
None yet
Milestone
Spring Cleaning
Development

No branches or pull requests
2 participants
@emilk @jprochazk