Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep torch dependency from bloating our common pixi environments #6965

Open
jleibs opened this issue Jul 23, 2024 · 1 comment
Open

Keep torch dependency from bloating our common pixi environments #6965

jleibs opened this issue Jul 23, 2024 · 1 comment
Labels
🧑‍💻 dev experience developer experience (excluding CI) 🚢 CI

Comments

@jleibs
Copy link
Member

jleibs commented Jul 23, 2024

Even though pixi is fairly efficient when it comes to downloading, caching, etc. the torch dep still adds several gigs to the environment.

This has two notable consequences:

  • It actually causes smaller github runners to fail because they run out of storage. (Fix nightly notebook test #6964)
  • It causes significantly more churn on the github cache of the pixi environments since we blow the 10GB limit with just a few environments.

We currently need torch for 2 things:

  • Running unitests that depend on the torch tensor type
  • Running a handful of examples that depend on torch

Proposal

  • Make the torch dep for unit-tests optional. Only include torch arrays if the dep is present.
    • If we're feeling particularly fancy we could fallback-import a torch tensor shim, but I'm not sure it's worth the effort.
  • Split out the torch environment dep so we end up with wheel-test and wheel-test-torch
  • Remove any examples that need torch from main
  • Only use wheel-test-torch on nightly to:
    • Validate the unittests still pass with torch
    • Run any examples that depend on torch
  • Disable saving the wheel-test-torch cache. Since we only run it nightly it's not worth causing cache churn for it.
@jleibs jleibs added 🧑‍💻 dev experience developer experience (excluding CI) 🚢 CI labels Jul 23, 2024
@jleibs jleibs changed the title Keep torch dependency from bloating our pixi common pixi environments Keep torch dependency from bloating our common pixi environments Jul 23, 2024
@emilk
Copy link
Member

emilk commented Aug 5, 2024

Maybe things depending on torch should be in its own repo?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🧑‍💻 dev experience developer experience (excluding CI) 🚢 CI
Projects
None yet
Development

No branches or pull requests

2 participants