Repository navigation
Add minimal wheel build mode - #19899
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19899
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 2 PendingAs of commit 54de550 with merge base 12684ef ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@claude review this PR |
|
Claude finished @JacobSzwejbka's task in 2m 7s —— View job Reviewing PR #19899: Add minimal wheel build mode
SummaryWell-structured PR that introduces an opt-in Feedback1. Potential race in CI:
|
|
Thanks for putting this together. The overall approach looks right: it is opt-in, defined upstream, and uses an allowlist of what to keep instead of a list of what to delete, so it will not quietly grow again on future versions. The keep set is correct, including the two parts that are easy to miss (extension/pytree and extension/flat_tensor), and reproducing the exact same .pte bytes is good evidence it works. A few things I think we should improve:
Minor:
None of this blocks the direction. Item 1 is small and the highest value. |
I thought they wanted the minimal build for the custom packages where they build the entire thing themself. So building from source is fine. |
|
Makes sense; building from source covers the distribution side. The main thing left is the dependency list. Minimal mode trims the files but not the declared deps, so a normal install still pulls coremltools, scikit-learn, pandas, hydra-core, omegaconf (coremltools is gated to Darwin or Linux, so it does hit the Linux/Jetson consumer). The test only passes because it uses --no-deps. These are static [project.dependencies] in pyproject.toml, so an env flag can't subtract them. To vary them by build mode you'd need to make deps dynamic: add "dependencies" to pyproject's dynamic=[...], remove the static dependencies array, and compute install_requires in setup.py branching on EXECUTORCH_BUILD_MINIMAL (roughly the set already installed in test_minimal_wheel.sh: flatbuffers, numpy, packaging, pyyaml, ruamel.yaml, sympy, tabulate, typing-extensions; torch is assumed present). Worth also deciding what to do with mpmath and pytorch-tokenizers, which are declared today too. Then drop --no-deps from the test and assert the heavy deps are absent, so it actually proves the slim claim. |
Yeah we intend to build the package from source for the container distribution |
8fd5caf to
454deef
Compare
The opt-in minimal wheel build (`EXECUTORCH_BUILD_MINIMAL=1`) previously trimmed the packaged files but not the declared dependencies. Because `pyproject.toml` `[project.dependencies]` is static, a normal `pip install` of the minimal wheel still pulled `coremltools`, `scikit-learn`, `pandas`, `hydra-core`, `omegaconf` and the test-only deps (`expecttest`, `hypothesis`, `kgb`, `parameterized`), which is most of the real install size. The CI smoke test only passed because it used `pip install --no-deps`. Make `dependencies` dynamic in `pyproject.toml` and compute `install_requires` in `setup.py` per build mode. The full (default) wheel keeps the existing set unchanged; the minimal wheel declares the subset of that set which `executorch.exir` needs to export a `.pte`: `flatbuffers`, `numpy`, `packaging`, `pyyaml`, `ruamel.yaml`, `sympy`, `tabulate`, `typing-extensions`. `torch` stays consumer-provided and `mpmath` resolves transitively via `sympy`. The minimal set is derived from the full set (with a build-time check that every kept name matched a full dep) so pins and markers cannot drift. Also make the minimal build skip the non-minimal cmake targets explicitly, and harden the smoke test: drop `--no-deps` so dependency resolution is exercised, assert the built wheel METADATA declares exactly the slim set, and run the bundled `flatc` (the only compiled artifact) to confirm it ships and resolves. Document the slim deps and that `torch` is consumer-provided in `README-wheel.md`. Co-authored-by: Jacob Szwejbka <20804483+JacobSzwejbka@users.noreply.github.com>
454deef to
2ba2a55
Compare
Adds an opt-in
EXECUTORCH_BUILD_MINIMAL=1wheel build mode that packages only the Python EXIR export path (plusflatc), for distributors that need ExecuTorch's ahead-of-time.pteexport but not its runtime. For example, Torch-TensorRT'soutput_format="executorch"uses ExecuTorch only to export and runs the result with its own runtime.The minimal wheel:
flatbuffers,numpy,packaging,pyyaml,ruamel.yaml,sympy,tabulate,typing-extensions) instead of the full set (coremltools,scikit-learn,pandas,hydra-core,omegaconf, and so on), so a normal install stays small;.pteoutput to the full wheel.The default (non-minimal) wheel is unchanged: its dependencies move from static
pyproject.tomlto a dynamicinstall_requiresinsetup.py, but the declared set is identical.Build from source and bundle it:
A redistributor (e.g. NVIDIA, for a Torch-TensorRT container) can build the slim wheel at a pinned ExecuTorch version and ship it.
torchis consumer-provided in both modes.CI (
test-minimal-wheel-linux) builds the minimal wheel, asserts the excluded runtime/backend content and heavy deps are absent, installs it in a clean venv with full dependency resolution (no--no-deps), runs the bundledflatc, and exports MobileNetV2 to a.pte.Local result: minimal Linux x86_64 wheel ~2.1 MiB compressed vs ~15 MiB for the full wheel; MobileNetV2
.pteis 13,995,880 bytes, byte-identical to the published 1.3.1 wheel.