Conversation
Documentation build overview
52 files changed ·
|
Also a minor wording improvement to one other section.
| knowledge lives, and who is able to add to it. | ||
|
|
||
| The approaches below differ in how much standardization they require, | ||
| and in whether that requirement returns each time a new compatibility |
There was a problem hiding this comment.
| and in whether that requirement returns each time a new compatibility | |
| and in whether that requirement returns every time a new compatibility |
There was a problem hiding this comment.
I didn't include this, "each time" is actually a little better here, and it's used once more below, and contrasts with "every case"
| verifying the validity of variant properties and embedding static | ||
| properties via plugins at build time. | ||
|
|
||
| - UX, maintainability, security and governance: the introduction |
There was a problem hiding this comment.
Here looks like we are trying to put a lot of concepts in 1 PEP. Should we separate this in 2 ?
-
PEP 4 — UX, maintainability, security. What installers do: the introduction strategy, opt-in mechanics, when a provider runs, what the user sees, how trust is expressed in tooling. A spec, reviewable by installer maintainers, mostly tractable.
-
PEP 5 — the trusted-provider repository and its governance. Who curates, on what criteria, with what appeal process, funded and staffed by whom. An ongoing institution, reviewable by PyPA/packaging-council, and by far the slowest-moving piece.
There was a problem hiding this comment.
I'd much prefer to not touch this now, because we've consistently said that we have four PEPs, and this set of topics goes quite well together. We haven't started working on this PEP yet; if it gets too large we can change it later, but I don't want to do that now.
| variant wheel ordering rules, environment markers and ``pylock.toml`` | ||
| integration. | ||
|
|
||
| - Providers: the governance of variant namespaces; extending variant |
There was a problem hiding this comment.
Maybe we can give a little bit more context here with examples ? Something like:
Each provider governs one namespace — nvidia for CUDA capabilities, x86_64 for CPU instruction-set levels, openblas for BLAS choice.
| provide the choice of PyTorch Build (stable or nightly), | ||
| operating system (Linux, Mac, Windows), package (Pip, | ||
| LibTorch, Source), language (Python, C++ / Java), and Compute | ||
| Platform (CUDA 12.6, CUDA 12.8, CUDA 13.0, ROCM 6.4, CPU). |
There was a problem hiding this comment.
Lets also mention Intel XPU here, since we did develop Intel XPU provided plugin for torch.
There was a problem hiding this comment.
Done in commit 727bc1e (not here, since alt text needs to describe what's in the figure, but I mentioned Intel XPU further down)
| at the time of writing reject (ignore) wheels containing that component, | ||
| making it possible to publish variant wheels on an index alongside | ||
| non-variant wheels without risk of them being installed accidentally. | ||
| component to the wheel filename, and that tools commonly used to install |
There was a problem hiding this comment.
Perhaps we can be more detailed here ?
Could Backwards Compatibility say explicitly whether variant and non-variant wheels are expected to live on the same index? The section implies yes ("publish variant wheels on an index alongside non-variant wheels"), but it's stated once in passing, and it's the first question a distributor will ask. Making it explicit would help.
What will be our options when hosting these wheels on pypi ?
How would transition from non variant wheels to variant wheels looks like ? From my understanding when we fully switch to variant wheels we would like to stop publishing non-variant wheels at some point. Would be nice to mention this.
There was a problem hiding this comment.
Good questions, thanks! I'll try to expand. Not all of that is backwards compatibility, it's mostly rollout probably. We may have a gap about rollout strategy that needs filling.
To answer the questions:
- Yes, non-variant and variant wheels should be published to the same index.
- PyPI is such an index and will allow variant wheels once this PEP series is accepted and implemented.
- Phased rollout:
- First a project will only add variant wheels to the non-variant ones it published before.
- After all relevant versions of tools are all variant-aware (will take years), it becomes possible to drop the non-variant wheels.
There was a problem hiding this comment.
Added all this, and added an Adoption and Rollout section, in commit 9f192d0
"Platform tag" is one of the three tags the wheel filename carries, alongside the Python tag and the ABI tag; "platform compatibility tags" is the collective name, and the name of the specification. Using the bare plural understated the claim being made - none of the three can express these properties, not just the third - and pointed readers at the wrong tag for the ABI example.
Focus on the use cases where this functionality is essential, and mention the "runs the least code" as a secondary benefit. The providers have a good security story, so there's no need to have the static file option if that were the only reason; instead it's a hard necessity for some use cases like building containers.
|
The changes I just pushed should address all open comments, please take another look. |
8628e2a to
e10c3c0
Compare
|
Addressed Michal's comments, ready again. |
|
This had three reviewers and two approvals, with @atalman's comments also addressed (and not controversial I think, just regular improvements). So I plan to merge this later tonight or tomorrow morning, unless anyone needs more time. |
This branch has not been deployed
4 participants
The largest addition is a side-by-side comparison of five design options, which are the ones we've considered and those that have been brought up in the various discussions on DPO. It's deliberately side-by-side, so reviewers can understand how we arrived at the current design.
Note that in a call a while back, I showed a similar table to the one added now, but with more decision criteria and explicit scoring (good, medium, bad with emoji's); I decided against that because any scoring as well as the choices of more detailed criteria can all be argued with, while the version in this PR is as objective as possible.
Other sections that are updated:
Sections added or touched in the design overview:
The Motivation and Prior Art sections were not touched.
With these updates, I think the PEP is ready to be used as an Informational umbrella PEP.