-
Notifications
You must be signed in to change notification settings - Fork 1.4k
fix an issue where conflict markers could instigate a very large lock file #11293
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
} | ||
} | ||
imp(self, &mut f); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ibraheemdev Is this the best way to do this? And is it even right? I spent an embarrassing amount of time on just this part. (I originally wrote it by walking the internal representation, but that ended up being wrong in a way that I didn't understand.)
Basically, the essential challenge I faced here was figuring out whether I was looking at extra == '...'
or extra != '...'
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess it really depends what expressions you are looking for.
Consider the expressions extra != 'x' or (extra == 'x' and b)
and extra != 'x' or b
. Both have equivalent truth tables. So what does it really mean for the expression extra == 'x'
to be in the tree?
A more meaningful question would be to ask, is extra == 'x'
a solution to the expression. Or, is extra == 'x'
required in every solution to the expression. Just looking for the expression itself doesn't seem very useful, because it depends on the internal tree representation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the way you have it written now will visit extra == 'x'
if there is no possible solution to the tree when extra != 'x'
. And it will visit extra != 'x'
when the tree has a potential solution when extra != 'x'
. I'm not sure whether you meant for those checks to be mutually exclusive. Should (extra == 'x' and a) or (extra != 'x' and b)
visit both extra == 'x'
and extra != 'x'
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah this is tricky. I'm not quite sure how to solve this. My suspicion is that those types of expressions don't actually appear in this context. Because it would imply that there's a fork that sometimes includes x
and sometimes excludes x
. I believe that by construction that can't happen.
That's a somewhat precarious invariant to rely on, but I'm not quite sure how to fix it.
The second lock snapshot in particular demonstrates the problem: we have a bunch of `python_version < '0'` in the resolution markers. Ref #9735
We'll need this to be able to decode conflict markers back into a series of inclusion/exclusion rules used during resolution.
This does the work to parse conflict markers back into a series of conflict inclusions and exclusions that can be used during resolution. Fixes #9735
9895d7f
to
099c690
Compare
The bad merge was a result of merging #11293 and #11513. I think even if I had only merged the former, it still would have resulted in a bad merge, since the snapshots hadn't been updated for `provide-extras` additions. This fixes the build failures seen here: https://github.com/astral-sh/uv/actions/runs/13390849790/job/37398029632
The bad merge was a result of merging #11293 and #11513. I think even if I had only merged the former, it still would have resulted in a bad merge, since the snapshots hadn't been updated for `provide-extras` additions. This fixes the build failures seen here: https://github.com/astral-sh/uv/actions/runs/13390849790/job/37398029632
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.6.0` -> `0.6.3` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.6.3`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#063) [Compare Source](astral-sh/uv@0.6.2...0.6.3) ##### Enhancements - Allow quotes around command-line options in `requirement.txt files` ([#​11644](astral-sh/uv#11644)) - Initialize PEP 723 script in `uv lock --script` ([#​11717](astral-sh/uv#11717)) ##### Configuration - Accept multiple `.env` files in `UV_ENV_FILE` ([#​11665](astral-sh/uv#11665)) ##### Performance - Reduce overhead in converting resolutions ([#​11660](astral-sh/uv#11660)) - Use `SmallString` on `Hashes` ([#​11756](astral-sh/uv#11756)) - Use a `Box` for `Yanked` on `File` ([#​11755](astral-sh/uv#11755)) - Use a `SmallString` for the `Yanked` enum ([#​11715](astral-sh/uv#11715)) - Use boxed slices for hash vector ([#​11714](astral-sh/uv#11714)) - Use install concurrency for bytecode compilation too ([#​11615](astral-sh/uv#11615)) ##### Bug fixes - Avoid installing duplicate dependencies across conflicting groups ([#​11653](astral-sh/uv#11653)) - Check subdirectory existence after cache heal ([#​11719](astral-sh/uv#11719)) - Include uppercase platforms for Windows wheels ([#​11681](astral-sh/uv#11681)) - Respect existing PEP 723 script settings in `uv add` ([#​11716](astral-sh/uv#11716)) - Reuse refined interpreter to create tool environment ([#​11680](astral-sh/uv#11680)) - Skip removed directories during bytecode compilation ([#​11633](astral-sh/uv#11633)) - Support conflict markers in `uv export` ([#​11643](astral-sh/uv#11643)) - Treat lockfile as outdated if (empty) extras are added ([#​11702](astral-sh/uv#11702)) - Display path separators as backslashes on Windows ([#​11667](astral-sh/uv#11667)) - Display the built file name instead of the canonicalized name in `uv build` ([#​11593](astral-sh/uv#11593)) - Fix message when there are no buildable packages ([#​11722](astral-sh/uv#11722)) - Re-allow HTTP schemes for Git dependencies ([#​11687](astral-sh/uv#11687)) ##### Documentation - Add anchor links to arguments and options in the CLI reference ([#​11754](astral-sh/uv#11754)) - Add link to environment marker specification ([#​11748](astral-sh/uv#11748)) - Fix missing a closing bracket in the `cache-keys` setting ([#​11669](astral-sh/uv#11669)) - Remove the last edited date from documentation pages ([#​11753](astral-sh/uv#11753)) - Fix readme typo ([#​11742](astral-sh/uv#11742)) ### [`v0.6.2`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#062) [Compare Source](astral-sh/uv@0.6.1...0.6.2) ##### Enhancements - Add support for constraining build dependencies with `tool.uv.build-constraint-dependencies` ([#​11585](astral-sh/uv#11585)) - Sort dependency group keys when adding new group ([#​11591](astral-sh/uv#11591)) ##### Performance - Use an `Arc` for index URLs ([#​11586](astral-sh/uv#11586)) ##### Bug fixes - Allow use of x86-64 Python on ARM Windows ([#​11625](astral-sh/uv#11625)) - Fix an issue where conflict markers could instigate a very large lock file ([#​11293](astral-sh/uv#11293)) - Fix duplicate packages with multiple conflicting extras declared ([#​11513](astral-sh/uv#11513)) - Respect color settings for log messages ([#​11604](astral-sh/uv#11604)) - Eagerly reject unsupported Git schemes ([#​11514](astral-sh/uv#11514)) ##### Documentation - Add documentation for specifying Python versions in tool commands ([#​11598](astral-sh/uv#11598)) ### [`v0.6.1`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#061) [Compare Source](astral-sh/uv@0.6.0...0.6.1) ##### Enhancements - Allow users to mark platforms as "required" for wheel coverage ([#​10067](astral-sh/uv#10067)) - Warn for builds in non-build and workspace root pyproject.toml ([#​11394](astral-sh/uv#11394)) ##### Bug fixes - Add `--all` to `uvx --reinstall` message ([#​11535](astral-sh/uv#11535)) - Fallback to `GET` on HTTP 400 when attempting to use range requests for wheel download ([#​11539](astral-sh/uv#11539)) - Prefer local variants in preference selection ([#​11546](astral-sh/uv#11546)) - Respect verbatim executable name in `uvx` ([#​11524](astral-sh/uv#11524)) ##### Documentation - Add documentation for required environments ([#​11542](astral-sh/uv#11542)) - Note that `main.py` used to be `hello.py` ([#​11519](astral-sh/uv#11519)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xNzEuMiIsInVwZGF0ZWRJblZlciI6IjM5LjE3OS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
The bad merge was a result of merging astral-sh#11293 and astral-sh#11513. I think even if I had only merged the former, it still would have resulted in a bad merge, since the snapshots hadn't been updated for `provide-extras` additions. This fixes the build failures seen here: https://github.com/astral-sh/uv/actions/runs/13390849790/job/37398029632
This PR fixes a problem where re-running
uv lock
could, in somecases, exponentially expand the contents of a lock file with more and
more resolution markers. This can only happen when conflict markers are
enabled.
The problem here is a bit in the weeds, but basically, during
resolution, we only use conflict exclusions. That's because, during
resolution, we don't really have state that tells us whether a
particular extra (or group) is enabled at this point in the graph (and
I'm not sure such a thing is possible). Instead, we simply do not
follow edges corresponding to a package and an extra/group that have
been marked as an exclusion for the current fork. However, we do keep
track of inclusion rules as we go.
Once resolution completes, we translate our inclusion and exclusion
rules for each fork into our "universal" marker that combines standard
PEP 508 markers with conflict markers (the result is still purpotedly
valid PEP 508, but somewhat of a bastardization).
Now, if you then change your
pyproject.toml
in a way that causesresolution to run again, we read the markers serialized in the lock
file as input to resolution. This helps retain stability (which is
not a concern specific to conflict markers). However, this caused
resolution to work with both inclusion and exclusion rules inside the
marker itself. Which... kinda works, but it's inconsistent with only
respecting exclusion rules. The end result is that we end up producing
extra forks that aren't technically possible, but this isn't known
until all of the constraints are combined at the end of resolution. In
reality, we should never create these forks in the first place.
The conflict forking logic does take impossible forks into account, but
because of how conflict markers were filtering back into resolution,
this wasn't working properly.
So in this PR, we fix the problem by deserializing the conflict marker
back into conflicting inclusion/exclusion rules. Resolution then starts
with these rules in the first place and our logic for not visiting
impossible forks (which were manifesting as
python_version < '0'
inthe lock file) applies.
Fixes #9735