Abort immediately when there's a build failure, instead of backtracking

[pradyunsg]

If you try to install a package that does not have wheels for your platform and you don't have the relevant build dependencies for it, pip will backtrack until it finds a version that it can build OR it has exhausted all the available versions of that package.

This can be especially painful when you're missing the build dependencies (in case of https://discuss.python.org/t/pip-without-setuptools-could-the-experience-be-improved/11810, it is setuptools) but in other cases, it can be a compiler / C library etc.

Would it make sense to fail immediately, instead of backtracking in these cases?

[pradyunsg]

This can be especially painful

To elaborate on this: here's a thought experiment -- a Cython-based project my-cython-wrapper that doesn't have wheels for your platform, which fails when trying to link the compiled package because you don't have the required C library. Each failure involves a download, building the isolated environment, metadata generation, doing a Cython compilation, compiling a bunch of C/C++ code and then... it fails. And then pip will go to an older version and try again, going all the way until it exhausts the available versions of my-cython-wrapper.

[DiddiLeija]

This could be good default behavior. I have experimented this issue in the past -- and yes, it can be really painful. I want to suggest some ideas:

• A non-default option to "backtrack instead of crashing", if the user prefers to backtrack and re-build everything. Maybe something like --backtrack-build-failure?
• An explanative documentation about this behavior. When the build abort, we could add a link that redirects to that page, so users won't get confused and open innecessary issues on GitHub...

[pradyunsg]

I mean, I think we'd need to have this go through a pass-a-flag-to-opt-out period.

As for the documentation, I think we should be able to get away with a clear-enough error message. With the progress I've made on #10421 so far, I'm pretty confident that we should be able to get most of the way there with just the error messages.

[pfmoore]

Would it make sense to fail immediately, instead of backtracking in these cases?

But doesn't that mean that if package X has wheels for your platform at version 1.0, and they release 2.0, initially without wheels because building on your platform is a PITA (ahem Windows), then rather than getting the 1.0 wheel you'd start getting failures? That seems like a rather bad regression...

[pradyunsg]

Yea... the "nice" thing about that though, is that it is an immediate failure where you have a clear actionable step, rather than a long-winded one, or something that subtly hides issues (eg: newest version of $package added a C dependency that you don't have, and you're quietly getting the older version even though you didn't intend to).

It's a much more explicit failure model (and matches what we had prior to the backtracking resolver, in a good way).

[pfmoore]

I'm still confused. At the moment, using the X example I gave above, if I say pip install X, I get X. Maybe not the latest version, but I get it. You seem to be suggesting that it's better for me for pip install X to suddenly start failing, and... what? I have to say pip install X==1.0 and keep an eye on when X release a 2.0 wheel so that I know when to remove my pin?

Maybe that works if you're talking about a dependency somewhere deep in a project's requirements, but it seems to me that it sucks for requirements specified directly on the command line.

And yes, IMO it was bad when this happened with the old resolver, too.

[pradyunsg]

You seem to be suggesting that it's better for me for pip install X to suddenly start failing,

No. I don't think it's "better". I agree that it's still suboptimal. I also don't think we have any "globally optimal" approach here.

To try and rephrase what I said earlier -- we're operating in tradeoffs here, and I think having an immediate failures in all of these cases (both the example you've picked as well as the ones I've provided) is better than having some of them work with backtracking while also having long-winded "resource" intensive failures in other cases.

[pfmoore]

OK, I get your point now. However, I don't think either solution is ideal, and given that's the case, I would prefer to stick with what we have for now, until we work out a genuinely better solution. I do think there's a better solution waiting for someone to discover it, so I'm optimistic that "explore options" is not going to be wasted time.

If you try to install a package that does not have wheels for your platform

Maybe we should make --prefer-binary the default, then? That way, if there's no need to build we won't build. Users won't get newer versions unless they opt into building sdists, or the project gets round to releasing wheels, but that seems like a reasonable compromise. The assumption is that projects don't release wheels for one version, and then stop - but that seems like a reasonable assumption to make, TBH. Obviously this would need a transition period - and it would effectively signal that pip considers installing from wheels to be the "normal approach". That's something that I, personally, am in favour of, but it might not match everyone's preference.

Or maybe we only ever do a single sdist build, and if it fails we switch to --only-binary mode, on the assumption that it's a missing build toolchain, rather than a single broken sdist?

As you see, there are lots of plausible (at least plausible to me 🙂) approaches to explore, so if this issue is "Explore alternatives to repeatedly trying to build older and older sdists" then I'm on board, but if it's specifically "Is abort immediately the approach we should use" then my answer is "no, I'd prefer something else".

One fundamental problem here is that we have no way of knowing how likely it is that building a sdist will work. So the best we can do is work on assumptions and heuristics:

• "Wheels are the main mode of delivering packages" leads to "--prefer-binary as default".
• "If it fails once, it's likely that it will fail every time" leads to "only do one build".
• "The users have said what they want and we should do whatever it takes to deliver it" leads to the current behaviour¹.
• I'm not 100% sure what you'd suggest leads to "abort on build failure" - maybe "users want the latest version or nothing"? But I fear that my biases are showing here, maybe you could come up with a better phrase that explains your approach?

Whatever we do here, we need to be very careful how we report what pip is doing to the user. "No suitable versions found" is a bad message when the user can see there's a valid wheel for version N-1. Or when there's two sdists and the user knows the older one built, but there's a compiler-dependent bug in the new one. We also need to tell the user how to get pip to skip a sdist that won't compile, if we do go with "stop on a build failure" - and I'm not at all clear how we'd do that ("X != 2.0 unless there's a wheel" isn't a valid requirement 🙁).

¹ Of course the underlying problem here is that when people say they want "foo", they probably don't actually mean that - and a 10-year old version of foo wasn't what they were after. Which is just another way of saying "if you lie to the computer, it will get you"... But punishing people for assuming that the computer will behave reaonably isn't a good answer, either.

[pfmoore]

Also, do we have to have a "one size fits all" solution here? Maybe keep searching (or build once, or prefer binary) if it's a top-level requirement, but abort if it's a dependency somewhere down the tree? That has its own problems but might be a viable compromise.

[pradyunsg]

One of the problems with prefer-binary is that we'd prefer to get an older version even if a newer version would build and work successfully. :)

[pradyunsg]

However, I don't think either solution is ideal, and given that's the case, I would prefer to stick with what we have for now, until we work out a genuinely better solution.

I promise I've read the entire post, and... the thing I wanna push back against is the first sentence.

Why is it not better to have eager failures, at a point where we can provide clear information about what the failure is?

• "A non-Python build dependency is missing", then they have a clear actionable information immeditely -- install it and try again!
• "the latest version of X doesn't support my system", then they have a clear actionable information immediately -- look up which is supported, and get a version that is supported.

I understand that this means that pip install X won't get you X if the latest version doesn't support it, and that you'd need to be more specific -- but surely that is not a worse-enough experience to justify requiring other users (or the same user later!) to wait on building all the sdists of a package. Note that we don't just have interactive users, we also have automated users where there's no easy way to interrupt an execution -- and especially in those case, an eager failure would be much better than non-eager failure.

We're operating in trade-offs, and I think we do have a better-than-status-quo answer here -- it isn't perfect, but it's certainly better IMO.