New resolver: “Nameless” constraint

[uranusjr]

Spawned from #8209.

According to test_install_distribution_union_with_constraints, this is a valid constraint:

C:\Temp\packages\foo[bar]

This is quite difficult to deal with. We can store the constraint information in a separate mapping (with the URL as key), but I’m not sure how it can be fetched. Here’s how we get the relevant constraint to use:

def find_matches(self, requirement):
    # type: (Requirement) -> Sequence[Candidate]
    constraint = self._constraints.get(requirement.name, SpecifierSet())
    return requirement.find_matches(constraint)

I guess we can do something like:

def find_matches(self, requirement):
    # type: (Requirement) -> Sequence[Candidate]
    constraint = self._constraints.get(requirement.name, SpecifierSet())
    if isinstance(requirement, ExplicitRequirement) and
            isinstance(requirement.candidate, (LinkCandidate, EditableCandidate)):
        constraint &= self._url_constraints.get(
            requirement.candidate.link.url, SpecifierSet(),
        )
    return requirement.find_matches(constraint)

but we probably can agree this looks super wrong. Some additional abstraction is needed here.

[pfmoore]

One question - do we have any indication that anyone actually uses this functionality? It looks incredibly weird, and I don't see what the purpose of such a constraint is. (I can technically work it out, but why would anyone use it?)

I'd be reasonably comfortable taking the approach of (at least initially) dropping support for constructs like this in the new resolver, and seeing if anyone complains.

It would affect the rollout process and how we handle backward compatibility, though, because if someone does pop up saying they use this, we'd need a workaround for them while we work out how to add it. It also changes the project goal, as "all tests must pass" won't be quite right in that case.

@pradyunsg @brainwane Any thoughts? Is there any value in asking the user survey population "do you use constraints and if so how?" Or is that too small a sample to be worth it?

PS I'm not going to review the rest of the constraint-related issues until after the weekend, but I wanted to add my thoughts to this one to give people a chance to respond.

[uranusjr]

I vaguely remember being taught about mentioned constraints as a workaround to pip’s inability to merge extras (the feature described in #8211), but I assume many such usages are not needed anymore since the new resolver does not have the problem. As for direct URL in constraints, no idea at all.

This is definitely a good case to delibreately not include the feature because YAGNI and see what happens IMO.

[pradyunsg]

100% on board for not implementing this, flagging this as a "this no longer works with the new resolver" line item somewhere, and seeing if anyone comes complaining about this during the beta. :)

[dstufft]

Constraints were not a work-around for merging extras. They were added by Openstack because they want to make sure that the entirety of openstack is installable together, without having to either maintain 20 separate requirements.txt files OR forcing the install of literally all of Openstack.

See #2731 for a tiny bit more information. We would definitely want to reach out to the Openstack community to see if they're still using it.

[dstufft]

Actually I think that the feature described in #8211 is wrong? At least the inteded purpose of constraint files were to not cause anything extra to be installed, but to only apply additional constraints to whatever would ultimately get installed.

[uranusjr]

It is my understanding as well. Sorry if my writing is causing misunderstandings, please feel free to add to/modify the issue.

[pfmoore]

We would definitely want to reach out to the Openstack community to see if they're still using it.

Openstack are actively testing the constraints implementation in the new resolver (@dhellmann contacted us to help out) so based on your comment, I'd suggest that we wait for feedback from them on whether their testing showed up any issues, and review these issues in the light of that feedback).

I wonder if these tests were mistakenly added from a "completeness" point of view ("because you can add any requirement in a constraints file, we need to decide what foo[bar] means in a constraints file") rather than focusing on what was requested ("we should only allow version specifiers in a constraints file, so other types of requirement should be invalid or at least not supported")?

[dhellmann]

As far as I know, OpenStack always uses package names, one rule with == or === and a version per constraint entry, and optionally an expression indicating which version of python needs the constraint (I can never remember the name of that feature, sorry). This allows the test jobs to "pin" to the same versions of a package across tests that have different combinations of applications installed.

The global constraints file for OpenStack is at https://opendev.org/openstack/requirements/src/branch/master/upper-constraints.txt if you want to see an example.

Most projects also have a per-project constraints file using the lower bounds for their requirements, used for a separate unit test job and possibly an application-specific functional test job. We do not run the integration tests with the lower bounds constraints. For an example of one of these files, have a look at https://opendev.org/openstack/nova/src/branch/master/lower-constraints.txt There are going to be a lot of those, since every repo will have its own.

I have asked specifically about nameless constraints on the mailing list http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014790.html

/cc @prometheanfire

[prometheanfire]

we currently do not allow nameless constraints or requirements. As I understand it, those are git/http based uris? we set permit_urls=False

https://opendev.org/openstack/requirements/src/branch/master/openstack_requirements/requirement.py#L91

[uranusjr]

@dhellmann @prometheanfire Could you aldo help verify whether there are currently projects depending on the extras behaviour, as described in #8211?

[jrosser]

I'm a bit unsure about the precise meaning of 'nameless' and if this is coupled to the discussion around extras. However I will outline the way openstack-ansible uses constraints files to install a mixture of packages and source code just to make sure that this is captured.

Example is here http://paste.openstack.org/show/793386/ but short story is we use constraints like "git+https://opendev.org/openstack/glance@6e3ced8251cd6e273aa73f553a24fc475b219db5#egg=glance" and put URLs to remote constraints in constraints files all the time.