Pip releases should test against all versions of Python for Windows

[notatallshaw]

What's the problem this feature will solve?

Currently for regular PRs pip only tests against the oldest supported Python version and the newest supported Python version:

pip/.github/workflows/ci.yml

Lines 175 to 182 in ec5faea

	- "3.8"
	# Commented out, since Windows tests are expensively slow,
	# only test the oldest and newest Python supported by pip
	# - "3.9"
	# - "3.10"
	# - "3.11"
	# - "3.12"
	- "3.13"

But when pip does a final release I think it would be reassuring for all versions of Python to be tested against.

Describe the solution you'd like

Somehow in the GitHub workflow file have a condition that runs all untested versions of Python for the final release.

Alternative Solutions

Do nothing.

Additional context

I don't know how to implement this yet, but if no one else does this I plan to spend a bit of time investigating this in the future and raise a PR. I'm guessing have a job associated with a specific path filter on NEWS.rst that tests these versions of Python on Windows.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[sbidoul]

The release process creates a branch named release/YY.N(.P), to using that to determine that more tests should run could be an option. Although as a RM I wouldn't want to discover issues at that stage.

Maybe a scheduled run of the pipeline with more tests enabled instead?

[pfmoore]

Although as a RM I wouldn't want to discover issues at that stage.

Agreed. I think if we want to test against all Python versions, we should be doing that sufficiently before the release that we have the time to assess and fix any issues before starting the release. And in particular, anything that makes it the RM's responsibility to deal with failures here isn't reasonable - we recently changed policy to merge vendoring updates well in advance of a release in order to remove the risk that they cause issues with the release. Any testing of all Python versions should follow the same principle.

Maybe a scheduled run of the pipeline with more tests enabled instead?

That sounds like a better idea. Maybe run it quarterly, in the month before each release is due?

[notatallshaw]

There's already a weekly scheduled test run:

pip/.github/workflows/ci.yml

Lines 11 to 12 in ec5faea

	schedule:
	- cron: 0 0 * * MON # Run every Monday at 00:00 UTC

Assuming there's a way to distguish that in the workflow file, seems like this would be a reasonable place to add these test runs based on feedback.

[pradyunsg]

We'd dropped this because Windows CI was exceedingly slow and we were hitting issues due to low quotas for the CI pipelines. I think we can reasonably remove that limitation now.

[ichard26]

IIRC we're on a [much larger] organisation plan now. Perhaps it would be easiest to run the full suite of Windows jobs across the entire Python version matrix. I'm a little worried that weekly runs may have poor visibility (AFAIK it's only maintainers that get notifications if they fail).