Convert change detection to a Python script

[AA-Turner]

The change detection workflow is becoming increasingly complex.

We have grown from a fairly simple grep command to skip documentation files (#19983) to a large and complex shell script (embedded within a YAML document), indeed one sufficiently complex to merit a dedicated workflow file (#122336)

A potted history of significant revisions is thus:

• Skip CI expensive checks on CODEOWNERS update #128754
• gh-122965: Fix reusable-change-detection.yml on workflow_dispatch #122966
• Move change detection to separate workflow in CI #122336
• gh-119146: Update regexp in build.yml to not trigger the jobs on *.md and *.ini files. #120435
• gh-60283: Check for redefined test names in CI #109161
• gh-107652: Set up CIFuzz to run fuzz targets continuously #107653
• GH-86275: Implementation of hypothesis stubs for property-based tests, with zoneinfo tests #22863
• bpo-40548: Fix "Check for source changes (pull_request)" GH Action job #21806
• bpo-40548: skip build/tests on documentation changes #19983

There are further proposed changes, such as skipping Windows tests on changes to the Unix build configuration:

• Skip Windows CI for configure/Makefile changes #128446
• gh-128446: Skip Windows CI for Unix build system changes #128450

Even having recently improved readability of the central grep command (#128754), this workflow remains difficult to correctly modify (#128450).

This PR converts the core logic into a short Python script, Tools/build/compute-changes.py, which determines which workflows to run. I imagine this will make it easier to introduce future conditional workflows, which we should probably adopt more of to reduce time and resources spent waiting for CI.

I have reused the "changed files" logic introduced in #108065, meaning we can also combine the duplicative MSI and Docs changes steps, which reduces the overall work done.

I've tested this quite a bit on my fork and detection works well, as does workflow dispatch.

A

cc @webknjaz (sorry for the ping; I can't request-review)

[AA-Turner]

Sample workflow run from this branch: https://github.com/python/cpython/actions/runs/13125212512/job/36620039265?pr=129627

[webknjaz]

I'll try to review later today. Thanks for the ping!

[hugovk]

Thank you for this! I was thinking of something like this given the wrangling of #128450 :)

Mostly nits here:

[webknjaz]

@AA-Turner were the CI jobs cancelled by accident? It's not immediately obvious what happened in there…

[webknjaz]

I wonder if this would belong in the same script… In the past I'd just use sha512 to compute hashes in python: https://github.com/ansible/awx-plugins/blob/0d569b5/.github/workflows/ci-cd.yml#L222C16-L222C52.

If not, I'd question if this reusable workflow should even be called “change detection”. I think I tend to call the computation job “pre-setup” or something, since the changes isn't the only thing being detected…

[AA-Turner]

I'd question if this reusable workflow should even be called “change detection”.

"reusable-choose-workflows"?

I wonder if this would belong in the same script

We could move it to compute-changes.py, sure. It seems it should be possible to replicate the same output as hashFiles

[webknjaz]

"reusable-choose-workflows"?

"reusable-build-settings"? "reusable-settings"? "reusable-workflow-run-context"?

[webknjaz]

We could move it to compute-changes.py, sure. It seems it should be possible to replicate the same output as hashFiles

Totally, although, it doesn't really matter if it's the same given the context where it's being used. It just has to be unique and predictable. Hashing is only used because you can't put entire file contents into cache keys 🤷‍♂️

[AA-Turner]

If we change the key, all current caches will be invalidated?

[webknjaz]

Yes, but they won't be picked up after this PR anyway because they depend on the contents of .github/workflows/build.yml which you're changing here. You're already invalidating the cache.

[cmaloney]

As a note, this change made it so PRs need to merge in main to get the script otherwise the github workflow tries running a script which doesn't exist and fails. I really like having it in Python, but is a hazard that breaking changes to the script may break outstanding PRs until they merge in main.

[AA-Turner]

Do you have a link to an example run? Not ideal, but it should be fixable by clicking the update branch button.

[cmaloney]

Update branch button definitely worked for my case and was straightforward to figure out and get working.

#129560 is a PR opened before this change, I made more commits and pushed, that caused this run after compute-changes.py was committed https://github.com/python/cpython/actions/runs/13163983268/job/36739530516, then I did "update branch" and https://github.com/python/cpython/actions/runs/13164025409/job/36739667617 run happened and succeeded

[webknjaz]

That's weird. actions/checkout usually gets the merged-in changes into jobs..

[hugovk]

Yeah, we often have to "Update branch" after making CI checks stricter, for example, recently when making Blurb validation stricter.

[miss-islington-app]

Thanks @AA-Turner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

[miss-islington-app]

Thanks @AA-Turner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @AA-Turner, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 7d9a22f50923309955a2caf7d57013f224071e6e 3.12

[miss-islington-app]

Sorry, @AA-Turner, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 7d9a22f50923309955a2caf7d57013f224071e6e 3.13

[bedevere-app]

GH-130367 is a backport of this pull request to the 3.13 branch.

[bedevere-app]

GH-130370 is a backport of this pull request to the 3.12 branch.

[webknjaz]

Yeah, we often have to "Update branch" after making CI checks stricter, for example, recently when making Blurb validation stricter.

I meant that this is supposed to be happening automatically. Just restarting the jobs should be enough.