make regen-all is not deterministic

[vstinner]

Bug report

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

A clear and concise description of the bug:

The make regen-all command is not deterministic. Depending in the order in which the target dependencies are executed, the result change.

regen1.sh:

set -e -x

git checkout .
git checkout 79823c103b66030f10e07e04a5462f101674a4fc

# WARNING: it removes all untracked files!
git clean -fdx

./configure --with-pydebug

make regen-cases
make regen-typeslots
make regen-token
make regen-ast
make regen-keyword
make regen-sre
make regen-frozen
make regen-pegen-metaparser
make regen-pegen
make regen-test-frozenmain
make regen-test-levenshtein

make clinic
make regen-global-objects

echo
grep 'ID(traceback)' Python/deepfreeze/deepfreeze.c || true
echo
wc -l Python/deepfreeze/deepfreeze.c

regen2.sh:

set -e -x

git checkout .
git checkout 79823c103b66030f10e07e04a5462f101674a4fc

# WARNING: it removes all untracked files!
git clean -fdx

./configure --with-pydebug

make clinic
make regen-global-objects

make regen-cases
make regen-typeslots
make regen-token
make regen-ast
make regen-keyword
make regen-sre
make regen-frozen
make regen-pegen-metaparser
make regen-pegen
make regen-test-frozenmain
make regen-test-levenshtein

echo
grep 'ID(traceback)' Python/deepfreeze/deepfreeze.c || true
echo
wc -l Python/deepfreeze/deepfreeze.c

regen1.sh output:

(...)

            &_Py_ID(traceback),
            &_Py_ID(traceback),

146433 Python/deepfreeze/deepfreeze.c

regen2.sh output:

(...)

146543 Python/deepfreeze/deepfreeze.c

• Using regen1.sh, there is a traceback identfier in Python/deepfreeze/deepfreeze.c.
• Using regen1.sh, there is no traceback identfier in Python/deepfreeze/deepfreeze.c.

These commands are coming from regen-all in Makefile.pre.in:

regen-all: regen-cases regen-typeslots \
        regen-token regen-ast regen-keyword regen-sre regen-frozen clinic \
        regen-pegen-metaparser regen-pegen regen-test-frozenmain \
        regen-test-levenshtein regen-global-objects

Linked PRs

• gh-108740: Fix "make regen-all" race condition #108741
• [3.12] gh-108740: Fix "make regen-all" race condition (#108741) #109019
• [3.11] gh-108740: Fix "make regen-all" race condition (#108741) #109021

[vstinner]

Simpler reproducer.

regen1.sh:

set -e -x

make clean

make regen-global-objects
make Python/deepfreeze/deepfreeze.c

echo "=== GLOBAL OBJECTS ==="
echo
grep 'ID(traceback)' ./Include/internal/pycore_global*.h ||:
echo
echo "=== DEEPFREEZE ==="
echo
grep 'ID(traceback)' Python/deepfreeze/deepfreeze.c ||:
echo
wc -l Python/deepfreeze/deepfreeze.c

VS

regen2.sh:

set -e -x

make clean

make Python/deepfreeze/deepfreeze.c
make regen-global-objects

echo "=== GLOBAL OBJECTS ==="
echo
grep 'ID(traceback)' ./Include/internal/pycore_global*.h ||:
echo
echo "=== DEEPFREEZE ==="
echo
grep 'ID(traceback)' Python/deepfreeze/deepfreeze.c ||:
echo
wc -l Python/deepfreeze/deepfreeze.c

Depending if regen-global-objects is run before or after Python/deepfreeze/deepfreeze.c is built / updated, the result is different. I suppose that it's a missing dependency in Makefile.