`lib/python3.12: File exists` when doing `make -jN install`

[indygreg]

Bug report

Bug description:

On CPython 3.12.0rc3 I am reliably seeing a new apparent race condition in the build system.

On prior CPython releases, make -jN install would reliably work. On 3.12, it no longer does.

Here is example build output:

To find the necessary bits, look in configure.ac and config.log.

Checked 111 modules (31 built-in, 73 shared, 4 n/a on macosx-13.5-x86_64, 0 disabled, 3 missing, 0 failed on import)
Creating directory /tools/host/lib/python3.12
Creating directory /tools/host/lib/python3.12/config-3.12-darwin
Creating directory /tools/host/lib/python3.12
install: mkdir /private/var/folders/dd/xb3jz0tj133_hgnvdttctwxc0000gn/T/tmpsgqb2dup/out/tools/host/lib/python3.12: File exists
Creating directory /tools/host/lib/python3.12/lib-dynload
make: *** [libainstall] Error 71
make: *** Waiting for unfinished jobs....
Creating directory /tools/host/lib/python3.12/asyncio

The Creating directory %s log message comes from CPython's Makefile.pre.in and we can see from these logs that there appear to be 2 callers racing to create lib/python3.12 around the same time.

Steps to reproduce are just a simple ./configure + make -jN install, where N is say 8.

I can reliably reproduce this failure on macOS 13.5.2 and it might be due to a buggy /usr/bin/install -d implementation on macOS. I think the problem is /usr/bin/install -d doesn't swallow/ignore EEXIST like mkdir -p does.

I want to say I encountered limitations/bugs in macOS /usr/bin/install a few years back when I was a maintainer of Firefox's build system.

CPython versions tested on:

3.12

Operating systems tested on:

macOS

[indygreg]

I can reproduce via make -jN && make -jN install. So the race is in the install target.

I can work around by evaluating a make target creating the lib/pythonX.Y directory before calling into make install. e.g. make -jN sharedinstall && make -jN install works since sharedinstall will create the racy directory.

[AA-Turner]

Are you able to bisect to a specific commit (or at least to between two alpha/beta/rc releases)?

A

[indygreg]

I looked at the Git history for suspect diffs and "bisected" it to commit 2a8bf25 as my first guess. Commit 35dd550 works. 2a8bf25 does not.

So gh-100221 / gh-100329 introduced the regression.

[AA-Turner]

cc: @zware @mgorny

[mgorny]

Perhaps it could be resolved by moving the directory creation into a common pseudotarget.