--with-system-libmpdec / --with-system-expat still uses the vendored headers

[hroncok]

Bug report

Python 3.12.0a1 or the main branch. Run:

$ sudo dnf install mpdecimal-devel expat-devel  # or similar for your OS
...
$ git clean -fdx
$ rm -r Modules/_decimal/libmpdec
$ rm -r Modules/expat
$ ./configure --config-cache --enable-shared --with-system-libmpdec --with-system-expat
$ make
...
make: *** No rule to make target 'Modules/_decimal/libmpdec/basearith.h', needed by 'Modules/_decimal/_decimal.o'.  Stop.

$ git restore Modules/_decimal/libmpdec
$ make
...
make: *** No rule to make target 'Modules/expat/ascii.h', needed by 'Modules/pyexpat.o'.  Stop.

This works fine with Python 3.11.0. I suspect somewhere along #94474 or a similar change, this has changed behavior, but I have not tried bisecting yet.

Your environment

• CPython versions tested on: 3.12.0a1 or the main branch
• Operating system and architecture: Fedora Linux x86_64

• PR: gh-98707: Don't let --with-system-libmpdec / --with-system-expat use the vendored headers #98711

• PR: [3.11] gh-98707: configure --with-system-libmpdec and --with-system-expat no longer include vendored headers (GH-98711) #99391

[hroncok]

Bisecting with only Modules/_decimal/libmpdec removed, because I got some other failures with removed Modules/expat on the way.

EDIT: Apparently, that's what you get when you spell expat as expact.

[hroncok]

Indeed, my gut feeling was correct:

81dca70 is the first new commit
commit 81dca70
Author: Christian Heimes @tiran
Date: Thu Jul 14 09:51:49 2022 +0200

gh-93939: Build C extensions without setup.py (GH-94474)

[hroncok]

I am really really bad at autotools, but my best guess is that this is caused by:

cpython/Makefile.pre.in

Line 2584 in de69816

MODULE_PYEXPAT_DEPS=$(LIBEXPAT_HEADERS) @LIBEXPAT_INTERNAL@

As LIBEXPAT_HEADERS contains a list that starts with Modules/expat/ascii.h.

cpython/Makefile.pre.in

Line 2589 in de69816

MODULE__DECIMAL_DEPS=$(srcdir)/Modules/_decimal/docstrings.h $(LIBMPDEC_HEADERS) @LIBMPDEC_INTERNAL@

As LIBMPDEC_HEADERS contains a list that starts with Modules/_decimal/libmpdec/basearith.h.

In both cases, the dependency needs to be made conditional on the relevant --with-system-xxx option.

[hroncok]

Testing a possible fix.

[hroncok]

FTR this was a problem on older Pythons as well, but it was not exposed. #94474 did not introduce it, but it exposed it. Another problem this will most likely fix is:

--- STDERR ---
/tmp/tmpt5xmo3n_/cpython/Objects/obmalloc.c:1449:1: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
 1449 | arena_map_get(block *p, int create)
      | ^~~~~~~~~~~~~
make: *** No rule to make target '/tmp/tmpt5xmo3n_/cpython/Modules/_decimal/libmpdec/basearith.c', needed by 'Modules/_decimal/libmpdec/basearith.o'.  Stop.
make: *** Waiting for unfinished jobs....
---- END ----
ERROR
======================================================================
ERROR: test_freeze_simple_script (test.test_tools.test_freeze.TestFreeze.test_freeze_simple_script)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.11.0/Lib/test/test_tools/test_freeze.py", line 27, in test_freeze_simple_script
    outdir, scriptfile, python = helper.prepare(script, outdir)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/Python-3.11.0/Tools/freeze/test/freeze.py", line 159, in prepare
    _run_quiet([MAKE, '-C', builddir, '-j8'])
  File "/builddir/build/BUILD/Python-3.11.0/Tools/freeze/test/freeze.py", line 25, in _run_quiet
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/Python-3.11.0/Lib/subprocess.py", line 569, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/make', '-C', '/tmp/tmpt5xmo3n_/python-build', '-j8']' returned non-zero exit status 2.
----------------------------------------------------------------------
Ran 1 test in 70.677s
FAILED (errors=1)

EDIT: It does not fix it.