Tests fail if `--disable-test-modules` is supplied

[chaosflaws]

Bug report

Bug description:

I compiled python 3.12 on a Debian system with --enable-optimizations and --disable-test-modules, and during the profile-guided optimization tests, multiple tests failed (see below).

Compiling python without --disable-test-modules makes the tests pass, and the output indicates that the failures are due to the test API not being available.

I suppose it's not too big of a deal, but this might normalize seeing errors when building python, leading users to ignore actual test failures.

# Next, run the profile task to generate the profile information.
LD_LIBRARY_PATH=/home/chaosflaws/Python-3.12.2 ./python -m test --pgo --timeout=1200 || true
Using random seed: 3095374302
0:00:00 load avg: 1.00 Run 44 tests sequentially (timeout: 20 min)
0:00:00 load avg: 1.00 [ 1/44] test_array
0:00:05 load avg: 1.00 [ 2/44] test_base64
0:00:07 load avg: 1.00 [ 3/44] test_binascii
0:00:08 load avg: 1.00 [ 4/44] test_binop
0:00:08 load avg: 1.00 [ 5/44] test_bisect
0:00:09 load avg: 1.00 [ 6/44] test_bytes
0:00:09 load avg: 1.00 [ 7/44] test_bz2 -- test_bytes skipped
0:00:12 load avg: 1.08 [ 8/44] test_cmath
0:00:13 load avg: 1.08 [ 9/44] test_codecs
0:00:20 load avg: 1.07 [10/44] test_collections
0:00:25 load avg: 1.07 [11/44] test_complex
0:00:27 load avg: 1.06 [12/44] test_dataclasses
0:00:29 load avg: 1.06 [13/44] test_datetime
0:00:44 load avg: 1.05 [14/44] test_decimal
0:01:08 load avg: 1.03 [15/44] test_difflib
0:01:14 load avg: 1.03 [16/44] test_embed
test test_embed failed
0:01:28 load avg: 1.02 [17/44] test_float -- test_embed failed (37 errors, 3 failures)
0:01:30 load avg: 1.02 [18/44] test_fstring
0:01:34 load avg: 1.02 [19/44] test_functools
0:01:37 load avg: 1.02 [20/44] test_generators
0:01:39 load avg: 1.02 [21/44] test_hashlib
0:01:41 load avg: 1.02 [22/44] test_heapq
0:01:45 load avg: 1.02 [23/44] test_int
test test_int failed
0:01:47 load avg: 1.01 [24/44] test_itertools -- test_int failed (2 errors)
0:02:15 load avg: 1.01 [25/44] test_json
0:02:26 load avg: 1.08 [26/44] test_long
0:02:44 load avg: 1.14 [27/44] test_lzma
0:02:45 load avg: 1.14 [28/44] test_math
0:03:02 load avg: 1.10 [29/44] test_memoryview
0:03:05 load avg: 1.10 [30/44] test_operator
0:03:05 load avg: 1.10 [31/44] test_ordered_dict
0:03:13 load avg: 1.08 [32/44] test_patma
0:03:14 load avg: 1.08 [33/44] test_pickle
0:03:48 load avg: 1.05 [34/44] test_pprint -- test_pickle passed in 34.1 sec
0:03:50 load avg: 1.05 [35/44] test_re
0:03:54 load avg: 1.04 [36/44] test_set
0:04:24 load avg: 1.02 [37/44] test_sqlite3
Failed to import test module: test.test_sqlite3.test_dbapi
Traceback (most recent call last):
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dbapi.py", line 37, in <module>
    from _testcapi import INT_MAX, ULLONG_MAX
ModuleNotFoundError: No module named '_testcapi'

Failed to import test module: test.test_sqlite3.test_dump
Traceback (most recent call last):
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dump.py", line 5, in <module>
    from .test_dbapi import memory_database
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dbapi.py", line 37, in <module>
    from _testcapi import INT_MAX, ULLONG_MAX
ModuleNotFoundError: No module named '_testcapi'

Failed to import test module: test.test_sqlite3.test_hooks
Traceback (most recent call last):
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_hooks.py", line 29, in <module>
    from test.test_sqlite3.test_dbapi import memory_database, cx_limit
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dbapi.py", line 37, in <module>
    from _testcapi import INT_MAX, ULLONG_MAX
ModuleNotFoundError: No module named '_testcapi'

Failed to import test module: test.test_sqlite3.test_regression
Traceback (most recent call last):
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_regression.py", line 31, in <module>
    from test.test_sqlite3.test_dbapi import memory_database, cx_limit
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dbapi.py", line 37, in <module>
    from _testcapi import INT_MAX, ULLONG_MAX
ModuleNotFoundError: No module named '_testcapi'

Failed to import test module: test.test_sqlite3.test_transactions
Traceback (most recent call last):
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_transactions.py", line 30, in <module>
    from test.test_sqlite3.test_dbapi import memory_database
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dbapi.py", line 37, in <module>
    from _testcapi import INT_MAX, ULLONG_MAX
ModuleNotFoundError: No module named '_testcapi'

Failed to import test module: test.test_sqlite3.test_userfunctions
Traceback (most recent call last):
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_userfunctions.py", line 35, in <module>
    from test.test_sqlite3.test_dbapi import cx_limit
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dbapi.py", line 37, in <module>
    from _testcapi import INT_MAX, ULLONG_MAX
ModuleNotFoundError: No module named '_testcapi'

0:04:25 load avg: 1.02 [38/44] test_statistics -- test_sqlite3 failed (uncaught exception)
0:04:38 load avg: 1.02 [39/44] test_struct
0:04:41 load avg: 1.02 [40/44] test_tabnanny
0:04:43 load avg: 1.02 [41/44] test_time
0:04:46 load avg: 1.02 [42/44] test_unicode
0:04:59 load avg: 1.01 [43/44] test_xml_etree
0:05:02 load avg: 1.01 [44/44] test_xml_etree_c

Total duration: 5 min 8 sec
Total tests: run=8,196 failures=3 skipped=267
Total test files: run=44/44 failed=3 skipped=1
Result: FAILURE

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-116303: Skip test module dependent tests if test modules are disabled #116307
• gh-116303: Explicitly check for the _testsinglephase module in configure.ac #116479
• gh-116303: Handle disabled test modules in test.support #116482
• gh-116303: Skip test module dependent tests if test modules are unavailable #117341
• gh-116303: Skip tests if C recursion limit is unavailable #117368
• gh-116303: Don't build xxlimited and xxlimited_35 if --disable-test-modules is given #117554

[erlend-aasland]

The test fails because _testcapi is not available. This should fix it:

diff --git a/Lib/test/test_sqlite3/test_dbapi.py b/Lib/test/test_sqlite3/test_dbapi.py
index 588272448b..e46c62cce8 100644
--- a/Lib/test/test_sqlite3/test_dbapi.py
+++ b/Lib/test/test_sqlite3/test_dbapi.py
@@ -31,11 +31,10 @@
 
 from test.support import (
     SHORT_TIMEOUT, check_disallow_instantiation, requires_subprocess,
-    is_apple, is_emscripten, is_wasi
+    is_apple, is_emscripten, is_wasi, import_helper
 )
 from test.support import gc_collect
 from test.support import threading_helper
-from _testcapi import INT_MAX, ULLONG_MAX
 from os import SEEK_SET, SEEK_CUR, SEEK_END
 from test.support.os_helper import TESTFN, TESTFN_UNDECODABLE, unlink, temp_dir, FakePath
 
@@ -1200,6 +1199,7 @@ def test_blob_seek_and_tell(self):
         self.assertEqual(self.blob.tell(), 40)
 
     def test_blob_seek_error(self):
+        testcapi = import_helper.import_module("_testcapi")
         msg_oor = "offset out of blob range"
         msg_orig = "'origin' should be os.SEEK_SET, os.SEEK_CUR, or os.SEEK_END"
         msg_of = "seek offset results in overflow"
@@ -1217,9 +1217,9 @@ def test_blob_seek_error(self):
         # Force overflow errors
         self.blob.seek(1, SEEK_SET)
         with self.assertRaisesRegex(OverflowError, msg_of):
-            self.blob.seek(INT_MAX, SEEK_CUR)
+            self.blob.seek(testcapi.INT_MAX, SEEK_CUR)
         with self.assertRaisesRegex(OverflowError, msg_of):
-            self.blob.seek(INT_MAX, SEEK_END)
+            self.blob.seek(testcapi.INT_MAX, SEEK_END)
 
     def test_blob_read(self):
         buf = self.blob.read()
@@ -1374,13 +1374,14 @@ def test_blob_mapping_invalid_index_type(self):
             self.blob["a"] = b"b"
 
     def test_blob_get_item_error(self):
+        testcapi = import_helper.import_module("_testcapi")
         dataset = [len(self.blob), 105, -105]
         for idx in dataset:
             with self.subTest(idx=idx):
                 with self.assertRaisesRegex(IndexError, "index out of range"):
                     self.blob[idx]
         with self.assertRaisesRegex(IndexError, "cannot fit 'int'"):
-            self.blob[ULLONG_MAX]
+            self.blob[testcapi.ULLONG_MAX]
 
         # Provoke read error
         self.cx.execute("update test set b='aaaa' where rowid=1")

[erlend-aasland]

We should have a buildbot for --disable-test-modules.

[erlend-aasland]

test_stable_abi_ctypes may also have the same problem.

[erlend-aasland]

Actually, there's more than a handful of tests failing (around 35 tests). I'll update the PR to cover all cases, but it will take a lot of time as it is tedious work.