gh-115421: Test that our Makefile has all needed test folders

[sobolevn]

This PR is the alternative to #115511

While we would still have to keep adding directories to TESTSUBDIRS manually, we would now have a test to catch any cases where people forget to update / remove these entries.

This test works with --disable-test-modules as well.

Which approach do you like best?

• Issue: Not all tests are installed. #115421

[sobolevn]

I've pushed a demo commit: f048fb5

It is missing test/archivetestdata from Makefile, some tests should fail. Let's see!

[sobolevn]

Correctly failed jobs (which detected missing test/archivetestdata in Makefile):

• Macos 14: https://github.com/python/cpython/actions/runs/8004529977/job/21862119274?pr=115813
• Macos 13: https://github.com/python/cpython/actions/runs/8004529977/job/21862119510?pr=115813
• Macos 14 (free-threaded): https://github.com/python/cpython/actions/runs/8004529977/job/21862119752?pr=115813
• Macos 13 (free-threaded): https://github.com/python/cpython/actions/runs/8004529977/job/21862119980?pr=115813
• Address Sanitizer (Ubuntu): https://github.com/python/cpython/actions/runs/8004529977/job/21862116610?pr=115813

Now, I will revert test/archivetestdata change and run all buildbots.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @sobolevn for commit 182822d 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[sobolevn]

Related failure:

======================================================================
FAIL: test_makefile_test_folders (test.test_tools.test_makefile.TestMakefile.test_makefile_test_folders)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/buildbot/buildarea/pull_request.ware-freebsd/build/Lib/test/test_tools/test_makefile.py", line 79, in test_makefile_test_folders
    self.check_existing_test_modules(actual_result)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/pull_request.ware-freebsd/build/Lib/test/test_tools/test_makefile.py", line 61, in check_existing_test_modules
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        unique_test_dirs.symmetric_difference(set(used)),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        set(),
        ^^^^^^
    )
    ^
AssertionError: Items in the first set but not the second:
''

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @sobolevn for commit 001c14b 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[sobolevn]

@Yhg1s it is ready :)

[encukou]

LGTM, with two suggestions:

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @encukou for commit 6caa8f9 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[sobolevn]

Thanks everyone!

[bedevere-app]

GH-116462 is a backport of this pull request to the 3.12 branch.

[bedevere-app]

GH-116462 is a backport of this pull request to the 3.12 branch.

[bedevere-app]

GH-116498 is a backport of this pull request to the 3.11 branch.