test_glob: test_selflink() fails randomly on Linux

[vstinner]

AMD64 RHEL8 Refleaks 3.x:

0:09:26 load avg: 9.08 [406/463/1] test_glob failed (1 failure) -- running (7): (...)
beginning 6 repetitions
123456
.....test test_glob failed -- Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/test_glob.py", line 396, in test_selflink
    self.assertIn(path, results)
AssertionError:
'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/' not found in
{'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'}

The tested path dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link has one less link/ than the first path of the test: 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'.

# Tested path
>>> len('dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'.split('/'))
33

# First (shorted) expected path
>>> len('dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'.split('/'))
34

build: https://buildbot.python.org/all/#/builders/259/builds/892

Linked PRs

• gh-109959: Log the current directory in test_glob.test_selflink() #128122
• gh-109959: attempt to fix hypothesis tests for test_glob #128255
• gh-109959: Skip test_glob.test_selflink() flaky test #128812
• [3.13] gh-109959: Skip test_glob.test_selflink() flaky test (GH-128812) #128821
• [3.12] gh-109959: Skip test_glob.test_selflink() flaky test (GH-128812) #128834
• gh-109959: Remove test_glob.test_selflink() #130551

[vstinner]

I didn't see this failure recently, I close the issue.

[encukou]

Seen recently in a PR CI run.

[ncoghlan]

Dumping the expected value and the set content from the failed run.

The shortest value in the set has one more link/ entry than the expected value that the test is looking for, so it's definitely the same failure mode, but the absolute numbers are different (27 vs 28 here, rather than the 33 vs 34 that Victor saw):

AssertionError:
'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'
not found in
{
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'
}

>>> s = ... pasted set data ...
>>> expected = 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'
>>> expected in s
False
>>> expected + 'link/' in s
True
>>> len(expected.split("/"))
27

[encukou]

I assume it gets up to a max path length, so the OS & CWD will affect that number.

[ncoghlan]

That was my first thought too, but it doesn't seem to be the case, since all the paths in the set were accessible on disk (the set being checked is derived from the glob result). The largest value in the set will be dictated by the OS and CWD, but the missing value isn't the largest one, it's somewhere in the middle. In the latest failure, the remaining 15 iterations would have worked:

>>> for remaining in sorted(s):
...     print(remaining.count("/"), end=", ")
...
27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41,

I also confirmed both tracebacks are pointing at the same assertion:

• Sep 2023 code:

  cpython/Lib/test/test_glob.py

  Line 396 in ae36cd1

  self.assertIn(path, results)

• Jun 2024 code:

  cpython/Lib/test/test_glob.py

  Line 553 in a86e625

  self.assertIn(path, results)

To get the reported failures:

• every entry up to the failing depth was in the set (otherwise the expected value would be shorter, as the loop starts at depth 0)
• this path is shorter than the possible maximum path length, otherwise the set would be empty

[ncoghlan]

I'm tempted to change the way this test works to build a full set of expected values and then do a set comparison rather than checking the entries one by one. Something like:

            # Check "**/" (or platform equivalent) has no duplicates or skipped entries
            results = glob.glob(os.path.join('**', ''), recursive=True)
            self.assertEqual(len(results), len(set(results)))
            results = set(results)
            max_depth = len(results)
            expected_results = {}
            for depth in range(max_depth):
                expected_results.add(os.path.join(*(['dir'] + ['link'] * depth + [''])))
            self.assertEqual(results, expected_results)

(and similarly for the other two search loops)

It won't fix the problem, but it should make it clearer how many entries are actually missing when the test does fail (since it will only show the diff, rather than all the remaining entries after the first missing entry).

[ncoghlan]

It's hard to see any possible source of discrepancies between "directories scanned" and "paths reported" in a recursive glob, since _rlistdir tries to unconditionally recurse down into every reported name: https://github.com/python/cpython/blob/main/Lib/glob.py#L197

The check for whether that recursion operation is valid then appears to just be whether os.scandir throws OSError or not: https://github.com/python/cpython/blob/main/Lib/glob.py#L152

[barneygale]

What a mysterious bug!

The pattern in question ("**/") triggers this bit of code:

cpython/Lib/glob.py

Lines 54 to 60 in 6b280a8

	if not pathname or recursive and _isrecursive(pathname[:2]):
	try:
	s = next(it) # skip empty string
	if s:
	it = itertools.chain((s,), it)
	except StopIteration:
	pass

I suppose it's possible there's a bug in itertools.chain(), or the way iglob() uses it.

(edit: removing the topic-pathlib label as pathlib isn't relevant here)

[vstinner]

Hypothesis tests on Ubuntu: https://github.com/python/cpython/actions/runs/12415820872/job/34663070210?pr=128097

FAIL: test_selflink (test.test_glob.SymlinkLoopGlobTests.test_selflink)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_glob.py", line 543, in test_selflink
    self.assertIn(path, results)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/file' not found in {'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/file', 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/file', 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/file'}