Annotate test cases which are timing sensitive

[rossburton]

Feature or enhancement

Proposal:

Many tests in the cpython test suite are sensitive to timing and assume an entirely unloaded system. For example, the TimerfdTests check for 1ms accuracy in expected durations. Some people (eg people building and testing python) may be running the test suite on build machines which are potentially heavily loaded so this sort of expectation isn't feasible.

It seems like a good solution to this would be to annotate the tests which are timing sensitive in some way, so they can be skipped easily.

For reference, we're manually patching python to skip tests which are failing under load but we're now having to rebase these patches on upgrades:

• https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-Skip-failing-tests-due-to-load-variability-on-YP-AB.patch
• https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-skip-no_stdout_fileno-test-due-to-load-variability.patch
• https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-test_active_children-skip-problematic-test.patch
• https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-test_deadlock-skip-problematic-test.patch
• https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-test_shutdown-skip-problematic-test.patch
• https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-test_storlines-skip-due-to-load-variability.patch

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

• gh-130363: add a test resource to mark tests that need an idle system #130508

[encukou]

Sounds like it could be a new resource, see python -m test --help and test.support.requires.

[tomasr8]

I think it makes sense to hide the tests that clearly depend on timings (e.g. test_process_time) behind a test resource. Though for some of the other tests (e.g. test_input_no_stdout_fileno) it's not immediately obvious to me why they would fail, so perhaps instead of skipping them we could look into making them more reliable?

[rossburton]

Agreed - we've a short list of known flaky tests, and need to triage it to identify the ones which are genuinely sensitive vs those with subtle race conditions which should be fixed.

I will prototype a new resource and see how it works.

[tomasr8]

If it helps, to add a new resource, you just need to add it here:

cpython/Lib/test/libregrtest/utils.py

Lines 35 to 36 in d8ce092

	ALL_RESOURCES = ('audio', 'curses', 'largefile', 'network',
	'decimal', 'cpu', 'subprocess', 'urlfetch', 'gui', 'walltime')

And then you can annotate the tests with @support.requires_resource('your_new_resource')

[picnixz]

Though for some of the other tests (e.g. test_input_no_stdout_fileno) it's not immediately obvious to me why they would fail, so perhaps instead of skipping them we could look into making them more reliable?

How about a flaky resources. At least, we know which tests are meant to be flaky and which are not. And for the CI, we could ask the CI to rerun flaky tests on failures so that contributors don't need to wait for a triager/core dev to rerun a failed workflow when needed.

I think it makes sense to hide the tests that clearly depend on timings (e.g. test_process_time) behind a test resource

I suggest adding stats resources which could cover all tests that are meant to gather timing and check that some condition is statistically verified. Other possible names that are more or less generic: timeprec, measures, experiments. I don't really have better names =/

[tomasr8]

I like the flaky test idea! For timing-related tests, I don't have a good suggestion either, though stats seems a bit too generic to me (but I guess how generic we want this resource to be). timeprec is more self-evident, or maybe timing or timings?

[picnixz]

I like the flaky test idea!

Let's open a separate issue for the flaky tests as it could be orthogonal to tests that require an idle system due to stats gathering.

I don't have a good suggestion either, though stats seems a bit too generic to me (but I guess how generic we want this resource to be). timeprec is more self-evident, or maybe timing or timings?

I suggested "stats" because I don't think we should have a test that doesn't check against a population that some hypothesis is verified. However, I'm not sure we have any test that would test something else than timings like this. Memory cannot really be flaky unless the kernel is doing some weird stuff and network can fail but generally it's a timeout issue, not a data issue itself. So timings looks quite promising.

[tomasr8]

Let's open a separate issue for the flaky tests as it could be orthogonal to tests that require an idle system due to stats gathering.

Issue's here: #130474

I suggested "stats" because I don't think we should have a test that doesn't check against a population that some hypothesis is verified. However, I'm not sure we have any test that would test something else than timings like this.

Yeah, I'm not sure if we assert anything else besides timings that is similarly stochastic.. maybe we can stick with some time-related name for now and adjust it if something else comes up?

[rossburton]

Posted a minimal PR: #130508. This just adds an "idle" resource.