test_idle test_configdialog.HighPageTest failing on macOS

[emmatyping]

Bug report

Saw these failure on a normal macOS 13 runner:

======================================================================
ERROR: test_set_theme_type (idlelib.idle_test.test_configdialog.HighPageTest.test_set_theme_type)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 295, in setUp
    d.set_highlight_target.called = 0
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'method' object has no attribute 'called' and no __dict__ for setting new attributes

======================================================================
ERROR: test_theme_source (idlelib.idle_test.test_configdialog.HighPageTest.test_theme_source)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 295, in setUp
    d.set_highlight_target.called = 0
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'method' object has no attribute 'called' and no __dict__ for setting new attributes

======================================================================
ERROR: tearDownClass (idlelib.idle_test.test_configdialog.HighPageTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 284, in tearDownClass
    del d.set_highlight_target, d.set_color_sample
        ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'HighPage' object has no attribute 'set_highlight_target'. Did you mean: 'highlight_target'?

======================================================================
FAIL: test_delete_custom (idlelib.idle_test.test_configdialog.HighPageTest.test_delete_custom)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_configdialog.py", line 748, in test_delete_custom
    eq(d.custom_theme_on.state(), ())
    ~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Tuples differ: ('background',) != ()

First tuple contains 1 additional elements.
First extra element 0:
'background'

- ('background',)
+ ()

<snip>

Example failed run:
https://github.com/python/cpython/actions/runs/15265517831/job/42930405567

[hugovk]

I restarted it and it passed this time: https://github.com/python/cpython/actions/runs/15265517831

[emmatyping]

Thanks! That's even more confusing though :(

[StanFromIreland]

I have seen this pop up several times over a few months.

cc expert @terryjreedy

[terryjreedy]

Unittest separately reports Errors (unanticipated exceptions) and Failures (assertXXX not true). For test_configdialog.HighTest, there were 4 Failures and 3 Errors (in that order, the opposite of the report order). Including the 1 fail reported above, the failures are summarized as follows, in the order the tests methods are run.

1. test_delete_custom 748: Tuples differ: ('background',) != ()
2. test_load_theme_cfg 310: Tuples differ: ('disabled', 'background') != ('disabled',)
3. test_on_new_color_set 571: '#de0000' != '#3f7cae'
4. test_set_highlight_target 644: Tuples differ: ('background',) != ('disabled', 'selected')

I think 1. and 3. are some sort of very rare timing issues, but I am not sure how to fix. Perhaps such occur more often macOS 13 than macOS 10, the version when we wrote these tests. Beside the main issue, 1. adds custom mocks which are not deleted when the assert fails. I will fix this possible omission, in all IDLE test files, in a separate issue. I think this cleanup failure is the cause of 2. and maybe 4.

Failure 4. must have caused the 3 AttributeErrors by removing but not restoring the mock with the 'called' attribute. In the first,
d is an instance of idlelib.configdialog.HighPage.
set_highlight_target is an instance of idlelib.idle_test.mock_idle.Func mocking the method of the same name.
called is an attribute of the mock set in __init__ and incremented in __called and reset in setUp.

This is the first report of such errors that I have seen in the last few years. macOS 13 seems flakier than some earlier versions. Are recent failures only on macOS? To fix the primary failures, I need more reports to to get a better idea of what fails. For instance, are some asserts failing multiple times, or are they more random. Links to failure runs, like the one above, are enough for me to make summaries like in this comment. (Easier after cascade failures are removed.)

[ned-deily]

FWIW, I have never seen this particular kind of behavior of test_idle on any version of macOS when doing installer sanity tests but those tests always run the test suite in the same order and run GUI-related tests, including test_idle, as separate test runs so there would be no interference from other tests or from random ordering of tests.

[StanFromIreland]

See this run from March (I hope the logs don't disappear), though it seems to have some slightly different fails.