AddressSanitizer: Skip tests directly in Python, not with external config

[vstinner]

BPO	46633
Nosy	@vstinner, @pablogsal
PRs	bpo-46633: Skip tests on ASAN and/or MSAN builds #31632 [3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) #31634 [3.9] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634) #31644

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-02.17:13:40.631>
created_at = <Date 2022-02-04.10:09:11.092>
labels = ['tests', '3.11']
title = 'AddressSanitizer: Skip tests directly in Python, not with external config'
updated_at = <Date 2022-03-02.17:13:40.630>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-03-02.17:13:40.630>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-02.17:13:40.631>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2022-02-04.10:09:11.092>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 46633
keywords = ['patch']
message_count = 11.0
messages = ['412499', '412501', '412506', '414247', '414248', '414250', '414262', '414266', '414370', '414379', '414380']
nosy_count = 2.0
nosy_names = ['vstinner', 'pablogsal']
pr_nums = ['31632', '31634', '31644']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue46633'
versions = ['Python 3.11']

Linked PRs

• gh-90791: Enable test___all__ on ASAN build #108286
• gh-90791: test.pythoninfo logs ASAN_OPTIONS env var #108289

[vstinner]

It seems like bpo-45200: "Address Sanitizer: libasan dead lock in pthread_create() (test_multiprocessing_fork.test_get() hangs)" is not fixed yet.

In the GitHub Action job, test_multiprocessing_fork is skipped because it's too slow, so the job doesn't hang.

Yesterday, I modified the ASAN buildbot to double its timeout from 20 to 40 minutes:
python/buildmaster-config@5a37411

But it's useful, when it hangs, it hangs forever. Exmaple on the AMD64 Arch Linux Asan Debug 3.9 buildbot (with the new config):

---
(test.test_multiprocessing_fork.WithProcessesTestPicklingConnections) ... ok
Timeout (0:35:00)!
---
https://buildbot.python.org/all/#/builders/588/builds/332

Tests are tuned for ASAN, but the configuration is copied and inconsistent between the GitHub Actions job and the buildbot configuration.

I propose to move this configuration directly into Python.

test_decimal.py checks for "-fsanitize=address" in CFLAGS and skip some tests if it's present.

[vstinner]

Notes:

• test.support has check_sanitizer() function. Tests using it:

  • test_crypt
  • test_idle
  • test_tix
  • test_tk
  • test_ttk_guionly

• test_decimal also checks for memory sanitizer and skip some tests if it's present:

MEMORY_SANITIZER = (
    '-fsanitize=memory' in _cflags or
    '--with-memory-sanitizer' in _config_args
)   

• test_faulthandler suppress crash report, similir to support.SuppressCrashReport, directly in the C code used by tests: faulthandler_suppress_crash_report() function.

• Objects/obmalloc.c checks "#if __has_feature(address_sanitizer)": is ASAN enabled?

• GitHub Action config: .github/workflows/build.yml. Skipped tests:

  • test___all__
  • test_concurrent_futures
  • test_multiprocessing_fork
  • test_multiprocessing_forkserver
  • test_multiprocessing_spawn
  • test_peg_generator
  • test_tools

Comment:
---
# Skip test_tools test_peg_generator test_concurrent_futures because
# there are too slow: between 5 and 20 minutes on this CI.

# Skip multiprocessing and concurrent.futures tests which are affected by
# bpo-45200 bug: libasan dead lock in pthread_create().

# test___all__ is skipped because importing some modules directly can trigger
# known problems with ASAN (like tk or crypt).
---

• Buildbot configuration: UnixAsanBuild class of master/custom/factories.py. Options:

  ASAN_OPTIONS='detect_leaks=0:allocator_may_return_null=1:handle_segv=0'

With the comment:
---
# See https://bugs.python.org/issue42985 for more context on why
# SIGSEGV is ignored on purpose.
---

Skipped tests:

• test_ctypes
• test_capi
• test_crypt
• test_decimal
• test_faulthandler
• test_interpreters

With the comment:
---
# These tests are currently raising false positives or are interfering with the ASAN mechanism,
# so we need to skip them unfortunately.
---

[pablogsal]

I have recently added some decorators in test.
support to deactivate tests if running under the sanitizers.

[vstinner]

I created a PR to no longer skip tests in buildbots:
python/buildmaster-config#314

[vstinner]

Status with the 2 pending PRs:

• Asyncio: SSL transport does not support set_protocol() #75813
• ASAN and UB workers no longer exclude tests buildmaster-config#314

Tests only skipped on ASAN:

• _test_multiprocessing

Skip on ASAN and MSAN:

• test___all__
• test_concurrent_futures
• test_crypt
• test_decimal.test_maxcontext_exact_arith()
• test_idle
• test_peg_generator
• test_tix
• test_tk
• test_tools
• test_ttk_guionly

No longer skipped:

• test_capi
• test_ctypes
• test_faulthandler
• test_interpreters

[vstinner]

I created bpo-46887: ./Programs/_freeze_module fails with MSAN: Uninitialized value was created by an allocation of 'stat.i'.

[vstinner]

"""
No longer skipped:

• test_capi
• test_ctypes
• test_faulthandler
• test_interpreters
  """

I built Python manually with:

./configure --with-pydebug CC=clang LD=clang --with-address-sanitizer

These tests no longer with ASAN_OPTIONS used on our CI:

ASAN_OPTIONS='detect_leaks=0:allocator_may_return_null=1:handle_segv=0' ./python -m test (...)

[vstinner]

New changeset 9204bb7 by Victor Stinner in branch 'main':
bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632)
9204bb7

[vstinner]

New changeset 9326445 by Victor Stinner in branch '3.10':
[3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634)
9326445

[vstinner]

New changeset 359bc39 by Victor Stinner in branch '3.9':
[3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634) (GH-31644)
359bc39

[vstinner]

python/buildmaster-config#314

I merged this PR as well. The new buildbot configuration will be deployed soon.