[NetBSD] test_socket creates too many locks

[serhiy-storchaka]

BPO	31925
Nosy	@rbtcollins, @ezio-melotti, @voidspace, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-11-02.14:16:30.938>
labels = ['3.7', 'type-bug', 'tests']
title = '[NetBSD] test_socket creates too many locks'
updated_at = <Date 2017-11-06.17:05:57.177>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2017-11-06.17:05:57.177>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2017-11-02.14:16:30.938>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 31925
keywords = []
message_count = 2.0
messages = ['305425', '305659']
nosy_count = 4.0
nosy_names = ['rbcollins', 'ezio.melotti', 'michael.foord', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31925'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7']

Linked PRs

• gh-76106: Remove the cleanup lock in test_socket  #110539
• [3.12] gh-76106: Remove the cleanup lock in test_socket (GH-110539) #110699
• [3.11] gh-76106: Remove the cleanup lock in test_socket (GH-110539) #110700

[serhiy-storchaka]

test_socket creates a lock in the constructor of ThreadSafeCleanupTestCase. But since unittest creates an instance of test class for every test method, it creates around 400 locks. This exceeds the limit of 200 locks in NetBSD 8 and causes a failure.

0:00:00 load avg: 0.00 [1/1] test_socket
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/runtest.py", line 176, in runtest_inner
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/test_socket.py", line 5801, in test_main
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/support/__init__.py", line 1934, in run_unittest
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/unittest/loader.py", line 503, in makeSuite
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/unittest/loader.py", line 92, in loadTestsFromTestCase
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/unittest/suite.py", line 24, in __init__
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/unittest/suite.py", line 57, in addTests
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/test_socket.py", line 544, in __init__
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/test_socket.py", line 144, in __init__
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/threading.py", line 86, in RLock
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
RuntimeError: can't allocate lock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.7/Lib/runpy.py", line 193, in _run_module_as_main
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/runpy.py", line 85, in _run_code
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/__main__.py", line 2, in <module>
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/main.py", line 585, in main
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/main.py", line 510, in main
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/main.py", line 536, in _main
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/main.py", line 458, in run_tests
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/main.py", line 379, in run_tests_sequential
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/runtest.py", line 140, in runtest
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/runtest.py", line 200, in runtest_inner
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/traceback.py", line 163, in format_exc
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/traceback.py", line 117, in format_exception
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/traceback.py", line 496, in __init__
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/traceback.py", line 357, in extract
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/traceback.py", line 281, in line
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/linecache.py", line 16, in getline
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/linecache.py", line 47, in getlines
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/linecache.py", line 136, in updatecache
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/tokenize.py", line 447, in open
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
RuntimeError: can't allocate read lock

[serhiy-storchaka]

test_socket creates many locks on all platforms.

[serhiy-storchaka]

It is a weird test. Initially the lock was introduced in bpo-31370 (#75551) bpo-6560 (#50809) to wrap addCleanup() (which appends to the list of callbacks) and doCleanups() (which pops from that list until it is empty). But in #110167 the lock was removed from doCleanups(), it now only wraps addCleanup(), that does not make sense, because list.append() is atomic.

Should the lock be removed at all? Or it is still needed in doCleanups() and should be restored?

cc @pitrou, @vstinner

[serhiy-storchaka]

Sorry, actually it was introduced in bpo-6560 (#50809). cc @ncoghlan.

[serhiy-storchaka]

ThreadableTest now uses threading_helper.wait_threads_exit(). It means that if the client thread is still running at test cleanup stage, doCleanups() waits until it exit, and then calls all callbacks added in the client thread while it waited for thread exit. No race condition, no leaked resources, no need to guard addCleanup() and doCleanups() with a lock.

[vstinner]

Yeah, it's really bad when a test leaves a thread running a thread in background. It's good that test_socket got fixed. I suppose that this issue can now be closed?