Closed
Description
This issue has been plaguing us for months since we started using xdist. We have a big test suite that extremely rarely fails with ValueError: signal only works in main thread
. I have managed to get the full traceback of all threads in a normal run and in this case where it crashes.
Normal:
Thread 0x000070000fd66000 (most recent call first):
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 192 in _select
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 178 in select
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 553 in _connect_attempt
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 512 in _connect_loop
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/retry.py", line 124 in __call__
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 473 in zk_loop
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 864 in run
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x000070000f863000 (most recent call first):
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 295 in wait
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/queue.py", line 164 in get
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 120 in _thread_worker
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 864 in run
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x000070000f360000 (most recent call first):
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 295 in wait
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/queue.py", line 164 in get
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 120 in _thread_worker
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 864 in run
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x000070000ee5d000 (most recent call first):
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 386 in read
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 418 in from_io
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 954 in _thread_receiver
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 213 in run
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn
Current thread 0x000000011088f5c0 (most recent call first):
File "/Users/andersh/triresolve/triresolve/matcher/matcher.py", line 1128 in go
File "/Users/andersh/triresolve/triresolve/matcher/matcher.py", line 3717 in match
File "/Users/andersh/triresolve/triresolve/matcher/cmdline.py", line 58 in update_match_view
File "/Users/andersh/triresolve/triresolve/matcher/test/integrationtest/test_3w_linking.py", line 304 in run_end_to_end_3w_linking_one_to_one
File "/Users/andersh/triresolve/triresolve/matcher/test/integrationtest/test_3w_linking.py", line 271 in test_end_to_end_3w_linking_one_to_one_normal_rec_trades_on_sub_entity
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/python.py", line 1425 in runtest
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 122 in pytest_runtest_call
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 194 in <lambda>
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 212 in __init__
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 196 in call_runtest_hook
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 172 in call_and_report
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 92 in runtestprotocol
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 77 in pytest_runtest_protocol
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__
File "<remote exec>", line 83 in run_one_test
File "<remote exec>", line 69 in pytest_runtestloop
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/main.py", line 225 in _main
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/main.py", line 185 in wrap_session
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/main.py", line 218 in pytest_cmdline_main
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__
File "<remote exec>", line 270 in <module>
File "<string>", line 1 in do_exec
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1072 in executetask
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 213 in run
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 259 in integrate_as_primary_thread
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1047 in serve
File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1534 in serve
File "<string>", line 8 in <module>
File "<string>", line 1 in <module>
Incorrect:
Thread 0x00007fc9108ba700 (most recent call first)://
File "/tmp/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 192 in _select//
File "/tmp/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 178 in select//
File "/tmp/venv/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 553 in _connect_attempt//
File "/tmp/venv/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 512 in _connect_loop//
File "/tmp/venv/lib/python3.6/site-packages/kazoo/retry.py", line 124 in __call__//
File "/tmp/venv/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 473 in zk_loop//
File "/usr/lib64/python3.6/threading.py", line 864 in run//
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fc9110bb700 (most recent call first)://
File "/usr/lib64/python3.6/threading.py", line 295 in wait//
File "/usr/lib64/python3.6/queue.py", line 164 in get//
File "/tmp/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 120 in _thread_worker//
File "/usr/lib64/python3.6/threading.py", line 864 in run//
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fc9118bc700 (most recent call first)://
File "/usr/lib64/python3.6/threading.py", line 295 in wait//
File "/usr/lib64/python3.6/queue.py", line 164 in get//
File "/tmp/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 120 in _thread_worker//
File "/usr/lib64/python3.6/threading.py", line 864 in run//
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fc912ffd700 (most recent call first)://
File "/usr/lib64/python3.6/threading.py", line 295 in wait//
File "/usr/lib64/python3.6/queue.py", line 164 in get//
File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 67 in _worker//
File "/usr/lib64/python3.6/threading.py", line 864 in run//
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fc9137fe700 (most recent call first)://
File "/usr/lib64/python3.6/threading.py", line 295 in wait//
File "/usr/lib64/python3.6/queue.py", line 164 in get//
File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 67 in _worker//
File "/usr/lib64/python3.6/threading.py", line 864 in run//
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fc913fff700 (most recent call first)://
File "/usr/lib64/python3.6/threading.py", line 299 in wait//
File "/usr/lib64/python3.6/threading.py", line 551 in wait//
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fc918d1e700 (most recent call first)://
File "/usr/lib64/python3.6/asyncore.py", line 183 in poll2//
File "/usr/lib64/python3.6/asyncore.py", line 207 in loop//
File "/tmp/venv/lib/python3.6/site-packages/cassandra/io/asyncorereactor.py", line 119 in loop//
File "/tmp/venv/lib/python3.6/site-packages/cassandra/io/asyncorereactor.py", line 258 in _run_loop//
File "/usr/lib64/python3.6/threading.py", line 864 in run//
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007fc921a79700 (most recent call first)://
File "/usr/lib64/python3.6/threading.py", line 295 in wait//
File "/usr/lib64/python3.6/queue.py", line 164 in get//
File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap
Current thread 0x00007fc94637f700 (most recent call first)://
File "/opt/trioptima/triresolve/triresolve/matcher/matcher.py", line 1124 in go//
File "/opt/trioptima/triresolve/triresolve/matcher/matcher.py", line 3713 in match//
File "/opt/trioptima/triresolve/triresolve/matcher/cmdline.py", line 58 in update_match_view//
File "/opt/trioptima/triresolve/triresolve/matcher/test/integrationtest/test_3w_linking.py", line 304 in run_end_to_end_3w_linking_one_to_one
File "/opt/trioptima/triresolve/triresolve/matcher/test/integrationtest/test_3w_linking.py", line 271 in// test_end_to_end_3w_linking_one_to_one_normal_rec_trades_on_sub_entity//
File "/tmp/venv/lib/python3.6/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__//
File "/tmp/venv/lib/python3.6/site-packages/_pytest/python.py", line 1425 in runtest//
File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 122 in pytest_runtest_call//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__//
File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 194 in <lambda>//
File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 212 in __init__//
File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 196 in call_runtest_hook//
File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 172 in call_and_report//
File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 92 in runtestprotocol//
File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 77 in pytest_runtest_protocol//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__//
File "<remote exec>", line 83 in run_one_test//
File "<remote exec>", line 66 in pytest_runtestloop//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__//
File "/tmp/venv/lib/python3.6/site-packages/_pytest/main.py", line 225 in _main//
File "/tmp/venv/lib/python3.6/site-packages/_pytest/main.py", line 185 in wrap_session//
File "/tmp/venv/lib/python3.6/site-packages/_pytest/main.py", line 218 in pytest_cmdline_main//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec//
File "/tmp/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__//
File "<remote exec>", line 270 in <module>//
File "<string>", line 1 in do_exec//
File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1072 in executetask//
File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 213 in run//
File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn
Thread 0x00007fc946bc0700 (most recent call first)://
File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 386 in read//
File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 418 in from_io//
File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 954 in _thread_receiver//
File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 213 in run//
File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn
Thread 0x00007fc9484c0740 (most recent call first)://
File "/usr/lib64/python3.6/threading.py", line 295 in wait//
File "/usr/lib64/python3.6/threading.py", line 551 in wait//
File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 255 in integrate_as_primary_thread//
File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1047 in serve//
File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1534 in serve//
File "<string>", line 8 in <module>
File "<string>", line 1 in <module>mysql strict mode! at least in tests://
Digging through the code it looks to me like _try_send_to_primary_thread
fails for some reason, even though we are running in process mode. This is obviously a guess though, as I don't understand this code very well.
Help! :)
Metadata
Metadata
Assignees
Labels
No labels