Skip to content

(via xdist) Intermittently ending up on non-main thread #96

Closed
@boxed

Description

@boxed

This issue has been plaguing us for months since we started using xdist. We have a big test suite that extremely rarely fails with ValueError: signal only works in main thread. I have managed to get the full traceback of all threads in a normal run and in this case where it crashes.

Normal:

Thread 0x000070000fd66000 (most recent call first):
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 192 in _select
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 178 in select
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 553 in _connect_attempt
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 512 in _connect_loop
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/retry.py", line 124 in __call__
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 473 in zk_loop
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 864 in run
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x000070000f863000 (most recent call first):
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 295 in wait
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/queue.py", line 164 in get
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 120 in _thread_worker
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 864 in run
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x000070000f360000 (most recent call first):
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 295 in wait
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/queue.py", line 164 in get
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 120 in _thread_worker
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 864 in run
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/local/Cellar/python@3.6.5/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x000070000ee5d000 (most recent call first):
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 386 in read
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 418 in from_io
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 954 in _thread_receiver
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 213 in run
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn

Current thread 0x000000011088f5c0 (most recent call first):
  File "/Users/andersh/triresolve/triresolve/matcher/matcher.py", line 1128 in go
  File "/Users/andersh/triresolve/triresolve/matcher/matcher.py", line 3717 in match
  File "/Users/andersh/triresolve/triresolve/matcher/cmdline.py", line 58 in update_match_view
  File "/Users/andersh/triresolve/triresolve/matcher/test/integrationtest/test_3w_linking.py", line 304 in run_end_to_end_3w_linking_one_to_one
  File "/Users/andersh/triresolve/triresolve/matcher/test/integrationtest/test_3w_linking.py", line 271 in test_end_to_end_3w_linking_one_to_one_normal_rec_trades_on_sub_entity
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/python.py", line 1425 in runtest
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 122 in pytest_runtest_call
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 194 in <lambda>
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 212 in __init__
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 196 in call_runtest_hook
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 172 in call_and_report
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 92 in runtestprotocol
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/runner.py", line 77 in pytest_runtest_protocol
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__
  File "<remote exec>", line 83 in run_one_test
  File "<remote exec>", line 69 in pytest_runtestloop
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/main.py", line 225 in _main
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/main.py", line 185 in wrap_session
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/_pytest/main.py", line 218 in pytest_cmdline_main
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__
  File "<remote exec>", line 270 in <module>
  File "<string>", line 1 in do_exec
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1072 in executetask
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 213 in run
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 259 in integrate_as_primary_thread
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1047 in serve
  File "/Users/andersh/triresolve/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1534 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>

Incorrect:

Thread 0x00007fc9108ba700 (most recent call first)://
  File "/tmp/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 192 in _select//
  File "/tmp/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 178 in select//
  File "/tmp/venv/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 553 in _connect_attempt//
  File "/tmp/venv/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 512 in _connect_loop//
  File "/tmp/venv/lib/python3.6/site-packages/kazoo/retry.py", line 124 in __call__//
  File "/tmp/venv/lib/python3.6/site-packages/kazoo/protocol/connection.py", line 473 in zk_loop//
  File "/usr/lib64/python3.6/threading.py", line 864 in run//
  File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
  File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fc9110bb700 (most recent call first)://
  File "/usr/lib64/python3.6/threading.py", line 295 in wait//
  File "/usr/lib64/python3.6/queue.py", line 164 in get//
  File "/tmp/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 120 in _thread_worker//
  File "/usr/lib64/python3.6/threading.py", line 864 in run//
  File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
  File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fc9118bc700 (most recent call first)://
  File "/usr/lib64/python3.6/threading.py", line 295 in wait//
  File "/usr/lib64/python3.6/queue.py", line 164 in get//
  File "/tmp/venv/lib/python3.6/site-packages/kazoo/handlers/threading.py", line 120 in _thread_worker//
  File "/usr/lib64/python3.6/threading.py", line 864 in run//
  File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
  File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fc912ffd700 (most recent call first)://
  File "/usr/lib64/python3.6/threading.py", line 295 in wait//
  File "/usr/lib64/python3.6/queue.py", line 164 in get//
  File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 67 in _worker//
  File "/usr/lib64/python3.6/threading.py", line 864 in run//
  File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
  File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fc9137fe700 (most recent call first)://
  File "/usr/lib64/python3.6/threading.py", line 295 in wait//
  File "/usr/lib64/python3.6/queue.py", line 164 in get//
  File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 67 in _worker//
  File "/usr/lib64/python3.6/threading.py", line 864 in run//
  File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
  File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fc913fff700 (most recent call first)://
  File "/usr/lib64/python3.6/threading.py", line 299 in wait//
  File "/usr/lib64/python3.6/threading.py", line 551 in wait//
  File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
  File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fc918d1e700 (most recent call first)://
  File "/usr/lib64/python3.6/asyncore.py", line 183 in poll2//
  File "/usr/lib64/python3.6/asyncore.py", line 207 in loop//
  File "/tmp/venv/lib/python3.6/site-packages/cassandra/io/asyncorereactor.py", line 119 in loop//
  File "/tmp/venv/lib/python3.6/site-packages/cassandra/io/asyncorereactor.py", line 258 in _run_loop//
  File "/usr/lib64/python3.6/threading.py", line 864 in run//
  File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
  File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fc921a79700 (most recent call first)://
  File "/usr/lib64/python3.6/threading.py", line 295 in wait//
  File "/usr/lib64/python3.6/queue.py", line 164 in get//
  File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner//
  File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007fc94637f700 (most recent call first)://
  File "/opt/trioptima/triresolve/triresolve/matcher/matcher.py", line 1124 in go//
  File "/opt/trioptima/triresolve/triresolve/matcher/matcher.py", line 3713 in match//
  File "/opt/trioptima/triresolve/triresolve/matcher/cmdline.py", line 58 in update_match_view//
  File "/opt/trioptima/triresolve/triresolve/matcher/test/integrationtest/test_3w_linking.py", line 304 in run_end_to_end_3w_linking_one_to_one
  File "/opt/trioptima/triresolve/triresolve/matcher/test/integrationtest/test_3w_linking.py", line 271 in// test_end_to_end_3w_linking_one_to_one_normal_rec_trades_on_sub_entity//
  File "/tmp/venv/lib/python3.6/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__//
  File "/tmp/venv/lib/python3.6/site-packages/_pytest/python.py", line 1425 in runtest//
  File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 122 in pytest_runtest_call//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__//
  File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 194 in <lambda>//
  File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 212 in __init__//
  File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 196 in call_runtest_hook//
  File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 172 in call_and_report//
  File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 92 in runtestprotocol//
  File "/tmp/venv/lib/python3.6/site-packages/_pytest/runner.py", line 77 in pytest_runtest_protocol//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__//
  File "<remote exec>", line 83 in run_one_test//
  File "<remote exec>", line 66 in pytest_runtestloop//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__//
  File "/tmp/venv/lib/python3.6/site-packages/_pytest/main.py", line 225 in _main//
  File "/tmp/venv/lib/python3.6/site-packages/_pytest/main.py", line 185 in wrap_session//
  File "/tmp/venv/lib/python3.6/site-packages/_pytest/main.py", line 218 in pytest_cmdline_main//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 61 in <lambda>//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/manager.py", line 67 in _hookexec//
  File "/tmp/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 284 in __call__//
  File "<remote exec>", line 270 in <module>//
  File "<string>", line 1 in do_exec//
  File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1072 in executetask//
  File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 213 in run//
  File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn

Thread 0x00007fc946bc0700 (most recent call first)://
  File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 386 in read//
  File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 418 in from_io//
  File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 954 in _thread_receiver//
  File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 213 in run//
  File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn

Thread 0x00007fc9484c0740 (most recent call first)://
  File "/usr/lib64/python3.6/threading.py", line 295 in wait//
  File "/usr/lib64/python3.6/threading.py", line 551 in wait//
  File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 255 in integrate_as_primary_thread//
  File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1047 in serve//
  File "/tmp/venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1534 in serve//
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>mysql strict mode! at least in tests://

Digging through the code it looks to me like _try_send_to_primary_thread fails for some reason, even though we are running in process mode. This is obviously a guess though, as I don't understand this code very well.

Help! :)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions