Segfault in multiple tests on OS X, Python 3.9

[hyanwong]

Tests all pass with OS X on python 3.8, but on 3.9 there are multiple failures. The first I get, from running python3 -m pytest -vs -n0 is in tests/test_cli.py::TestCommandsDefaults::test_augment_ancestors:

python3 -m pytest -vs -n0
================================================================ test session starts =================================================================
platform darwin -- Python 3.9.0, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /usr/local/anaconda3/envs/python39/bin/python3
cachedir: .pytest_cache
rootdir: /Users/yan/Documents/Research/BDI/tsinfer, configfile: pytest.ini, testpaths: tests
plugins: xdist-2.1.0, cov-2.10.1, forked-1.3.0
collected 598 items                                                                                                                                  

tests/test_cli.py::TestMain::test_cli_main PASSED
tests/test_cli.py::TestMain::test_main PASSED
tests/test_cli.py::TestDefaultPaths::test_get_default_path PASSED
tests/test_cli.py::TestDefaultPaths::test_get_ancestors_path PASSED
tests/test_cli.py::TestDefaultPaths::test_get_ancestors_trees_path PASSED
tests/test_cli.py::TestDefaultPaths::test_get_output_trees_path PASSED
tests/test_cli.py::TestCommandsDefaults::test_augment_ancestors Fatal Python error: Segmentation fault

Current thread 0x0000000118bb1dc0 (most recent call first):
  File "/Users/yan/Documents/Research/BDI/tsinfer/tsinfer/inference.py", line 1095 in __complete_epoch
  File "/Users/yan/Documents/Research/BDI/tsinfer/tsinfer/inference.py", line 1122 in __match_ancestors_single_threaded
  File "/Users/yan/Documents/Research/BDI/tsinfer/tsinfer/inference.py", line 1169 in match_ancestors
  File "/Users/yan/Documents/Research/BDI/tsinfer/tsinfer/inference.py", line 370 in match_ancestors
  File "/Users/yan/Documents/Research/BDI/tsinfer/tests/test_cli.py", line 132 in setUp
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/unittest/case.py", line 547 in _callSetUp
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/unittest/case.py", line 589 in run
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/unittest/case.py", line 653 in __call__
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/unittest.py", line 278 in runtest
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/runner.py", line 163 in pytest_runtest_call
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/runner.py", line 256 in <lambda>
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/runner.py", line 310 in from_call
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/runner.py", line 216 in call_and_report
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/runner.py", line 127 in runtestprotocol
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/runner.py", line 110 in pytest_runtest_protocol
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/main.py", line 338 in pytest_runtestloop
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/main.py", line 313 in _main
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/main.py", line 257 in wrap_session
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/main.py", line 306 in pytest_cmdline_main
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/config/__init__.py", line 164 in main
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/_pytest/config/__init__.py", line 187 in console_main
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/site-packages/pytest/__main__.py", line 5 in <module>
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/runpy.py", line 87 in _run_code
  File "/usr/local/anaconda3/envs/python39/lib/python3.9/runpy.py", line 197 in _run_module_as_main
zsh: segmentation fault  python3 -m pytest -vs -n0

Running in parallel identifies at least these others as problems

maximum crashed workers reached: 16

====================================================================== FAILURES ======================================================================
_________________________________________________________________ tests/test_cli.py __________________________________________________________________
[gw1] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw1' crashed while running 'tests/test_cli.py::TestCommandsDefaults::test_infer_from_ts'
_________________________________________________________________ tests/test_cli.py __________________________________________________________________
[gw2] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw2' crashed while running 'tests/test_cli.py::TestCommandsDefaults::test_augment_ancestors'
_________________________________________________________________ tests/test_cli.py __________________________________________________________________
[gw3] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw3' crashed while running 'tests/test_cli.py::TestCommandsDefaults::test_infer'
_________________________________________________________________ tests/test_cli.py __________________________________________________________________
[gw0] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw0' crashed while running 'tests/test_cli.py::TestCommandsDefaults::test_infer_bad_file'
______________________________________________________________ tests/test_inference.py _______________________________________________________________
[gw6] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw6' crashed while running 'tests/test_inference.py::TestAugmentedAncestorsRoundTrip::test_simple_example'
______________________________________________________________ tests/test_inference.py _______________________________________________________________
[gw7] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw7' crashed while running 'tests/test_inference.py::TestMissingDataRoundTrip::test_random_data_invariant_sites_ancestral_state'
______________________________________________________________ tests/test_inference.py _______________________________________________________________
[gw5] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw5' crashed while running 'tests/test_inference.py::TestUnfinalisedErrors::test_match_samples_unfinalised'
______________________________________________________________ tests/test_inference.py _______________________________________________________________
[gw8] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw8' crashed while running 'tests/test_inference.py::TestAlgorithmsExactlyEqualPrecision24::test_four_samples'
______________________________________________________________ tests/test_inference.py _______________________________________________________________
[gw9] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw9' crashed while running 'tests/test_inference.py::TestExtractAncestors::test_simple_simulation'
______________________________________________________________ tests/test_evaluation.py ______________________________________________________________
[gw10] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw10' crashed while running 'tests/test_evaluation.py::TestPerfectInference::test_single_tree_defaults'
______________________________________________________________ tests/test_inference.py _______________________________________________________________
[gw4] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw4' crashed while running 'tests/test_inference.py::TestAugmentedAncestorsRoundTrip::test_random_data_large_example'
______________________________________________________________ tests/test_evaluation.py ______________________________________________________________
[gw11] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw11' crashed while running 'tests/test_evaluation.py::TestPerfectInference::test_small_smc_path_compression'
______________________________________________________________ tests/test_inference.py _______________________________________________________________
[gw13] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw13' crashed while running 'tests/test_inference.py::TestSparseAncestorsRoundTrip::test_single_locus_two_samples'
______________________________________________________________ tests/test_inference.py _______________________________________________________________
[gw14] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw14' crashed while running 'tests/test_inference.py::TestGeneratedAncestors::test_no_recombination'
______________________________________________________________ tests/test_inference.py _______________________________________________________________
[gw12] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw12' crashed while running 'tests/test_inference.py::TestAugmentedAncestorsRoundTrip::test_single_locus'
______________________________________________________________ tests/test_inference.py _______________________________________________________________
[gw15] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw15' crashed while running 'tests/test_inference.py::TestMatchSamples::test_partial_bad_indexes'
______________________________________________________________ tests/test_inference.py _______________________________________________________________
[gw16] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw16' crashed while running 'tests/test_inference.py::TestPathCompressionSamplesCEngine::test_simulation_with_error'
______________________________________________________________ tests/test_evaluation.py ______________________________________________________________
[gw18] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw18' crashed while running 'tests/test_evaluation.py::TestMeanSampleAncestry::test_random_data_inferred_no_simplify'
______________________________________________________________ tests/test_evaluation.py ______________________________________________________________
[gw17] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw17' crashed while running 'tests/test_evaluation.py::TestCountSampleChildEdges::test_inferred_simplify'
______________________________________________________________ tests/test_evaluation.py ______________________________________________________________
[gw19] darwin -- Python 3.9.0 /usr/local/anaconda3/envs/python39/bin/python3
worker 'gw19' crashed while running 'tests/test_evaluation.py::TestNodeSpan::test_inferred'

[jeromekelleher]

Great, thanks @hyanwong. Can you run this through gdb please?

gdb python
run -m pytest -n0 -vs tests/test_cli.py::TestCommandsDefaults::test_augment_ancestors
bt

This should run until it segfaults, and then we want to print out the backtrace. (bt)

Or whatever equivalent debugger is present in the clang toolchain.

[hyanwong]

OS X needs to codesign gdb, or run under sudo. Once I do that, it's not super helpful with the default install of Python 3.9 (below). I guess I need to recompile Python 3.9 for this to work?

Reading symbols from /usr/local/bin/python3...
(No debugging symbols found in /usr/local/bin/python3)
(gdb) run -m pytest -n0 -vs tests/test_cli.py::TestCommandsDefaults::test_augment_ancestors
Starting program: /usr/local/bin/python3 -m pytest -n0 -vs tests/test_cli.py::TestCommandsDefaults::test_augment_ancestors
[New Thread 0x2603 of process 34225]
[New Thread 0x1903 of process 34225]
warning: unhandled dyld version (16)
[New Thread 0x2303 of process 34225]

Thread 3 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 0x2303 of process 34225]
0x000000010000e000 in ?? ()
(gdb) bt
#0  0x000000010000e000 in ?? ()
#1  0x0000000100000000 in ?? ()
#2  0x0000000000000006 in ?? ()
#3  0x00007ffeefbffbc0 in ?? ()
#4  0x00007ffeefbffc39 in ?? ()
#5  0x00007ffeefbffc3c in ?? ()
#6  0x00007ffeefbffc43 in ?? ()
#7  0x00007ffeefbffc47 in ?? ()
#8  0x00007ffeefbffc4b in ?? ()
#9  0x0000000000000000 in ?? ()

[benjeffery]

Can you run

make clean
CFLAGS="-ggdb -O0" python3 setup.py build_ext --inplace

before the pytest?

Although if your stack is only 9 deep then we are probably still in Python-land, which is weird.

[hyanwong]

Oddly, that hangs when running.

I'll try to recompile Python 3.9 with the debug flags.

[hyanwong]

Hmm https://unconj.ca/blog/setting-up-gdb-for-debugging-python-on-os-x.html:

"Note that for some reason, Python always sends a SIGTRAP before it begins executing the script. Don’t worry, just continue. If you want to find out why your program segfaults, you can run bt:"

[benjeffery]

Ah didn't know that! Just do continue then!

[hyanwong]

Can't get the damn thing to do anything other than hang now. Argh.

[benjeffery]

make clean
make

?

[hyanwong]

Yep. Tried all that

[jeromekelleher]

Where did you get your Python from @hyanwong? Is there a debug version? We'll need the debug symbols everywhere to get to the bottom of this.

[hyanwong]

Slightly weirdly, after make; make clean, I can't now run the tests normally:

/usr/local/bin/python3.9 -m pytest tests/test_cli.py::TestCommandsDefaults::test_augment_ancestors
================================================================ test session starts =================================================================
platform darwin -- Python 3.9.0, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /Users/yan/Documents/Research/BDI/tsinfer, configfile: pytest.ini
plugins: xdist-2.1.0, cov-2.10.1, forked-1.3.0
gw0 [0] / gw1 [0] / gw2 [0] / gw3 [0]                                                                                                                

======================================================================= ERRORS =======================================================================
_________________________________________________________ ERROR collecting tests/test_cli.py _________________________________________________________
ImportError while importing test module '/Users/yan/Documents/Research/BDI/tsinfer/tests/test_cli.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/Cellar/python@3.9/3.9.0_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_cli.py:35: in <module>
    import tsinfer
tsinfer/__init__.py:38: in <module>
    from .inference import *  # NOQA
tsinfer/inference.py:36: in <module>
    import _tsinfer
E   ModuleNotFoundError: No module named '_tsinfer'
============================================================== short test summary info ===============================================================
ERROR tests/test_cli.py

[hyanwong]

Where did you get your Python from @hyanwong? Is there a debug version? We'll need the debug symbols everywhere to get to the bottom of this.

I'm trying it on 2 different python 3.9 versions, one from conda, another from the OS X installation repo "Homebrew". Neither seem to have debug symbols compiled in, so I guess I'll need to compile my own.

[jeromekelleher]

Yes, looks like compiling your own might be the simplest for getting debug symbols, as there's mac-specific technical issues with the conda distributed binaries.