exception/warnings about self._alive (in weakref) when running pytest

[kratsg]

Description

See https://bugs.python.org/issue41757. Likely related PR fixing it: python/cpython#18189 .

Some sort of race condition is happening where weakref callbacks are called after objects are garbage-collected.

Expected Behavior

No exceptions/warnings.

Actual Behavior

Actually saw exceptions/warnings (sometimes).

AttributeError: 'NoneType' object has no attribute '_alive'
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x1696c2a70>
Traceback (most recent call last):
  File "/Users/kratsg/.virtualenvs/pyhf-dev/lib/python3.7/weakref.py", line 55, in _cb
    if self._alive:

Steps to Reproduce

Clone repository (I ran this on v0.5.2) and set up everything for development. Then run

pytest tests/test_validation.py -vvvv -k "test_optimizer_stitching[scipy-numpy or test_optimizer_stitching[minuit-numpy" -s

backtrace is

  /Users/kratsg/pyhf/tests/test_validation.py(1082)test_optimizer_stitching()
-> pdf = pyhf.simplemodels.hepdata_like([50.0], [100.0], [10])
  /Users/kratsg/pyhf/src/pyhf/simplemodels.py(64)hepdata_like()
-> return Model(spec, batch_size=batch_size)
  /Users/kratsg/pyhf/src/pyhf/pdf.py(590)__init__()
-> config=self.config, batch_size=self.batch_size
  /Users/kratsg/pyhf/src/pyhf/pdf.py(339)__init__()
-> self.config.auxdata_order,
  /Users/kratsg/pyhf/src/pyhf/parameters/paramview.py(66)__init__()
-> self._precompute()
  /Users/kratsg/pyhf/src/pyhf/parameters/paramview.py(78)_precompute()
-> self.allpar_viewer, self.selected_viewer, self.all_indices
  /Users/kratsg/pyhf/src/pyhf/parameters/paramview.py(27)extract_index_access()
-> index_selection = baseviewer.split(indices, selection=subviewer.names)
  /Users/kratsg/pyhf/src/pyhf/tensor/common.py(59)split()
-> data = tensorlib.einsum('...j->j...', tensorlib.astensor(data))
  /Users/kratsg/pyhf/src/pyhf/tensor/numpy_backend.py(268)einsum()
-> return np.einsum(subscripts, *operands)
  <__array_function__ internals>(6)einsum()
> /Users/kratsg/.virtualenvs/pyhf-dev/lib/python3.7/weakref.py(56)_cb()
-> if self._alive:

Checklist

• Run git fetch to get the most up to date version of master
• Searched through existing Issues to confirm this is not a duplicate issue
• Filled out the Description, Expected Behavior, Actual Behavior, and Steps to Reproduce sections above or have edited/removed them in a way that fully describes the issue

[kratsg]

Also related: matplotlib/matplotlib#14843

[kratsg]

This is indeed confirmed to be fixed in python 3.8: https://github.com/python/cpython/blob/master/Modules/gcmodule.c#L780.

[kratsg]

and fixed by python/cpython#16495 .

[matthewfeickert]

and fixed by python/cpython#16495 .

This Issue has been resolved, but we'll keep this Issue open until pyhf drops support for Python 3.7. According to the Python 3.7 release schedule that won't be until 2026, though NEP 29 will drop support for Python 3.7 in December 2021 — it is hard to see the HEP world already being fully on Python 3.8+ by 2022 though.