log_gamma diagnostic tests failing frequently

[elseml]

In #575, test_calibration_log_gamma_end_to_end (implemented in #522) failed with the following error:

=========================== short test summary info ===========================
FAILED tests/test_diagnostics/test_diagnostics_metrics.py::test_calibration_log_gamma_end_to_end - assert not np.int64(87) <= np.float64(100.0)

• where np.int64(87) = <function sum at 0x000001FC8D648470>([np.True_, np.True_, np.True_, np.False_, np.True_, np.True_, ...])
• where <function sum at 0x000001FC8D648470> = np.sum
  !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!
  = 1 failed, 128 passed, 85 skipped, 308 deselected, 10105 warnings in 282.46s (0:04:42) =
  Error: Process completed with exit code 1.

Briefly skimming through the latest test workflow history brought up another test failure in #572:

=========================== short test summary info ============================
FAILED tests/test_diagnostics/test_diagnostics_metrics.py::test_calibration_log_gamma_end_to_end - assert np.float64(87.0) <= np.int64(86)

• where np.int64(86) = <function sum at 0x7fa95488bd30>([np.True_, np.True_, np.True_, np.True_, np.True_, np.True_, ...])
• where <function sum at 0x7fa95488bd30> = np.sum
  !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
  = 1 failed, 128 passed, 85 skipped, 308 deselected, 10187 warnings in 155.43s (0:02:35) =
  Error: Process completed with exit code 1.

and one it #564:

=========================== short test summary info ============================
FAILED tests/test_diagnostics/test_diagnostics_metrics.py::test_calibration_log_gamma_end_to_end - assert not 90 <= 100.0

• where 90 = <function sum at 0x7ff3aea32f70>([True, True, False, False, True, True, ...])
• where <function sum at 0x7ff3aea32f70> = np.sum
  !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
  = 1 failed, 128 passed, 85 skipped, 308 deselected, 18 warnings in 212.72s (0:03:32) =
  Error: Process completed with exit code 1.

This might hint at either too conservative/brittle test settings or some unexpected behavior of the metric itself @daniel-habermann @paul-buerkner @stefanradev93

[daniel-habermann]

Here is one additional data point: I've ran the unit test 1000 times and observed 5 failures. That is indeed a bit more than expected, but probably just a result of discretization errors from the fact that we are asking for quite extreme quantiles of the null distribution based on only 200 draws.

My current best guess is therefore something CI related, I will have a look and see if I can run the containers locally.

[daniel-habermann]

#1449 - unrelated
#1447 - unrelated
#1441 - unrelated
#1438 - unrelated
#1436 - ubuntu-latest 3.10, tensorflow (2nd assert fails)
#1434 - unrelated
#1433 - unrelated
#1432 - unrelated
#1431 - windows-latest, 3.10, tensorflow (2nd assert fails)
#1423 - unrelated
#1422 - unrelated
#1421 - unrelated
#1420 - unrelated
#1419 - unrelated
#1416 - unrelated
#1415 - unrelated
#1398 - ubuntu-latest, 3.10, jax (1st assert fails)
#1391 - unrelated
#1385 - unrelated
#1384 - unrelated
#1383 - unrelated
#1373 - ubuntu-latest, 3.10, jax (2nd assert fails)
#1371 - unrelated 
#1345 - unrelated

Another data point: I've went through the CI history to see if there are any patterns. That doesn't seem to be case, but combined with my local runs, it looks like the 2nd assertion is failing more often than the first.

The second assertion is testing SBC sensitivity. Right now, we are asking SBC to correctly fail if we observe $\theta \sim Beta(31, 31)$ when the true posterior is $\theta \sim Beta(30, 30)$, based on 1000 SBC runs. While SBC can identify underdisperion, that is probably a bit too difficult.

I suggest for now we 1) update the test settings a bit and 2) continue to monitor if we observe more test failures in the future.

[stefanradev93]

I would also suggest to make the test slightly easier.

[daniel-habermann]

fixed by #582