CI related fixes

[YigitElma]

• Skip memory benchmarks for only-docs-comments labeled PRs
• Fail memory benchmark if the underlying test fails
• Skip codecov if no report is uploaded for any of the workflows
• Add comment on the possible noise on benchmarks

Resolves #1913
Resolves #1929
Resolves #1852

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.76%. Comparing base (587fd85) to head (d2af11b).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1946      +/-   ##
==========================================
- Coverage   95.77%   95.76%   -0.01%     
==========================================
  Files         100      100              
  Lines       27541    27541              
==========================================
- Hits        26378    26376       -2     
- Misses       1163     1165       +2     

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Memory benchmark result

|               Test Name                |      %Δ      |    Master (MB)     |      PR (MB)       |    Δ (MB)    |    Time PR (s)     |  Time Master (s)   |
| -------------------------------------- | ------------ | ------------------ | ------------------ | ------------ | ------------------ | ------------------ |
  test_objective_jac_w7x                 |    4.93 %    |     3.839e+03      |     4.028e+03      |    189.28    |       33.45        |       31.24        |
  test_proximal_jac_w7x_with_eq_update   |    1.83 %    |     6.764e+03      |     6.888e+03      |    123.46    |       160.21       |       159.69       |
  test_proximal_freeb_jac                |    0.29 %    |     1.317e+04      |     1.321e+04      |    37.64     |       77.60        |       77.72        |
  test_proximal_freeb_jac_blocked        |    1.12 %    |     7.543e+03      |     7.628e+03      |    84.68     |       68.87        |       68.49        |
  test_proximal_freeb_jac_batched        |    0.64 %    |     7.576e+03      |     7.624e+03      |    48.11     |       68.69        |       69.28        |
  test_proximal_jac_ripple               |   -0.56 %    |     7.620e+03      |     7.578e+03      |    -42.55    |       69.40        |       69.50        |
  test_proximal_jac_ripple_spline        |    1.70 %    |     3.447e+03      |     3.505e+03      |    58.50     |       72.22        |       72.22        |
  test_eq_solve                          |    1.91 %    |     2.034e+03      |     2.073e+03      |    38.86     |       125.55       |       124.54       |

For the memory plots, go to the summary of Memory Benchmarks workflow and download the artifact.

[f0uriest]

just so i understand this:

• if no reports are uploaded (ie, no tests were run bc it just doc updates) then this will succeed
• If eg, 13/14 reports are uploaded (ie bc one of the unit tests fails) then this will still fail? (I think we want it to fail in this case)

[YigitElma]

Yes, this is the way I understand it from their documentation.

Our unit and regression tests upload a report even if they fail (upload step runs even for pytest failure, see always() at the condition). However, if 14 tests ran and still the status is pending/expecting, then this change should make the codecov pass.

We can use Perlmutter PR as a test for this. I am also not 100% sure if it is going to work.