Issues with reproducibility tests

[szy21]

When the ref counter is increased in a PR that has not been rebased to the main branch, and when the PR is not in conflict with the main branch (e.g. some other PR increased the ref counter), the regression test would not assume that it is a self-reference because of this line. @charleskawczynski may be able to explain the issue better.

[charleskawczynski]

I'm going to reopen this since we've been discussing reproducibility a lot lately.

[charleskawczynski]

One issue I recently thought about is the following:

Consider the critical time window of time between: 1) when a PR "A" is in the merge queue and data is moved from scratch to the cluster_data_prefix = "/central/scratch/esm/slurm-buildkite/climaatmos-main" folder (where we store solutions to compare against) and 2) when github receives the success status from buildkite and the PR is merged into the main branch.

In this critical time window, if another build (e.g., PR "B") is running, and compares against the latest data that exists on main, that build may be running with a version that is not up-to-date with the main branch (as the main branch is in the process of being updated by PR "A"). Therefore, the comparison may be against new data, resulting in a failed comparison (e.g., non-zero MSE tables on PR "B", that could be only doc changes).

One patch fix here would be to rebase PR "B" once PR "A" lands on github. Perhaps we could prevent this by filtering out folders that have been merged in the last 5 minutes?

I need to think this through a bit more, and perhaps discuss with folks.