Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move coarse-graining tests out of docker build #340

Closed
spencerkclark opened this issue Nov 11, 2022 · 0 comments · Fixed by #379
Closed

Move coarse-graining tests out of docker build #340

spencerkclark opened this issue Nov 11, 2022 · 0 comments · Fixed by #379

Comments

@spencerkclark
Copy link
Member

In #339 we have started to be more judicious about running tests in CI. Not all fortran PRs touch the coarse-graining code, so there is often not a need to check that those tests pass upon each push to every PR. One might ask why we can't just remove the coarse-graining tests from docker image since we already have some native regression tests for those features. The answer there is that the docker tests do two things that the nix tests currently do not:

  1. They exercise the code in debug mode. This is useful for catching things like index errors and the like.
  2. They test to make sure the code reproduces across different domain decompositions.

If we added nix tests in these settings, we could then remove these tests from the docker build and mark these tests with the coarse marker, further reducing the default time required in CI.
@spencerkclark spencerkclark mentioned this issue Apr 18, 2023
Open
@spencerkclark spencerkclark mentioned this issue Aug 28, 2023
Merged
spencerkclark added a commit that referenced this issue Sep 7, 2023
@spencerkclark
Run all tests in the nix-shell; eliminate docker infrastructure (#379)
adafc50 
This PR refactors the build infrastructure in this repo to eliminate the need for the Docker component.  All development and testing is now done in the `nix` shell.  This should be a quality of life improvement for anyone developing the fortran model, as it no longer requires maintaining checksums in two separate build environments.

In so doing it introduces the following changes:
- New `make` rules are provided for compiling the model in different modes:
  - `build` -- build executables in `repro` (our production mode) and `debug` mode.
  - `build_repro` -- build only the `repro` mode executable.
  - `build_debug` -- build only the `debug` mode executable.
- Tests are run with each of the executables available in the local `bin` directory, and are tagged with the associated compile mode.  
- An option, `check_layout_invariance`, is provided to trigger regression tests be run with a 1x2 domain decomposition instead of a 1x1 domain decomposition to check invariance to the domain decomposition layout; this is used for the all the coarse-graining regression tests and replaces the previous `test_run_reproduces_across_layouts` test that would run in the docker image.
- `debug`-mode and `repro`-mode simulations produce different answers, which is something we noticed in #364 when upgrading compiler versions as well, and so require different reference checksums.

In working on this PR, we ran the fortran model in `debug` mode in more contexts than we had previously, some of which turned up errors, which we currently work around by using `pytest.skip` (something we had implicitly already been doing before):
- #365
- #381 

Working on this PR also brought my attention to the fact that `pytest`'s `tmpdir` fixture does not automatically get cleaned up after each test; `pytest` versions older than 7.3.0 keep around directories from the last three runs of `pytest`, which fill up disk space quickly since running these tests requires creating 10's of run directories, each with their own initial conditions and input files (#380).  For the time being I manually clean up these run directories after successful tests.

Resolves #340.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Run all tests in the nix-shell; eliminate docker infrastructure ai2cm/fv3gfs-fortran
1 participant
@spencerkclark