Automate the QC tests

[mattdturner]

There needs to be an automated process to validate the QC script when updates are made. My current thoughts are to have 2 scripts. 1 of the scripts would build and submit 4 cases (qc, qc_bfb, qc_nonbfb, and qc_fail; see #161 for more information on the 4 cases), and the other script would then run the QC script in the combinations of cases. This could be done as a single script, but due to the runtime (and queue time) of the QC simulations a script might sit there idle for too long.

Additionally, there needs to be some additional thought that goes into the QC script process in order to make it easier for the user to use.

[eclare108213]

See also #109

[apcraig]

@eclare108213 @dabail10 @duvivier @JFLemieux73 We are looking for input about what other features might be useful to improve the qc validation tool. At this point, it's relatively easy to create qc cases from two sandboxes using the documentation provided here, https://cice-consortium-cice.readthedocs.io/en/latest/user_guide/ug_testing.html#end-to-end-testing-procedure. We also have a script that tests the qc test procedure. What else might be useful? Would it be useful to provide a script that a user could edit that provides the steps to run the qc test? Would it be useful to have an option on cice.setup that would automatically generate a baseline for a qc test from the master? Is the qc test straight-forward enough that what we have is fine? Is the documentation clear?

Whatever we do should take into account the fact that the qc test could be used to compare

• modified code vs baseline code where both the baseline and modified code may be specific to the testing being done (for instance, it may or may not be master)
• different options (--sets)
• different compilers
• different machines (which can't really be addressed thru any script automation)
• ability to run the baseline and non-baseline from a single sandbox or from two sandboxes, although the single sandbox can always degenerate to two sandboxes

We can develop some tools/feature that address some or all of these use cases as needed. What's missing or what's unclear? What would be useful? How is the documentation?

[eclare108213]

I agree that creating the QC tests manually is intuitive and easy to do, given the instructions that @apcraig linked above. I think my issue with it was not having the instructions at my fingertips. It would be nice to be able to push a "go" button and have it generate the results, but considering the many variations, that's probably too complex to be worth the effort (apologies to @mattdturner who already spent time on it). My recommendation at this point is that we check the user documentation (@duvivier) to make sure that it's clear to users that when they submit a PR, they (and we!) need to check for BFB, and if the changes are not BFB, then do the QC tests and provide documentation of the results. We might already be making that clear, but let's check. Sound okay?

[duvivier]

@apcraig I think the documentation is pretty clear since I just tested it recently. The only thing that is a bit (intentionally?) murky is the following line in the first paragraph: "Exactly what to test is a separate question and depends on the kinds of code changes being made."

I know that we're basically saying the tests we need for b4b is different from non-b4b. However, it might be nice to have a very simple and clear description of the procedure for the b4b tests. We already do this for the non-b4b (https://cice-consortium-cice.readthedocs.io/en/master/user_guide/ug_testing.html#code-compliance-test-non-bit-for-bit-validation AND https://cice-consortium-cice.readthedocs.io/en/master/user_guide/ug_testing.html#end-to-end-testing-procedure)

I am thinking just another subsection on the test suites section (https://cice-consortium-cice.readthedocs.io/en/master/user_guide/ug_testing.html#test-suites) saying "If you are adding code that is b4b, you should do the following procedure:

1. checkout the consortium master and run a test suite (give the command line stuff here)
2. Run the following precise command with the above test to check b4b (give command line stuff).
3. check the results and make sure all have passed. Publish to the Test-results repo in order to get a link that can be provided in your PR. "

IF changes are made, I would prefer a script that the user edits rather than an option in cice.setup. I like scripts because it eliminates some of the errors I can make typing or copying/pasting stuff in at the command line and makes reproducibility of testing a bit easier. This might also address @eclare108213 's issue of not having the instructions at your fingertips since it would be there in the script. But I don't think this is essential.

[eclare108213]

I agree with @duvivier 's suggestion for modifying the documentation. I separate user script for this purpose would be fine, and it would be nice to have in the v6 release but I'd give it lower priority than some of our other tasks.

(edited to change reference above from @apcraig to @duvivier)

[duvivier]

@apcraig is this still something we want to do before release? Or is it on hold?