Update test features: add create_fails script, QC process test, and wait on baseline results #644
Conversation
|
I just updated the regression testing so qcchk tests actually uses the QC validation process as part of their regression validation. I tested this on the new gx3 cases and it seems to work well. This will be applied to the gx1 qc test moving forward as well. These tests are defined in the prod_suite and are only functional on cheyenne at the moment (because of the python env setup). The prod_suite is run weekly on only cheyenne for 3 compilers. |
|
|
||
| rm $tmpfile | ||
| echo "$0 done" | ||
| echo "Not passed tests written to file...... $outfile" |
There was a problem hiding this comment.
Maybe "Failed tests" instead ?
There was a problem hiding this comment.
Failed tests isn't quite accurate. Failed != Not passed. But I think you're right on reflection that failed is probably a better word here. I'll change it.
| echo "$0 done" | ||
| echo "Not passed tests written to file...... $outfile" | ||
| echo "To setup a new test suite, try something like" | ||
| echo " ./cice.setup --suite fails.ts ..." |
There was a problem hiding this comment.
does this work ? I guess you first have to move fails.ts to configuration/scripts/tests/ in the sandbox ? it this what you mean by "something like" ?
There was a problem hiding this comment.
You're right that It wouldn't work exactly as described. You can move fails.ts to the cice.setup directory too. It works from there. But you're right, maybe I'll clarify that a bit. I don't want to get too deep in the weeds though.
PR checklist
Short (1 sentence) summary of your PR:
Add create_fails script, QC process test, and batch wait on baseline results when using --diff in test suites
Developer(s):
apcraig
Suggest PR reviewers from list in the column to the right.
Please copy the PR test results link or provide a summary of testing completed below.
All tests are bit-for-bit with testing on cheyenne, https://github.com/CICE-Consortium/Test-Results/wiki/cice_by_hash_forks#adab3d33996089d66c9a69a26815409c3edc584c. Tested scripts several ways/times.
How much do the PR code changes differ from the unmodified code?
Does this PR create or have dependencies on Icepack or any other models?
Does this PR add any new test cases?
Is the documentation being updated? ("Documentation" includes information on the wiki or in the .rst files from doc/source/, which are used to create the online technical docs at https://readthedocs.org/projects/cice-consortium-cice/. A test build of the technical docs will be performed as part of the PR testing.)
Please provide any additional information or relevant details below:
Add create_fails script. This looks at the results of a test suite and creates a new test suite file (fails.ts) with only the failed tests. That can be edited and passed into cice.setup (--suite fails) to continue code validation. Hopefully this addresses and closes Re-running selected cases in suites #609.
Add qcchk and qcchkf tests which use the QC validation process to compare to other results. qcchk verifies the QC test passes and qcchkf verifies that the QC test fails. Add gx3 qc test to prod_suite.ts and verify it works on cheyenne. Had to add the appropriate python env setup to the cheyenne env files. The current tests just validate the QC process in bfbcomp (--diff) mode.
Use QC check for regression testing for qcchk tests. With the gx1 qc and gx3 gc tests in the prod_suite, this will now use the QC check as part of the regression testing.
Add batch checks when bfbflag (--diff) checking is done to make sure the baseline is complete before the comparison is done. Up to now, there has been an issue if the baseline test is NOT completed before a test that is comparing to that baseline. With this implementation, if the baseline job is still in the queue, then the comparison waits for it to complete in similar style to poll_queue.csh.