Skip to content

Modified config_archive.xml to archive CAM+DART files. #1302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: cam_development
Choose a base branch
from
Open

Modified config_archive.xml to archive CAM+DART files. #1302

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f69921e
Added code to make st-archive handle DART files
kdraeder Apr 22, 2025
1084a90
Draft of entry to satisfy development workflow recommendations
kdraeder Apr 22, 2025
a42315e
Added line to archive .i. files as both restarts and history
kdraeder May 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions cime_config/config_archive.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,14 @@
<rest_file_extension>r</rest_file_extension>
<rest_file_extension>rh\d*</rest_file_extension>
<rest_file_extension>rs</rest_file_extension>
<hist_file_extension>h\d*.*\.nc$</hist_file_extension>
<hist_file_extension>i\..*\.nc$</hist_file_extension>
Comment on lines -7 to -8
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering about the implications of removing this line: <hist_file_extension>i\..*\.nc$</hist_file_extension> on other CAM jobs. I will admit that I am by no means an expert on this file and archiving in general, but it appears that we'd no longer be archiving intantaneous history files? Should this line perhaps remain?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That line was replaced by a <rest_file_extension>i\..* line.
This line handles the CAM "initial file", which is the only '.i.' file that I know of.
There may be '.i[a-z0-9]+.' files (someday), but they would be handled differently.
But I would like to hear the opinion of CAM developers about this.
DART needs to have the .i. files archived with the restarts because that's the file
containing the model state which DART needs to use to start CAM at the beginning of each DA cycle.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to be sure we are talking about the same thing.
instantaneous files are the .h0i., .h1i., ... which is different from the initial file .i..

Where will these files go?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that the .h[0-9]i. files will still be handled by the xlm line:
<hist_file_extension>h\d*.*\.nc(\.gz)?$</hist_file_extension>
which I parse as "h+any_digit+any_number_of_any_characters".

The '.i.' file is currently handled by the line:
<hist_file_extension>i\.\d.*\.nc(\.gz)?$</hist_file_extension>
I believe that the parsing code adds '.' before the regex listed in these .xml files,
so this line only applies to '.i.' and not to the .h[0-9]i. files.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto @cecilehannay . We need to know if this change moves the location of history output and breaks everyone's analysis scripts. Fixable, yes, but it would be good to know in advance.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my test the .h0i. files ended up in archive/atm/hist, archive/rest/$date, and in $rundir.
I only requested 1 CAM history file.
The .i. files ended up in archive/rest/$date and $rundir.
That's what I intended, but someone else should test it too, especially if I failed to create a CAM file type.
The archive directory is /glade/derecho/scratch/raeder/St_BHISTC_LTso-SE_st-arch/archive.
Thanks for looking into this!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cecilehannay - Would you be able to checkout @kdraeder's branch and make a quick run similar to ones you are currently making and make sure all that all the history files end up where they should?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cecilehannay - I have two CAM checkouts for you to test:

  • /glade/derecho/scratch/cacraig/cam6_4_089 - A straight checkout of cam6_4_089
  • /glade/derecho/scratch/cacraig/cam6_4_089_raeder - A checkout of cam6_4_089 with the DART changes

Please let us know what your runs indicate for both the history and the initial condition files (and any other files for that matter if they change)

<rest_file_extension>i\.\d.*\.nc(\.gz)?$</rest_file_extension>
<hist_file_extension>i\..*\.nc(\.gz)?$</hist_file_extension>
<hist_file_extension>h\d*.*\.nc(\.gz)?$</hist_file_extension>
<hist_file_extension>e</hist_file_extension>
<rest_history_varname>nhfil</rest_history_varname>
<rpointer>
<rpointer_file>rpointer.atm$NINST_STRING</rpointer_file>
<rpointer_content>$CASE.cam$NINST_STRING.r.$DATENAME.nc </rpointer_content>
<rpointer_content>$CASE.cam$NINST_STRING.r.$DATENAME.nc</rpointer_content>
</rpointer>
<test_file_names>
<tfile disposition="copy">rpointer.atm</tfile>
Expand Down
60 changes: 60 additions & 0 deletions doc/ChangeLog
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,65 @@
===============================================================

Tag name: cam6_4_081
Originator(s): kdraeder
Date: 22 April 2025
One-line Summary: Make st_archive archive DART output files.
Github PR URL: None yet; instructions said to modify this file and open an issue before opening a PR.

Purpose of changes (include the issue number and title text for each relevant GitHub issue):
Make st_archive archive CAM+DART output files. #1301

Describe any changes made to build system: None

Describe any changes made to the namelist: None

List any changes to the defaults for the boundary datasets: None

Describe any substantial timing or memory changes: None

Code reviewed by:

List all files eliminated: None

List all files added and what they do: None

List all existing files that have been modified, and describe the changes:
cime_config/config_archive.xml:
Changed initial files to rest_file_extension because DART needs them to be handled like restart files.
Enabled archive of file type .e. (generated by the esp component; DART)
Made st_archive recognize compressed (.gz) files, which is extremely helpful in DART runs.

If there were any failures reported from running test_driver.sh on any test
platform, and checkin with these failures has been OK'd by the gatekeeper,
then copy the lines from the td.*.status files for the failed tests to the
appropriate machine below. All failed tests must be justified.
I did not run test_driver.sh.
I did test this in a B compset and will include details in the PR.

derecho/intel/aux_cam:
derecho/nvhpc/aux_cam:
izumi/nag/aux_cam:
izumi/gnu/aux_cam:
These are xml changes and should be independent of fortran compiler.
I used the default compiler for this CESM branch.

CAM tag used for the baseline comparison tests if different than previous
tag:
I didn't compare results against a baseline run;
changes are only to the config_archive.xml file.

Summarize any changes to answers, i.e.,
- what code configurations: None
- what platforms/compilers: None
- nature of change (roundoff; larger than roundoff but same climate; new
climate): None

If this tag changes climate describe the run(s) done to evaluate the new
climate in enough detail that it(they) could be reproduced, i.e.,
None

===============================================================

Tag name: cam6_4_087
Originator(s): eaton
Date: 21 April 2025
Expand Down