Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KPP standalone interface from Obin Sturm #2482

Open
wants to merge 24 commits into
base: dev/no-diff-to-benchmark
Choose a base branch
from

Conversation

lizziel
Copy link
Contributor

@lizziel lizziel commented Sep 26, 2024

Name and Institution (Required)

Name: Lizzie Lundgren
Institution: Harvard University

Describe the update

This draft PR contains updates for KPP standalone interface from Obin Sturm (GMAO) contained in GEOS-ESM#9. That original PR is built off of branch geos/develop in the GEOS-Chem fork in GMAO's GitHub. That version is not mergable into the standard model so this PR will be used instead.

Tagging @obin1, @yantosca

Expected changes

I have pushed several updates to this PR:

  1. kpp_standalone_interface.yml configuration file:

    • Comments have been updated
    • This is now always copied to any fullchem run directory (GCClassic only).
    • The following YAML tags have been added. You now must manually activate the KPP standalone interface. Also, you can set the interval during which model state will be archived. By default this is set to the interval 1900..2100, but you can manually restrict this to e.g. the last timestep of a run.
    settings:
      activate: true                       # Main on/off switch
      start_output_at: [19000101, 000000]  # Save model state for KPP standalone
      stop_output_at:  [21000101, 000000]  # ... if between these 2 datetimes
  2. GeosCore/kpp_standalone_interface.yml has been renamed to GeosCore/kppsa_interface_mod.F90. All derived types, variables, and public-facing routines have been renamed to use the prefix KppSa_, for consistency. Derived-type
    object Kpp_Standalone_YAML has new been renamed to KppSa_State for brevity.

  3. We have added a new derived-type object KppSa_ActiveCell to hold the ActiveCell logical flag and Active_Cell_Name fields. This is declared with !$OMP THREADPRIVATE because it is updated within the main parallel loop within the DO_FULLCHEM routine.

  4. In the KppSa_Write_Samples routine (formerly Write_Samples), we have placed the I/O code within an !$OMP CRITICAL block. This will prevent other threads from trying to write to the file, which can cause incomplete and/or corrupted files.

  5. We have added a new function KppSa_Check_Time to determine if we are within the time window for archiving model state specified in the YAML file. This updates a new flag KppSa_State%SkipWriteAtThisTime, which is used to determine if we need to skip writing output on this timestep.

  6. We now print the following output at initialization:

===============================================================================
KPP STANDALONE INTERFACE

Model state will be archived at these sites:

LosAngeles               ( -118.2430,   34.0522 )
McMurdo                  (  166.6698,  -77.8455 )
Paris                    (    2.3522,   48.8566 )
Beijing                  (  116.4074,   39.9042 )
Kinshasa                 (   15.3105,   -4.3033 )
Kennaook                 (  144.6833,  -40.6833 )
Graciosa                 (  -28.0069,   39.0525 )
Utqiagvik                ( -156.7886,   71.2906 )
Ozarks                   (  -91.2590,   37.5020 )
Amazon                   (  -62.2159,   -3.4653 )
Congo                    (   12.5484,   -5.9175 )
Borneo                   (  114.0000,    0.0000 )
IndianOcean              (   87.2000,   23.0000 )
AtlanticOcean            (  -41.5748,   34.7079 )
PacificOcean             ( -121.9645,    0.0000 )
ElDjouf                  (   -6.6661,   21.5008 )

For GEOS-Chem vertical levels:
   1   2  10  23  35  48  56
Starting at 20190701 004000
Ending at   20190701 010000
===============================================================================

Reference(s)

None

Related Github Issue

@lizziel lizziel added category: Feature Request New feature or request no-diff-to-benchmark This update will not change the results of fullchem benchmark simulations Topic: Fullchem Simulations Related to full-chemistry simulations labels Sep 26, 2024
Signed-off-by: Lizzie Lundgren <elundgren@seas.harvard.edu>
@obin1
Copy link

obin1 commented Sep 26, 2024

Hi Lizzie, thanks for getting this started! I'm still working on redoing a few c360 runs with the new compute resources on Discover before we can write up the manuscript about this tool and species-level tolerances. I will keep you all posted when that's out as a preprint, and thanks again for moving this PR upstream

@yantosca yantosca added this to the 14.5.1 milestone Sep 26, 2024
@yantosca yantosca self-assigned this Sep 27, 2024
@yantosca yantosca added the topic: Chemical Mechanisms Related to KPP and/or GEOS-Chem chemistry mechanisms label Sep 27, 2024
@yantosca yantosca modified the milestones: 14.5.1, 14.5.0 Sep 27, 2024
Copy link
Contributor

@yantosca yantosca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@obin1: Also not related to this PR, but if you have any python scripts to read & plot output from the box model we shoudl think about bringing those into GCPy.

GeosCore/CMakeLists.txt Outdated Show resolved Hide resolved
run/shared/kpp_standalone_interface.yml Show resolved Hide resolved
@lizziel lizziel changed the base branch from dev/14.5.0 to dev/no-diff-to-benchmark September 27, 2024 18:18
run/shared/kpp_standalone_interface.yml
- Fixed typo: "output_dir" -> "output_directory"

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
GeosCore/fullchem_mod.f90
- Use keyword arguments, for clarity

GeosCore/kpp_standalone_interface.F90
- Remove Active_Cell and Active_Cell_Name from KPP_Standalone_YAML
- Add new derived type KPP_Standalone_ActiveCell_Type, which is now
  declared with !$OMP THREADPRIVATE.  This contains the Active_Cell
  and Active_Cell_Name fields.
- Added new variable KPP_Standalone_ActiveCell, based on
  new KPP_Standalone_ActiveCell_Type

NOTE: The KPP_Standalone_YAML is initialized outside of a parallel
loop, so it does not need to be declared !$OMP THREADPRIVATE.  But the
choice of whether an (I,J,L) location corresponds to one of the
"active_cells" (listed in the kpp_standalone_interface.yml file) happens
within a parallelized loop.  Thus we need to move the Active_Cell and
Active_Cell_Name fields out of KPP_Standalone_YAML and into
KPP_Standalone_ActiveCell.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@yantosca
Copy link
Contributor

@obin1 @lizziel: See the updates in my bugfix branch below. This fixes the parallelization issue listed above.

@msulprizio msulprizio removed this from the 14.5.0 milestone Sep 30, 2024
run/shared/kpp_standalone_interface.yml
- Reorganize file so that "settings:" comes first, then "active_cells:",
  and then "locations:"
- Add "settings:activate" YAML tag to toggle the KPP standalone
  interface on or off

GeosCore/kpp_standalone_interface.F90
- Add call to QFYAML_Add_Get to parse the "settings:activate" YAML tag
- Set KPP_Standalone_YAML%SkipIt to .TRUE. if "settings:activate"
  is false.  This will cause the code to ignore saving out the state
  of the model for the KPP standalone even if kpp_standalone_interface.yml
  is present in the run directory.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
run/GCClassic/createRunDir.sh
- Copy run/shared/kpp_standalone_interface.yml to fullchem rundirs

run/shared/cleanRunDir.sh
- Skip removing bpch files (and {diag,tracer}info.dat files), we no
  longer generate bpch output
- Add comments and usage examples
- Remove all fort.* files
- Remove OututDir/*.txt files as well (these are KPP standalone
  interface files)

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
.gitignore
- Also ignore *.exe files

GeosCore/fullchem_mod.F90
- Remove State_Grid from call to Check_ActiveCell

GeosCore/kpp_standalone_interface.F90
- Updated subroutine header comments
- Added cosmetic changes for clarity (mostly making code fit within
  80 characters, for better readability)
- Removed Active_Cell and Active_Cell_name from the
  KPP_Standalone_Interface_Type (should have been done previously)
- Remove State_Grid argument from Check_ActiveCell, this was only
  used for debugging.   Also removed commented out debug prints.
- Use Format statements with write statments where expedient
- Echo a message when the KPP standalone interface is manually disabled
  (i.e. when settings:activate = false)
- Added display of location names and lon/lats at end of the routine
  Config_KPP_Standalone
- In routine Write_Samples, wrap file I/O in an !$OMP CRITICAL block,
  in order to prevent more than one thread from writing to each file.
- Removed RETURN statement from within !$OMP CRITICAL block, this
  is not allowed.
- Bug fix: Write Kpp_Standalone_ActiveCell%Active_Cell_Name to file,
  and not Kpp_Standalone_YAML%Active_Cell_Name (which has been removed).

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
GeosCore/kpp_standalone_interface.F90
- Moved to kppsa_interface_mod.F90

GeosCore/kppsa_interface_mod.F90
- Moved from kpp_standalone_interface
- Renamed types, variables and routines using "KppSa" prefix:
  - KPP_Standalone_Interface_Type -> KppSa_Interface_Type
  - KPP_Standalone_ActiveCell_Type -> KppSa_ActiveCell_Type
  - KPP_Standalone_YAML -> KppSa_State
  - KPP_Standalone_ActiveCell -> KppSa_ActiveCell
  - Check_Domain -> KppSa_Check_Domain
  - Check_ActiveCell -> KppSa_Check_ActiveCell
  - Config_KPP_Standalone -> KppSa_Config
  - Write_Samples -> KppSa_Write_Samples
  - Cleanup_KPP_Standalone -> KppSa_Cleanup
- Added KppSa_Check_Time function to determine if it is time to
  write KPP standalone output.  This allows you to only write output
  e.g. at the end of a run instead of for each timestep.
- Added KppSa_State%SkipWriteAtThisTime field, which is used to
  determine if we need to exit a routine early.

GeosCore/fullchem_mod.F90
- "USE Kpp_Standalone_Interface" -> "USE KppSa_Interface_Mod"
- Call renamed routines from kppsa_interface_mod.F90
- Now call KppSa_Check_Domain only if it is the first call to
  DO_FULLCHEM.  This is to avoid repeated computations.
- Now call KppSa_Check_Time to determine if we are in the time window
  when the model state should be archived to disk.
- Updated comments and comment headers

run/shared/kpp_standalone_interface.yml
- Added "start_output_at" to denote starting date time for archiving
  model state
- Added "stop_output_at" to denote ending date time for archiving
  model state
- Updated comments

CHANGELOG.md
GeosCore/CMakeLists.txt
- Updated accordingly

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
GeosCore/kppsa_interface_mod.F90
- In routine KppSa_Check_Time:
  - rRmoved leftover debug print statements
- In routine Kpp_Check_ActiveCell
  - Added an IF statement to exit after setting KppSa_State%Active_Cell to .FALSE.
    and  KppSa_Active_Cell_Name to '' if we are outside of the time
    window specified in the kpp_standalone_interface.yml fiel.
  - This will ensure that we only archive model state to disk during
    the specified time window, which helps with computatonal efficiency.
- In routine KppSa_Config:
  - Now write starting & ending date of archival window to log file

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@yantosca yantosca marked this pull request as ready for review October 8, 2024 19:51
run/GCHP/createRunDir.sh
- Added an if statement to copy run/shared/kpp_standalone_interface.yml
  to GCHP fullchem run directories (any option)

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@yantosca
Copy link
Contributor

yantosca commented Oct 9, 2024

@lizziel, @obin1: I have confirmed that this PR works in GCHP. I was able to restrict output to the last timestep of a 1-hr GCHP run and got a bunch of files timestamped at 00:40z.

@yantosca yantosca requested a review from obin1 October 9, 2024 19:09
@yantosca yantosca added this to the 14.6.0 milestone Oct 10, 2024
@yantosca
Copy link
Contributor

@obin1 @lizziel: FYI I have also modified this PR to save out the ICNTRL and RCNTRL solver options to the output file. This is necessary in case we want to use different solvers.

Now that we have per-species ATOL settings in GEOS-Chem 14.5.0, I will add another fix to save that out. This will be needed to set the ATOL for family/dummy species to a high value so as to keep them out of the error norm computations.

GeosCore/fullchem_mod.F90
- Pass KPP integrator arguments ICNTRL and RCNTRL to KppSa_Write_Samples
  so that we can include it in the KPP standalone output.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
GeosCore/CMakeLists.txt
- Changed permission from chmod 755 to chmod 644

GeosCore/kppsa_interface_mod.F90
- Now accept ICNTRL and RCNTRL as inputs to routine KppSa_Write_Samples
- Change number of header lines from 48 to 60
- Better separate Meteorological metadata and KPP parameters
  for readability
- Write out ICNTRL as 2 lines of format 10i6
- Write out RCNTRL as 4 lines of format 5f13.6
- Changed format e25.16e3 to es25.16e3, which changes the output from
  e.g. 0.5e1 to 5.0e0, etc., which is more standard
- Write out ATOL after the species concentration with format es10.2e2
- Cosmetic changes (indentation, comments)

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
GeosCore/kppsa_interface_mod.F90
- In routine KppSa_Write_Samples:
  - Added a "," in between the concentration and abs tolerance output
  - Simplified FORMAT statements 120 and 130
  - Added FORMAT statements 140 and 150

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: Feature Request New feature or request no-diff-to-benchmark This update will not change the results of fullchem benchmark simulations topic: Chemical Mechanisms Related to KPP and/or GEOS-Chem chemistry mechanisms Topic: Fullchem Simulations Related to full-chemistry simulations
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants