Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update MPI usage when using CESM #1951

Merged
merged 1 commit into from
Sep 12, 2023
Merged

Update MPI usage when using CESM #1951

merged 1 commit into from
Sep 12, 2023

Conversation

lizziel
Copy link
Contributor

@lizziel lizziel commented Sep 11, 2023 •
edited
Loading

Name and Institution (Required)

Name: Lizzie Lundgren
Institution: Harvard University

Describe the update

This pull request updates CESM-only code to use new MPI conventions in CESM. It has no impact on GEOS-Chem Classic or GCHP.

Expected changes

SPMD ifdefs are removed in MODEL_CESM code blocks and calls to subroutine MPIBCAST are changed to use MPI_BCAST. Module USE statements and subroutine arguments are changed accordingly. Error handling is also added for the MPI broadcast calls.

Related Github Issue(s)

This update is associated with a pull request to ESCOMP/CAM to bring GEOS-Chem chemistry into CESM. ESCOMP/CAM#484

@lizziel
Update MPI usage when using CESM
e84804a 
- Remove SPMD cpp keyword usage
- Replace mpishorthand module with spmd_utils
- Specify exactly what is used in spmd_utils
- Use mpi_bcast instead of mpibcast
- Use subroutine endrun from cam_abortutils for broadcast error handling
- Removed unused CESM modules from fjx_interface_mod.F90

Signed-off-by: Lizzie Lundgren <elundgren@seas.harvard.edu>
@lizziel lizziel added category: Interface to External Model Related to GEOS-Chem updates needed to interface with other models topic: CESM Related to running GEOS-Chem in CESM labels Sep 11, 2023
@lizziel lizziel added this to the 14.2.1 milestone Sep 11, 2023
yantosca
yantosca approved these changes Sep 11, 2023
View reviewed changes
Copy link
Contributor

@yantosca yantosca left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These updates look good to merge

@lizziel: Question for the future. Assuming that there are more PRs like this, do you think that these #ifdefs can be abstracted out of the code (maybe in #include *.H files). Just trying to think of a way to potentially prevent a lot of extra code from making the modules too long. Open to your thoughts.

@yantosca yantosca self-assigned this Sep 11, 2023
@lizziel
Copy link
Contributor Author

lizziel commented Sep 11, 2023

Yes, I think I should go through and try to minimize code via abstration. I will make a PR for this. This is especially true for MODEL_GEOS!

@lizziel
Copy link
Contributor Author

lizziel commented Sep 11, 2023

See #1953

@yantosca
Copy link
Contributor

Thanks @lizzlel. In the meantime I'll go ahead and bring this PR into the 14.2.1 branch.

@yantosca
Copy link
Contributor

yantosca commented Sep 12, 2023 •
edited
Loading

All GEOS-Chem Classic integration tests passed (except for TOMAS, which is a known issue:

==============================================================================
GEOS-Chem Classic: Execution Test Results

GCClassic #f3509f8 Merge PR #1946 (Fix build issues in GCClassic w/ the custom mech)
GEOS-Chem #e696799c2 Merge PR #1917 (Update GEPA CH4 emissions to Gridded GHGI v2)
HEMCO     #daf54d0 Merge PR #235 (Bug fix for vertical regridding)

Using 24 OpenMP threads
Number of execution tests: 26

Submitted as SLURM job: 1747431
==============================================================================
 
Execution tests:
------------------------------------------------------------------------------
gc_05x0625_NA_47L_merra2_CH4........................Execute Simulation....PASS
gc_05x0625_NA_47L_merra2_fullchem...................Execute Simulation....PASS
gc_4x5_47L_merra2_fullchem..........................Execute Simulation....PASS
gc_4x5_47L_merra2_fullchem_TOMAS15..................Execute Simulation....FAIL
gc_4x5_47L_merra2_fullchem_TOMAS40..................Execute Simulation....FAIL
gc_4x5_merra2_aerosol...............................Execute Simulation....PASS
gc_4x5_merra2_carbon................................Execute Simulation....PASS
gc_4x5_merra2_CH4...................................Execute Simulation....PASS
gc_4x5_merra2_CO2...................................Execute Simulation....PASS
gc_4x5_merra2_fullchem..............................Execute Simulation....PASS
gc_4x5_merra2_fullchem_aciduptake...................Execute Simulation....PASS
gc_4x5_merra2_fullchem_APM..........................Execute Simulation....PASS
gc_4x5_merra2_fullchem_benchmark....................Execute Simulation....PASS
gc_4x5_merra2_fullchem_complexSOA...................Execute Simulation....PASS
gc_4x5_merra2_fullchem_complexSOA_SVPOA.............Execute Simulation....PASS
gc_4x5_merra2_fullchem_LuoWd........................Execute Simulation....PASS
gc_4x5_merra2_fullchem_marinePOA....................Execute Simulation....PASS
gc_4x5_merra2_fullchem_RRTMG........................Execute Simulation....PASS
gc_4x5_merra2_Hg....................................Execute Simulation....PASS
gc_4x5_merra2_metals................................Execute Simulation....PASS
gc_4x5_merra2_POPs_BaP..............................Execute Simulation....PASS
gc_4x5_merra2_tagCH4................................Execute Simulation....PASS
gc_4x5_merra2_tagCO.................................Execute Simulation....PASS
gc_4x5_merra2_tagO3.................................Execute Simulation....PASS
gc_4x5_merra2_TransportTracers......................Execute Simulation....PASS
gc_4x5_merra2_TransportTracers_LuoWd................Execute Simulation....PASS
 
Summary of test results:
------------------------------------------------------------------------------
Execution tests passed: 24
Execution tests failed: 2
Execution tests not yet completed: 0

Also all integration tests were identical to the prior integration test (PR #1946) except for:

  • Carbon, CH4, tagCH4 simulations (updated emissions)
  • APM (parallelization issue)
  • RRTMG diagnostic output (probable parallelization issue)
Checking gc_05x0625_NA_47L_merra2_CH4
   -> 4 differences found in OutputDir
      * GCC_14.2.1_r22_g10/rundirs/gc_05x0625_NA_47L_merra2_CH4/OutputDir/GEOSChem.CH4.20190101_0000z.nc4 
        GCC_r23/rundirs/gc_05x0625_NA_47L_merra2_CH4/OutputDir/GEOSChem.CH4.20190101_0000z.nc4 
      * GCC_14.2.1_r22_g10/rundirs/gc_05x0625_NA_47L_merra2_CH4/OutputDir/GEOSChem.Metrics.20190101_0000z.nc4 
        GCC_r23/rundirs/gc_05x0625_NA_47L_merra2_CH4/OutputDir/GEOSChem.Metrics.20190101_0000z.nc4 
      * GCC_14.2.1_r22_g10/rundirs/gc_05x0625_NA_47L_merra2_CH4/OutputDir/GEOSChem.SpeciesConc.20190101_0000z.nc4 
        GCC_r23/rundirs/gc_05x0625_NA_47L_merra2_CH4/OutputDir/GEOSChem.SpeciesConc.20190101_0000z.nc4 
      * GCC_14.2.1_r22_g10/rundirs/gc_05x0625_NA_47L_merra2_CH4/OutputDir/HEMCO_diagnostics.201901010000.nc 
        GCC_r23/rundirs/gc_05x0625_NA_47L_merra2_CH4/OutputDir/HEMCO_diagnostics.201901010000.nc 
   -> 1 difference found in Restarts
      * GCC_14.2.1_r22_g10/rundirs/gc_05x0625_NA_47L_merra2_CH4/Restarts/GEOSChem.Restart.20190101_0020z.nc4 
        GCC_r23/rundirs/gc_05x0625_NA_47L_merra2_CH4/Restarts/GEOSChem.Restart.20190101_0020z.nc4 

Checking gc_05x0625_NA_47L_merra2_fullchem
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_47L_merra2_fullchem
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_47L_merra2_fullchem_TOMAS15
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_47L_merra2_fullchem_TOMAS40
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_aerosol
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_carbon
   -> 2 differences found in OutputDir
      * GCC_14.2.1_r22_g10/rundirs/gc_4x5_merra2_carbon/OutputDir/GEOSChem.SpeciesConc.20190101_0000z.nc4 
        GCC_r23/rundirs/gc_4x5_merra2_carbon/OutputDir/GEOSChem.SpeciesConc.20190101_0000z.nc4 
      * GCC_14.2.1_r22_g10/rundirs/gc_4x5_merra2_carbon/OutputDir/HEMCO_diagnostics.201901010000.nc 
        GCC_r23/rundirs/gc_4x5_merra2_carbon/OutputDir/HEMCO_diagnostics.201901010000.nc 
   -> 1 difference found in Restarts
      * GCC_14.2.1_r22_g10/rundirs/gc_4x5_merra2_carbon/Restarts/GEOSChem.Restart.20190101_0100z.nc4 
        GCC_r23/rundirs/gc_4x5_merra2_carbon/Restarts/GEOSChem.Restart.20190101_0100z.nc4 

Checking gc_4x5_merra2_CH4
   -> 4 differences found in OutputDir
      * GCC_14.2.1_r22_g10/rundirs/gc_4x5_merra2_CH4/OutputDir/GEOSChem.CH4.20190101_0000z.nc4 
        GCC_r23/rundirs/gc_4x5_merra2_CH4/OutputDir/GEOSChem.CH4.20190101_0000z.nc4 
      * GCC_14.2.1_r22_g10/rundirs/gc_4x5_merra2_CH4/OutputDir/GEOSChem.Metrics.20190101_0000z.nc4 
        GCC_r23/rundirs/gc_4x5_merra2_CH4/OutputDir/GEOSChem.Metrics.20190101_0000z.nc4 
      * GCC_14.2.1_r22_g10/rundirs/gc_4x5_merra2_CH4/OutputDir/GEOSChem.SpeciesConc.20190101_0000z.nc4 
        GCC_r23/rundirs/gc_4x5_merra2_CH4/OutputDir/GEOSChem.SpeciesConc.20190101_0000z.nc4 
      * GCC_14.2.1_r22_g10/rundirs/gc_4x5_merra2_CH4/OutputDir/HEMCO_diagnostics.201901010000.nc 
        GCC_r23/rundirs/gc_4x5_merra2_CH4/OutputDir/HEMCO_diagnostics.201901010000.nc 
   -> 1 difference found in Restarts
      * GCC_14.2.1_r22_g10/rundirs/gc_4x5_merra2_CH4/Restarts/GEOSChem.Restart.20190101_0100z.nc4 
        GCC_r23/rundirs/gc_4x5_merra2_CH4/Restarts/GEOSChem.Restart.20190101_0100z.nc4 

Checking gc_4x5_merra2_CO2
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_aciduptake
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_APM
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_benchmark
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_complexSOA
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_complexSOA_SVPOA
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_LuoWd
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_marinePOA
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_fullchem_RRTMG
   -> 1 difference found in OutputDir
      * GCC_14.2.1_r22_g10/rundirs/gc_4x5_merra2_fullchem_RRTMG/OutputDir/GEOSChem.RRTMG.20190701_0000z.nc4 
        GCC_r23/rundirs/gc_4x5_merra2_fullchem_RRTMG/OutputDir/GEOSChem.RRTMG.20190701_0000z.nc4 
   -> No differences in Restarts

Checking gc_4x5_merra2_Hg
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_metals
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_POPs_BaP
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_tagCH4
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_tagCO
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_tagO3
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_TransportTracers
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gc_4x5_merra2_TransportTracers_LuoWd
   -> No differences in OutputDir
   -> No differences in Restarts

@yantosca
Copy link
Contributor

All GCHP integration tests passed:

==============================================================================
GCHP: Execution Test Results

GCClassic #9285236 GEOS-Chem submod update: Merge PR #1946 (Fix custom mech build issues)
GEOS-Chem #e696799c2 Merge PR #1917 (Update GEPA CH4 emissions to Gridded GHGI v2)
HEMCO     #daf54d0 Merge PR #235 (Bug fix for vertical regridding)

Number of execution tests: 5

Submitted as SLURM job: 1747529
==============================================================================
 
Execution tests:
------------------------------------------------------------------------------
gchp_merra2_fullchem................................Execute Simulation....PASS
gchp_merra2_fullchem_benchmark......................Execute Simulation....PASS
gchp_merra2_fullchem_RRTMG..........................Execute Simulation....PASS
gchp_merra2_tagO3...................................Execute Simulation....PASS
gchp_merra2_TransportTracers........................Execute Simulation....PASS
 
Summary of test results:
------------------------------------------------------------------------------
Execution tests passed: 5
Execution tests failed: 0
Execution tests not yet completed: 0

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%  All execution tests passed!  %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

All GCHP integration tests were also zero-diff w/r/t the prior integration test: (PR #1946)

Checking gchp_merra2_fullchem
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gchp_merra2_fullchem_benchmark
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gchp_merra2_fullchem_RRTMG
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gchp_merra2_tagO3
   -> No differences in OutputDir
   -> No differences in Restarts

Checking gchp_merra2_TransportTracers
   -> No differences in OutputDir
   -> No differences in Restarts

@yantosca yantosca merged commit 9afd218 into dev/14.2.1 Sep 12, 2023
@yantosca yantosca deleted the feature/update_cesm_mpi_usage branch September 12, 2023 13:31
This was referenced Sep 12, 2023
Merged
Merged
@lizziel lizziel mentioned this pull request Sep 14, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yantosca yantosca yantosca approved these changes

Assignees

@yantosca yantosca

Labels
category: Interface to External Model Related to GEOS-Chem updates needed to interface with other models topic: CESM Related to running GEOS-Chem in CESM
Projects
None yet
Milestone
14.2.1
Development

Successfully merging this pull request may close these issues.
2 participants
@lizziel @yantosca