Subset mesh files for regional cases and add user-mods for regional cases.

[negin513]

Description of changes

Specific notes

This PR adds capability to subset mesh files for a regional case based on the region of choice.

Contributors other than yourself, if any: @swensosc and @adrifoster

CTSM Issues Fixed (include github issue #): #1513

Are answers expected to change (and if so in what way)? No

Any User Interface Changes (namelist or namelist defaults changes)? No, option --subset-mesh is added to subset_data.

Testing performed, if any:
Plotting the mesh files to make sure the extracted mesh file makes sense.

[jkshuman]

@negin513 this appears to subset the files appropriately when I view them.
I am trying to run a regional case and running into problems. I have been trying out a few things, but we should set up a time to meet.

[jkshuman]

@negin513 comments on using the script:
Is there help? I wasn't able to access it when I tried ./subset_data -help or ./subset_data --help
I like the progress bar. Would be great to add a print statement that identifies the datasets being subset.
I got this message, but it still seemed to create the regional subset:

/glade/work/jkshuman/sfcdata/subset_mesh/ctsm/tools/site_and_regional/../../python/ctsm/site_and_regional/regional_case.py:460: SerializationWarning: saving variable elementMask with floating point data as an integer dtype without any _FillValue to use for NaNs
f_out.to_netcdf(mesh_out)

I did not see a way to specify the directory where these files would go. It took me a bit to figure out where they were located.

[jkshuman]

In conversation with @adrifoster on a separate issue I shared this error in trying to get this to run a regional simulation. @adrifoster suggested that the below error was associated with a typo and to adjust to LND_DOMAIN_MESH

errput: ERROR: No variable LAND_DOMAIN_MESH found in case

after fixing the typo to LND_DOMAIN_MESH I got it to build and submit, but then had the regional case fail with this error in the cesm.log:

10:MPT ERROR: Rank 10(g:10) is aborting with error code 1.
10: Process ID: 61846, Host: r2i0n25, Program: /glade/scratch/jkshuman/test9_mesh_user-mods_SAmer_0.9x1.25_b8c91151_e54a4947/bld/cesm.exe
10: MPT Version: HPE MPT 2.22 03/31/20 15:59:10
10:
10:MPT: --------stack traceback-------
-1:MPT ERROR: MPI_COMM_WORLD rank 7 has terminated without calling MPI_Finalize()
-1: aborting job

[negin513]

@jkshuman Thanks for your comments.

@negin513 comments on using the script: Is there help? I wasn't able to access it when I tried ./subset_data -help or ./subset_data --help I like the progress bar.

You can see the subset_data help options using the following:

./subset_data --help 
or 
./subset_data -h

You can see specify to see options only specific to region or single point:

Show only regional option:

./subset_data region -h
or 
./subset_data region --help

Show only single point options:

./subset_data point -h
or 
./subset_data point --help

[jkshuman]

@negin513 thanks. the help wasn't working yesterday, but it seems to be a python error. If I update my python environment it works.

(base) jkshuman@cheyenne1:/glade/work/jkshuman/sfcdata/subset_mesh/ctsm/tools/site_and_regional> ./subset_data region -h
Traceback (most recent call last):
File "./subset_data", line 39, in
from ctsm.subset_data import main
File "/glade/work/jkshuman/sfcdata/subset_mesh/ctsm/tools/site_and_regional/../../python/ctsm/subset_data.py", line 65, in
from ctsm.site_and_regional.base_case import DatmFiles
File "/glade/work/jkshuman/sfcdata/subset_mesh/ctsm/tools/site_and_regional/../../python/ctsm/site_and_regional/base_case.py", line 17, in
import numpy as np
ModuleNotFoundError: No module named 'numpy'
(base) jkshuman@cheyenne1:/glade/work/jkshuman/sfcdata/subset_mesh/ctsm/tools/site_and_regional>

for the help description on the regional lat and lon can you include the directions, as in NE corner etc? (rather than start and end)

[negin513]

Would be great to add a print statement that identifies the datasets being subset.

Thank Jackie. The additional messages will be printed out using --debug or -v and --verbose. The interface is similar to ./subset_data point. I am in favor of adding those as print statements but the SE team decided to refrain from any print statements from python code in #1461.

I got this message, but it still seemed to create the regional subset:

/glade/work/jkshuman/sfcdata/subset_mesh/ctsm/tools/site_and_regional/../../python/ctsm/site_and_regional/regional_case.py:460: SerializationWarning: saving variable elementMask with floating point data as an integer dtype without any _FillValue to use for NaNs
f_out.to_netcdf(mesh_out)

This is an xarray warning but I will change this to ignore this message.

[negin513]

@negin513 comments on using the script: Is there help? I wasn't able to access it when I tried ./subset_data -help or ./subset_data --help I like the progress bar. Would be great to add a print statement that identifies the datasets being subset. I got this message, but it still seemed to create the regional subset:

/glade/work/jkshuman/sfcdata/subset_mesh/ctsm/tools/site_and_regional/../../python/ctsm/site_and_regional/regional_case.py:460: SerializationWarning: saving variable elementMask with floating point data as an integer dtype without any _FillValue to use for NaNs
f_out.to_netcdf(mesh_out)

I did not see a way to specify the directory where these files would go. It took me a bit to figure out where they were located.

The interface is similar to single point subsetting. So by default it create a folder called subset_data_regional in the directory where you are running this from. For single point it does the same thing with subset_data_single_point directory.

You can specify the output directory using command line argument --outdir.
For example the following will create a directory called this_region where you are running the code from.

./subset_data region --create-surface --create-mesh --outdir this_region

[jkshuman]

@negin513 I updated the comment about the help failure to include the error associated with python.

[negin513]

@negin513 thanks. the help wasn't working yesterday, but it seems to be a python error. If I update my python environment it works.

for the help description on the regional lat and lon can you include the directions, as in NE corner etc? (rather than start and end)

Good idea.I will change the help message to show westernmost/easternmost language instead of start and end.

[jkshuman]

path to the latest case: /glade/work/jkshuman/FATES_cases/test/test9_mesh_user-mods_SAmer_0.9x1.25_b8c91151_e54a4947

[wwieder]

@negin513 do you local changes to this PR that you can push so we can work on bringing this into main?

[negin513]

I would like to give a quick presentation at this CTSM SE meeting (Nov 3 2022) about how this works and what additional capabilities it has.
This includes solutions for creating mesh files for WRF-CTSM runs and can be used with any netcdf files with 1d or 2d lat and lon coordinates.

[chrystali2002]

I would like to give a quick presentation at this CTSM SE meeting (Nov 3 2022) about how this works and what additional capabilities it has. This includes solutions for creating mesh files for WRF-CTSM runs and can be used with any netcdf files with 1d or 2d lat and lon coordinates.

Looking forward to the presentation

[ekluzek]

@chrystali2002 here are the joining details for the meeting. This will be 9:30am MDT and you can join on google meet, or in the 315 conference room at the Mesa Lab.

https://meet.google.com/pkq-ghfn-gpv

all of the conferencing info. is here...

CTSM Software Engineering Team
Thursday, November 3 · 9:30 – 11:00am
Google Meet joining info
Video call link: https://meet.google.com/pkq-ghfn-gpv
Or dial: ‪(US) +1 563-538-1719‬ PIN: ‪417 402 863‬#
More phone numbers: https://tel.meet/pkq-ghfn-gpv?pin=9595653316138

[ekluzek]

This is replaced by #1892

[ekluzek]

The user-mods part of this already came in. So I'm closing this one.