High resolution regional simulations

[wwieder]

We need a clearer workflow for how users can run high resolution regional cases.

Briefly, Xiulin is wanting to run a high resolution CTSM-FATES case over regions of California with DATM inputs that are archived from a WRF simulation. @lmkueppers and @ckoven is this something you've done previously? Moving forward, encouraged use of nuopc, and thought the following would need to be done:

1. Subset surface data (from NLDAS) to the region of interest, given Some issues with creating surface datasets for the nldas2 grid... #1408, I wonder if this will be very straightforward?
2. Create a mesh file, Subset mesh files for regional cases and add user-mods for regional cases.  #1735 (the mesh file for nuopc is similar in function to the old domain file for mct, but is created differently).
3. Create a regional case that points to the surface data, mesh, and datm files. You can point to the correct DATM files using the datm.streams.xml file in a nuopc case. This may be easier to set up with a usermods_dir.

Here are the questions that may trip us up.

• Are the existing surface data and datm files at the same resolution as the NLDAS grid?
• How does CLM handle information on soil properties coming from NLDAS (presumably they aren't the same)?
• @negin513 and @adrifoster, can the ./subsetdata script be manually changed to create a mesh and surface dataset from higher resolution base maps (3 or 9 km NLDAS). If so,
• Do we have an appropriate resolution mesh file to subset for this regional case?
• @ckoven , masking out regions within the region may be a little trickier. I'd assume this is done with the surface dataset and mesh files? @ekluzek may have ideas based on how we're creating the sparse grid for nuopc.

I'm sure there are additional considerations I'm not thinking about here, but wanted to move this discussion onto github to avoid confusion

[ckoven]

@wwieder thanks!

Are the existing surface data and datm files at the same resolution as the NLDAS grid?

No, the resolution is different, and the WRF data is on a non-lat/lon grid. But we can probably make use of the NUOPC runtime interpolation?

Assuming that we also need to translate the native WRF output to DATM format (and subset it at the same time to match the boundaries of the CTSM simulation), and that the format hasn't changed at all with the NUOPC update.

cc @XiulinGao

[XiulinGao]

@wwieder Thanks Will! and Charlie is correct. WRF is on curvilinear grid. In the forcing data file, lon/lat are only index of each grid, one needs an external file to refer back to the actual grid center. Also, WRF has a very differnt formating that their time information is hiding in the file name, so each file is only for one hour and there's no actual time dimension in the data (if file name does not count). For running CLM1PT, we might still need to reprocess these huge amount of files so they are in the correct format required by CLM1PT DATM mode.

[glemieux]

@XiulinGao during a conversation with @wwieder in a separate email thread, he was wondering if you or Lara have experience with high res regional cases using WFR forcing. Having some example cases using this would be helpful.

[XiulinGao]

@glemieux Actually no. This idea about using WRF for high res regional simulation just came up during our coversation few weeks ago. The purpose of doing this is to get more pixels for running our grass only simulations in California so that we can do model benchmark with enough pixels for only grasslands regions.I do have the archived WFR forcing available on cloud if that's something also can be helpful. Polly might be the person to ask as I know she has done some high resolution regional runs but not with WRF forcing I assume.

[lmkueppers]

As Xiulin says, Polly has been running regional CLM-FATES simulations in California for an irregular domain. She is not using NUOPC. She is running at a fine resolution (4km) using custom boundary conditions.

I've asked our WRF collaborators (Stefan Rahimi and Alex Hall) if they have any other collaborators who have used their WRF output for land surface model simulations. Will let you know.

[wwieder]

@ekluzek mentioned this in an email, but it seemed worth noting here too. Actually following through with this suggestion may require some additional discussion.

The issue already has this, but subset_data should be able to subset our NLDAS2 data fine, but they need to provide WRF data for the forcing. So they'll have to convert WRF history files into datasets that look like our forcing datasets. That's likely to be a non-trivial task.

[slevis-lmwg]

1. Subset surface data (from NLDAS) to the region of interest, given Some issues with creating surface datasets for the nldas2 grid... #1408, I wonder if this will be very straightforward?

I posted a comment in #1408 about successfully generating this surface dataset using the new mksurfdata_esmf tool:
`fsurdat = '/glade/work/slevis/git/mksurfdata_toolchain/tools/mksurfdata_esmf/surfdata_0.125nldas2_hist_78pfts_CMIP6_2000_c220614.nc'

[slevis-lmwg]

Do we have an appropriate resolution mesh file to subset for this regional case?

I think this is the same as the mesh file that I see in the mksurfdata_esmf namelist (mentioned in the previous post when I generated the fsurdat):
/glade/p/cesm/cseg/inputdata/share/meshes/0.125nldas2_ESMFmesh_cd5_241220.nc

[slevis-lmwg]

[...] subset_data should be able to subset our NLDAS2 data fine

True, so generating a nldas2 surface dataset (my second-to-last post) is unnecessary because the subset_data tool can subset an existing nldas2 fsurdat file.

[XiulinGao]

According to Mariana's talk this morning at the CESM workshop, it seems that to use WRF forcing we should use CDEPS: https://escomp.github.io/CDEPS/html/index.html. And it also seems like we have to reprocess the WRF forcing into data files that can be used.

[mvertens]

@XiulinGao - I want to confirm that you are aiming to using WRF forcing in an "I" compset simulation where the input forcing comes from DATM. Is the WRF output on the same grid as CLM?

[XiulinGao]

@mvertens Yes, correct. No, the WRF output is on a curvilinear grid not the regular longitude and latitude used by CLM. So one of the questions we have is that does the NUOPC regridding is able to handle this curvelinear grid used by WRF and do the regridding at runtime or we have to reproject WRF to regular, even spaced grid ourselves offline. I also have a very specific question about what exact format the forcing should be in? I know that with the mct driver to run DATM with CLM1PT (assuming we are still using the same DATM mode for this case?) mode there's specific requirement for how one should prepare their own forcing files in certain way, but I didn't see that information when I went o checkout CDEPS. But very likely that we still have to reprocess our WRF foricng files even reprojection is nor required. Thanks!

[mvertens]

@XiulinGao -
In terms of regridding, ESMF can handle it because for curvilinear grids we have all the coordinates (for every point explicitly) - i.e., no assumption about grids being rectilinear. You don't have to reproject the WRF data.
In terms of the format, the input forcing data format for CDEPS is the same as for the mct data models. But CDEPS will handle more regridding options at run time than the mct data models - which just did bilinear with a restriction that the input forcing was on a logically rectangular grid.

[XiulinGao]

Thanks @mvertens ! That makes it more clear now. I think one of the most important things to do now is to reformat all the WRF forcing files into the data files that can be read in. One last question, by any chance that you know what will be the easist way to mask out regions based on vegetation cover? For instance, we'd like to run land only simulations withtin grid cells that have grass cover > 80% in California region. I assume we'll have to make changes to both the mesh and surface files. But how should we do that with NUOPC?

[ekluzek]

@XiulinGao we have new capability to collapse vegetation types in gridcells down to remove complexity. I think this might be part of what you are referring to.

If I understand your statement "only run grid cells where grass is > 80% of the gridcell" I think you want to only run grass PFT gridcells. So you'd mask out non-grass gridcells.

One way to do this would be to tell CTSM to run for only the dominant landunit type in the gridcell, as well as only for the dominant vegetation type in the gridcell. This would still run for non-grass grid cells (including urban and lake, and other landunits, as well as other vegetation types), but it would only run with one type so would be faster. The next step would be to mask out running for the non-grass gridcells, which you would then do by changing the mesh file.

So I would suggest start with adding this to your user_nl_clm

n_dom_landunits = 1
n_dom_pfts = 1

and run that way. Then you could figure out what gridcells you want to mask out and create a new mesh file that masks out the non-grass gridcells.

[wwieder]

This may be a good idea, @ekluzek, but since they're interested in FATES I'm not sure single PFT grid cells are desired?

[ekluzek]

n_dom_landunits should work with FATES (although we don't test it). n_dom_pfts won't help for FATES nor work correctly for FATES, but you could do a non-FATES simulation just to figure out what gridcells to mask out. We are adding something that will prevent users from trying to use n_dom_pfts with FATES. It will likely work that way, just not really do anything. But ,doing a small non-FATES simulation should help you to figure out the grid-cells you want to focus on though.

[dlawrenncar]

You'd want to use the same approach as the sparse grid, where the sparseness is just the grass grid cells (presumably they can be pre-defined). I'm not sure if the sparse grid has been replicated in NUOPC?

…

On Wed, Jun 15, 2022 at 2:58 PM Erik Kluzek ***@***.***> wrote: n_dom_landunits should work with FATES (although we don't test it). n_dom_pfts won't help for FATES nor work correctly for FATES, but you could do a non-FATES simulation just to figure out what gridcells to mask out. We are adding something that will prevent users from trying to use n_dom_pfts with FATES. It will likely work that way, just not really do anything. But ,doing a small non-FATES simulation should help you to figure out the grid-cells you want to focus on though. — Reply to this email directly, view it on GitHub <#1773 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFABYVCKYYOSUKQABICWOF3VPI7WJANCNFSM5XZISM4A> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[XiulinGao]

@ekluzek I'm running FATES with only one grass PFT, and interested in using only grassland regions for model benchmarking (so exclude forested area, croplands etc.). So yes, basically we'd want to have a mesh file where non-grass grids are masked out using a threshold value of grass cover. We could of course just run the whole CA regions and then mask out non-grassland grids using land cover data (e.g. NLCD), but since we are doing high resolution runs this would be computationally expensive. I think this might be a good way to go. But what about using land cover information to mask out non-grass grids in the mesh file directly? Is that possible? if it is possible, any change need to be made for surface data as well?

David mentioned sparse grid, which I'm not familiar with. but it seems that with this approach we can turn off selectively run in certain grids with some dort of definition?

Thank you all for the thoughts on this, the discussion has been very informative and helpful!

[slevis-lmwg]

Update of progress (working on cheyenne)

TODO This workflow should ultimately appear in the User's Guide.

@XiulinGao provided three WRF files with three grids of interest:
wrfinput_d01, wrfinput_d02, wrfinput_d03
A) I renamed the files:
wrfinput_d01.nc, wrfinput_d02.nc, wrfinput_d03.nc
...and repeated the following steps for all three files:

1. lat/lon vars in wrfinput_d01.nc are 3d, but step (2) needs lat/lon as 1d or 2d variables, so...
   ncwa -a Time wrfinput_d01.nc wrfinput_d01_2d_lat_lon.nc
2. I generated SCRIP grid files from the WRF files
   ncks --rgr infer --rgr scrip=wrfinput_d01_scrip.nc wrfinput_d01_2d_lat_lon.nc wrfinput_d01_foo.nc
   *foo.nc contains metadata and will not be used.
3. I generated mesh files from the SCRIP grid files...
   /glade/u/apps/ch/opt/esmf-netcdf/8.0.0/intel/19.0.5/bin/bing/Linux.intel.64.mpiuni.default/ESMF_Scrip2Unstruct wrfinput_d01_scrip.nc wrfinput_d01_unstruct.nc 0
4. In a copy of the mesh files, I changed the mask variable from 1 to 0 everywhere. I used matlab for this, though it's prob. easy to do with nco. This will serve as the MASK_MESH file required in one of the last steps before submitting the run.

To proceed with steps B, one needs the ctsm5.2.mksurfdata branch where the new mksurfdata_esmf is located:

git clone https://github.com/ESCOMP/CTSM.git
cd CTSM
git checkout ctsm5.2.mksurfdata
./manage_externals/checkout_externals
cd tools/mksurfdata_esmf

B) Next made surface datasets (fsurdat files) using the mksurfdata_esmf tool

1. ./gen_mksurfdata_build.sh
2. ./gen_mksurfdata_namelist.py --start-year 2000 --end-year 2000 --model-mesh-nx 109 --model-mesh-ny 104 --model-mesh /glade/work/slevis/data_UCB/ca_fates_input_data/wrfinput_to_meshes/wrfinput_d01_unstruct.nc
3. ./gen_mksurfdata_jobscript_single.py --number-of-nodes 24 --tasks-per-node 12 --namelist-file surfdata_wrfinput_d01_hist_78pfts_CMIP6_2000_c220622.namelist
4. qsub mksurfdata_jobscript_single

C) Now run the CTSM with the new fsurdat files

1. ./create_newcase --case ~/cases_FATES/test_wrfinput_d01 --compset I2000Clm50Fates --res CLM_USRDAT --run-unsupported
2. ./case.setup
3. Add this line to user_nl_clm:
   fsurdat = '/glade/work/slevis/data_UCB/ca_fates_input_data/fsurdat/wrfinput/surfdata_wrfinput_d01_hist_78pfts_CMIP6_2000_c220622.nc'
4. Update lines in env_run.xml:

./xmlchange LND_DOMAIN_MESH='/glade/work/slevis/data_UCB/ca_fates_input_data/wrfinput_to_meshes/wrfinput_d01_unstruct.nc'
./xmlchange ATM_DOMAIN_MESH='/glade/work/slevis/data_UCB/ca_fates_input_data/wrfinput_to_meshes/wrfinput_d01_unstruct.nc'
./xmlchange MASK_MESH='/glade/work/slevis/data_UCB/ca_fates_input_data/wrfinput_to_meshes/wrfinput_d01_unstruct_mask0.nc'

5. ./case.build
6. ./case.submit and the simulation test_wrfinput_d01 is still running. I did not change datm settings, so the model is reading global GSWP3 data in this test.
7. After the fact I changed env_mach_pes.xml for the d02 and d03 cases to run on more nodes and processors because they were running out of memory.

P.S. CTSM version: ctsm5.1.dev099-17-gd7bc3ba48

[XiulinGao]

@slevisconsulting Thank you Sam for the great work! Just for your information that wrfinput_d01 is in 45km resolution, so it's not a high resolution grids. But the whole workflow seems working pretty well! I'm still working on the reprocessing of the 9km res WRF forcing, which is moving quite slow. Hopefully we can put everything together to do a test run in California soon using WRF forcing. Of course, still have to figure out how to run the 'sparse grid' using some vegetation cover mask. cheers

[slevis-lmwg]

Documenting process similar to above but starting from @XiulinGao's wrf_CA grid

@XiulinGao provided monthly datm files for 1981 for testing.
@XiulinGao provided mesh file wrf_CA_unstruct.nc generated from a datm file using steps A above.

I can jump ahead to step A4. I'm documenting here only steps that differ from above:
B2) ./gen_mksurfdata_namelist.py --start-year 1981 --end-year 1981 --res wrf_CA --model-mesh /glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/wrf_CA_unstruct.nc --model-mesh-nx 147 --model-mesh-ny 151 --nocrop
B3) ./gen_mksurfdata_jobscript_single.py --number-of-nodes 24 --tasks-per-node 12 --namelist-file surfdata_wrf_CA_hist_16pfts_CMIP6_1981_c220715.namelist
C1) ./create_newcase --case ~/cases_FATES/wrf_CA_01 --compset I2000Clm50Fates --res CLM_USRDAT --run-unsupported
C3) Add this line to user_nl_clm:
fsurdat = '/glade/work/slevis/data_UCB/ca_fates_input_data/fsurdat/wrf_CA/surfdata_wrf_CA_hist_16pfts_CMIP6_1981_c220715.nc'
C4) Update lines in env_run.xml:

./xmlchange LND_DOMAIN_MESH='/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/wrf_CA_unstruct.nc'
./xmlchange ATM_DOMAIN_MESH='/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/wrf_CA_unstruct.nc'
./xmlchange MASK_MESH='/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/wrf_CA_unstruct_mask0.nc'

C7) Simulation successful (5 days). CTSM version: alpha-ctsm5.2.mksrf.06_ctsm5.1.dev090
...but update env_mach_pes.xml to run faster on more nodes/procs and env_mach_pes.xml to run on the regular queue.

D1) Update user_nl_datm_streams to include the next 3 lines three times (i.e. repeated for Precip and for TPQW):

CLMGSWP3v1.Solar:mapalgo = none
CLMGSWP3v1.Solar:meshfile = none
CLMGSWP3v1.Solar:datafiles = /glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/1981-01.nc,/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/1981-02.nc,/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/1981-03.nc,/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/1981-04.nc,/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/1981-05.nc,/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/1981-06.nc,/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/1981-07.nc,/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/1981-08.nc,/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/1981-09.nc,/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/1981-10.nc,/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/1981-11.nc,/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/1981-12.nc

D2) Update env_run.xml

./xmlchange DATM_YR_START=1981
./xmlchange DATM_YR_END=1981

Simulation successful (30 days)!

PS. @mvertens recommended changing PIO_TYPENAME to pnetcdf for faster i/o (that is, writing to history and restarts). I haven't tested, yet.

[slevis-lmwg]

Next steps from 2022/7/19 meeting with @XiulinGao @lmkueppers @ckoven @pollybuotte

In the case that I described above (wrf_CA_01) which runs on a wrf grid (147x151), I will limit the simulation to grid cells with C3 grass >= 80% according to the fsurdat pft data:

• In a copy of the wrf_CA_01 simulation's fsurdat file, I used matlab to add landmask & landmask_diff variables ( = 0 where C3 grass < 80% and = 1 where C3 grass >= 80%): /glade/work/slevis/data_UCB/ca_fates_input_data/fsurdat/wrf_CA/wrf_CA_c3_grass_nc3.nc
• I used fsurdat_modifier to update the surface dataset according to landmask_diff in wrf_CA_c3_grass_nc3.nc: /glade/work/slevis/data_UCB/ca_fates_input_data/fsurdat/wrf_CA/surfdata_wrf_CA_c3grass_16pfts_CMIP6_1981_c220715.nc
  Turns out this step was unnecessary (redundant). It would have been necessary if we were modifying fsurdat variables in the land cells of interest or if the land cells of interest were located where the model normally expects ocean.
• I used mesh_modifier to update the mesh file according to landmask in wrf_CA_c3_grass_nc3.nc:
  /glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/wrf_CA_c3grass_unstruct.nc
• In the same case (wrf_CA_01) I updated env_run.xml to point to the new mesh file and user_nl_clm (unnecessary since fsurdat didn't change). I submitted a new 1-month simulation.

Simulation completed successfully:
/glade/scratch/slevis/wrf_CA_01/run/wrf_CA_01.clm2.h0.2000-01.nc

[slevis-lmwg]

I am consolidating the instructions that I have posted over time in this issue
in this google doc.

[slevis-lmwg]

I have updated the above mentioned google doc to include a simpler set of instructions for running regional CTSM on a regular 1-degree grid. Comments always welcome.

[glemieux]

I have updated the above mentioned google doc to include a simpler set of instructions for running regional CTSM on a regular 1-degree grid. Comments always welcome.

How about transfering the google doc to a Github Discussion in the Q&A section? I.e similar to @adrifoster single point NUOPC instructions: #1833

[slevis-lmwg]

I have updated the above mentioned google doc to include a simpler set of instructions for running regional CTSM on a regular 1-degree grid. Comments always welcome.

How about transfering the google doc to a Github Discussion in the Q&A section? I.e similar to @adrifoster single point NUOPC instructions: #1833

Seems like a good idea. I'm open to that.

[slevis-lmwg]

Performance optimization is:

• discussed in the above mentioned google doc
• investigated more thoroughly in slow regional runs with nuopc vs mct #1907