Setting up a sandbox to test coupled model restarts

[DeniseWorthen]

This procedure will set up a "rundir" which contains all the input used by the RT test (in this case cpld_bmark_v16), two directories containing the control and restart namelists or configuration files and a script to copy the checkpoint restarts from a "base" run to a "restart" run. This setup allows repeated re-build of the model executable or changes to the input namelists to test various combinations of features or code changes.

The nems.configure will be modified to write the coupler history file at every pass through the fast coupling loop. These files can be compared to identify which ATM fields do not reproduce on restart. When comparing the coupler history files, it is important to remember that the terms "import" and "export" are relative to the coupler. Fields which are labeled "atmImp" are fields which the coupler is importing from the ATM; fields labeled "atmExp" are fields which the coupler is exporting to the ATM.

The ATM fields in the coupler history file are stored on the mesh. The files must be processed to write the fields on the ATM tiles. I use an NCL script (unwind_cpl_hi_v2.ncl) to do this. It is included in the test case described below.

The following example uses the cpld_bmark_v16 test from the current develop branch. A sample sandbox directory has been set up here: /scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/REST_V16.

1. Prepare the sandbox:

   Run the existing cpld_bmark_v16 and cpld_restart_bmark_v16 tests from the current ufs-weather-model model develop branch
   and keep the run directories:

   ./rt.sh -k -l rt.v16 >output 2>&1 &

   where rt.v16 contains just those two tests and the compile line.

   Make a run directory from the initial run and then clean it up.

   cp -r cpld_bmark_v16/ rundir

   in rundir remove the following files:

    PET* 
    out, err, logf*, mediator.log
    sfcf*nc, atmf*nc, history/*, MOM6_OUTPUT/* , ocn_2013_04_01_03.nc, SST_2013_04_01.nc
    RESTART/*
    rpointer.cpl, ice.restart_file

Make a control nml directory and move the following items from the rundir to the nmlctl directory:

   mkdir nmlctl 
   mv rundir/nems.configure nmlctl 
   mv rundir/input.nml nmlctl 
   mv rundir/model_configure nmlctl 
   mv rundir/ice_in nmlctl

Make a restart nml directory and move the following items from the cpld_restart_bmark_v16 directory to nmlrest directory:

   mkdir nmlrest 
   mv cpld_restart_bmark_v16/nems.configure nmlrest 
   mv cpld_restart_bmark_v16/input.nml nmlrest 
   mv cpld_restart_bmark_v16/model_configure nmlrest 
   mv cpld_restart_bmark_v16/ice_in nmlrest

2. Edit the nems.configure in both the nmlctl and nmlrest directories to write out the history file at the end of the fast coupling loop. Add a MED med_phases_history_write after the prep_ocn_accum phase:

     MED med_phases_post_ice
     MED med_phases_prep_ocn_accum
     MED med_phases_history_write

And change the write frequency of the coupler history to write at every pass:

      history_n = 1
      history_option = nsteps

3. Edit the sorc and dest directories in the cpintermedrestarts.sh script to point to your location.

4. Test a restart case

   After re-compiling the code or deciding on a change in the input.nml (eg. turning on/off a feature), a base and restart run
   can be made from the three prepared directories. For example:

    cp -r rundir/ base/ 
    cp nmlctl/* base 
    cd base 
    sbatch job_card

After the base case completes:

   cp -r rundir rest 
   cp nmlrest/* rest 
   ./cpintermediaterestarts.sh
   cd rest
   sbatch job_card

Remember that if you change an input.nml in the base directory, you must also change it in the rest directory! The same
goes for re-compiled executable. Also be careful with the pointer files (ice.restart_file and rpointer.cpl). These are over-
written at the end of each integration and must be "re-set" if you do not start with a clean restart directory. The
cpintermedrestarts.sh automatically creates the right pointer files.

If you rebuild, then make sure the rebuilt fv3.exe is copied into both base and rest. My personal practice is to keep the three
directories (rundir, nmlctl, nmlrest) unchanged and make changes only in the base and rest directories.

5. Compare the coupler history files at the first step through the fast loop. If the fields from the ATM are not B4B at this timestep, the coupled model will not reproduce. For the cpld_bmark_v16 case, the file to compare is ufs.cpld.cpl.hi.2013-04-01-11100.nc.

6. If the coupler history file is not identical at the first pass through the fast loop, then the fields which are not reproducing can be identified. To visualize the field differences, the NCL script can be run to write the mesh ATM files on the tiles.

[DeniseWorthen]

Using the above procedure, I have tested restart reproducibility for the P7 RT tests in PR #765.

For the C192mx050 test case, using tile2 as an example, the following fields imported from ATM do not reproduce on restart:

40: RMS atmImp_Faxa_lwnet                3.0398E-03            NORMALIZED  4.6263E-05
49: RMS atmImp_Faxa_rain                 1.0660E-05            NORMALIZED  2.5795E-01
121: RMS atmImp_Sa_pbot                   6.2722E-02            NORMALIZED  6.4568E-07
137: RMS atmImp_Sa_shum                   3.0399E-07            NORMALIZED  2.7052E-05
146: RMS atmImp_Sa_tbot                   2.3368E-03            NORMALIZED  7.9310E-06
169: RMS atmImp_Sa_z                      8.2664E-05            NORMALIZED  7.6896E-06

Plotting these fields on only water points shows that of these, the atmImp_Faxa_lwnet field is not reproducing on land (the final panel shows a constant field of 0), the others do not reproduce on water:

Making an identical run but setting do_ca = .false. and ca_sgs = .false. shows that on tile2, all the water points now reproduce. The above plot made in this case shows all constant fields of 0 (not shown). Instead, only the atmImp_Faxa_lwnet on land does not reproduce:

[DeniseWorthen]

A further test case w/ both ca turned off and the recent sfcsub fix (ccpp-physics PR 378) reverted gives restart reproducibility for the test. In this case, the coupler restart file at the end of the rest run is B4B with the coupler restart file from the base run.

cprnc -m ca_off_nosfcsubfix/rest/RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc ca_off_nosfcsubfix/base/RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc
SUMMARY of cprnc:
 A total number of    251 fields were compared
          of which      0 had non-zero differences
               and      0 had differences in fill patterns
               and      0 had different dimension sizes
 A total number of      0 fields could not be analyzed
 A total number of      0 time-varying fields on file 1 were not found on file 2.
 A total number of      0 time-constant fields on file 1 were not found on file 2.
 A total number of      0 time-varying fields on file 2 were not found on file 1.
 A total number of      0 time-constant fields on file 2 were not found on file 1.
  diff_test: the two files seem to be IDENTICAL

[lisa-bengtsson]

Denise, are these atmImp* fields some coupling fields between land/ocean and atmosphere? Or is it atmosphere output, e.g "rain"?

If you revert the recent sfcsub fix (ccpp-physics PR 378) and have do_ca = true, does it reproduce? I'm thinking if it is a combination with these two on it could be easier to figure out what has happened.

[lisa-bengtsson]

thank you, yes the coupled model.

[DeniseWorthen]

If you're not going to use the cpintermedrestarts.sh script to copy the needed files to the restart run, you'll need to be careful about setting up the pointer files for the mediator and ice restart files.

[lisa-bengtsson]

I may try your sandbox, what I found easiest in the past has been to run the cpld_control and cpld_restart, then turn do_ca=true in the respective namelists, run control, copy over the restart files, run restart and compare the 24 hour forecast files.

[DeniseWorthen]

You'll need to be careful in that case about setting the rpointer.cpl and ice.restart_file contents. These files get written at the end of a run and they'll point to the wrong restart file if you don't re-set them. But if you have a system that works for you, that is fine.

[lisa-bengtsson]

I see, I will test to set up your sandbox for the test. Thanks.

[DeniseWorthen]

I have run an additional test now at C192mx050. First, turning off ca leaves only the lwnet imported from ATM as not reproducing. If I then turn off frac_grid, I get full restart reproducibility.

[HelinWei-NOAA]

I have run an additional test now at C192mx050. First, turning off ca leaves only the lwnet imported from ATM as not reproducing. If I then turn off frac_grid, I get full restart reproducibility.

@DeniseWorthen Is this test with Noah MP? If so, the problem is very likely caused by the surface emissivity with fractional grid.

[DeniseWorthen]

yes, this test was with noahmp. I can't run w/ noahmp off and the other options on (nsst,ugwpv1) because I don't have a suite file. Trying lsm=1 gave me a seg fault.

[DeniseWorthen]

@HelinWei-NOAA Please see the figure from the additional test I just completed below

[lisa-bengtsson]

It seems that the SST restart field was a red herring. Looking closer at this yesterday I found that I never read in the restart fields, as I never get to the line that reads them:

fname = 'INPUT/'//trim(fn_phy)
if (file_exist(fname)) then
   !--- read the CA restart data
   call mpp_error(NOTE,'reading CA restart data from INPUT/ca_data.tile*.nc')
   call restore_state(CA_restart)
else
   call mpp_error(NOTE,'No CA restarts - cold starting CA')
   return
endif

Here fname = INPUT/ca_data.nc

I'm testing now to comment out the "if file_exist" line to see if the coupled CA restart reproduce. If it does I will try to figure out why this works in atmosphere only simulations, and find a better logic to enter this loop. Will get back to you shortly.

[SMoorthi-emc]

Yes, my run had "NOTE from PE 0: No CA restarts - cold starting CA" Moorthi

…

On Tue, Sep 14, 2021 at 9:32 AM lisa-bengtsson ***@***.***> wrote: It seems that the SST restart field was a red herring. Looking closer at this yesterday I found that I never read in the restart fields, as I never get to the line that reads them: fname = 'INPUT/'//trim(fn_phy) if (file_exist(fname)) then !--- read the CA restart data call mpp_error(NOTE,'reading CA restart data from INPUT/ca_data.tile*.nc') call restore_state(CA_restart) else call mpp_error(NOTE,'No CA restarts - cold starting CA') return endif Here fname = INPUT/ca_data.nc I'm testing now to comment out the "if file_exist" line to see if the coupled CA restart reproduce. If it does I will try to figure out why this works in atmosphere only simulations, and find a better logic to enter this loop. Will get back to you shortly. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#797 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYSUZJCKI4UJCRJEFZ3UB5FF3ANCNFSM5D552CEA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[lisa-bengtsson]

Same for me. It looks like FV3GFS_io.F90 has been recently updated. I used the logic as listed for physics and surface fields in that routine, but now they have removed restore_state, and it looks like it is using read_restart instead... so maybe it is related to this update.

Although I did confirm that this still works for atmosphere only runs... I was thinking if the files are not named ca_data.nc but rather ca_data.tileX.nc then they would not be found, so maybe there is a step in the atmosphere only runs where the tileX is stripped? Confusing.

[pjpegion]

They way I understand the logic, you need ca_data.tileX.nc AND a file ca_data.nc, the latter can be anything, but the code looks for its existence before trying to open the restart files.

For our replay experiments, we are just touching ca_data.nc so it exists.

[lisa-bengtsson]

@pjpegion do you see a ca_data.nc in your RESTART directory in your unit test setup? I can only see the ones with the tileX suffix and some with a date prefix and tileX suffix.

[pjpegion]

No, it doesn't exist in the unit tests. I removed the check for ca_data.nc for the unit test, thinking it wasn't necessary, but had to put it back when integrating to the UFS, but haven't updated my unit test scripts yet.

[lisa-bengtsson]

I don't see it it in my UFS atmos only RESTART directory either. But I don't see the phys or surface files without the tileX either, and the fname in FV3GFS_io is still e.g: character(len=32) :: fn_phy = 'phy_data.nc'.

[lisa-bengtsson]

@DeniseWorthen is there a way to search for recent PR's in github? I'm looking for the one in where the routine FV3/io/FV3GFS_io.F90 was updated.

[pjpegion]

git log FV3/io/FV3GFS_io.F90

will show you the history of that file

[SMoorthi-emc]

Why not just check for "ca_data.tile1.nc"?

…

On Tue, Sep 14, 2021 at 10:26 AM Phil Pegion ***@***.***> wrote: No, it doesn't exist in the unit tests. I removed the check for ca_data.nc for the unit test, thinking it wasn't necessary, but had to put it back when integrating to the UFS, but haven't updated my unit test scripts yet. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#797 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYVHW5P4RJKNVES2X6LUB5LP3ANCNFSM5D552CEA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[lisa-bengtsson]

Yes, I can try this, I'm a bit puzzled over the logic, as it is working in atmosphere only runs, but I don't see the ca_data.nc file there either. I will do a couple tests with atmosphere only in the exact same code base to confirm. And can also try to check for ca_data.tile1.nc in those tests.

[DeniseWorthen]

OK, maybe a stupid question, but if you specify "tile1" directly, is it reading all the other CA tiles also or just filling the ca restart fields w/ zeros on other tiles?

[lisa-bengtsson]

This is just to check that files exist in the directory, to enter the call to restore. There used to be a ca_data.nc and the tileX files, but we suspect something changed recently in FMS.

[lisa-bengtsson]

Ok, I can confirm that if I simply replace:

fname = 'INPUT/'//trim(fn_phy)

with

fname = 'INPUT/ca_data.tile1.nc'

in the routine: stochastic_physic/update_ca.F90

I get reproducible results in the coupled control restart regression test. @SMoorthi-emc do you think you can try this also in your setup?

I will test now in atmosphere only run.

[DeniseWorthen]

You said above that you switched CA on for the cpld_control and cpld_restart RT. Are you testing this w/ the P7 tests or the current develop coupled control/restart?

[lisa-bengtsson]

Yes, the current develop coupled control/restart. You can find my test directory here: /scratch2/BMC/rem/Lisa.Bengtsson/stmp2/Lisa.Bengtsson/FV3_RT/CA_COUPLED_TEST , I used your script cpintermedrestarts.sh to copy over the restart files to the INPUT directory. The sub directories cpld_control and cpld_restart are from the regression test, then I manually turned CA on. You can see the input.nml file. I was hoping someone else who has the P7 test setup could test that? It seems that Moorthi has a P7 test, but slightly different from yours with Noah MP?

[lisa-bengtsson]

If possible, perhaps you could do do_ca = true in your sandbox (frac_grid = true) and plot the figures you had yesterday?

[DeniseWorthen]

Regarding the CICE failure, it happens when 'inst_temp_height_lowest' being set to zero. CICE uses this value to calculate rhoa in the cap, but sets rhoa to 0.0 if the inst_temp_height_lowest is 0.0. Once rhoa=0, the model fails on a subsequent step where rhoa is in the denominator. I'm not going to track this down any further right now. I don't know why frac_grid=false and do_ca=true would result in the lowest air temp being zero.

[DeniseWorthen]

I've done a test of the P7 C192mx050 test with the change to stochastic_physic/update_ca.F90, do_ca=true and frac_grid=true. This is a repeat of the test w/ the figures shown above. I now see a message "reading CA restart data from INPUT/ca_data.tile*.nc" in the log file.

There are no longer any ocean points which fail to reproduce on restart. However, the lwnet exported by the ATM still fails to reproduce. For this test, the majority of these points on are tile3 (321 of 332), 3 are on tile6 and 8 are on tile2. On tile3, 3 of these changed lwnet values get mapped to the ocean where they then break restart reproducibility. No other tile exports changed lwnet values to the ocean. The runs for this setup are /scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/REST192/ca_on_frac_grid_on_stochfix

[yangfanglin]

If CA were to be applied to a nest or a single-tile model like RRFS, will fname='INPUT/ca_data.tile1.nc' still work ?

…

On Tue, Sep 14, 2021 at 11:10 AM lisa-bengtsson ***@***.***> wrote: Ok, I can confirm that if I simply replace: fname = 'INPUT/'//trim(fn_phy) with fname = 'INPUT/ca_data.tile1.nc' in the routine: stochastic_physic/update_ca.F90 I get reproducible results in the coupled control restart regression test. @SMoorthi-emc <https://github.com/SMoorthi-emc> do you think you can try this also in your setup? I will test now in atmosphere only run. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#797 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKY5N2PTWXJDMYRO3HBU2H3UB5QUHANCNFSM5D552CEA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- *Fanglin Yang, Ph.D.* *Chief, Model Physics Group* *Modeling and Data Assimilation Branch* *NOAA/NWS/NCEP Environmental Modeling Center* *https://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/ <https://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/>*

[lisa-bengtsson]

Good question. I don't fully get the logic, because I see for instance fn_phy = 'phy_data.nc' in the equivalent restore code of the physics fields, but I only see phy_data.tileX.nc in the RESTART directory.

[lisa-bengtsson]

I can also confirm that if I replace:

fname = 'INPUT/'//trim(fn_phy)

with

fname = 'INPUT/ca_data.tile1.nc'

in the routine: stochastic_physic/update_ca.F90

atmosphere only runs are also still restart reproducible with CA switched on.

[lisa-bengtsson]

I did one additional test with the same code base, atmosphere only, and using fname = 'INPUT/'//trim(fn_phy) now fails. So something has changed, perhaps in the PR with FV3GFS_io.F90 updates... another incentive to add a ca_restart test.

[SMoorthi-emc]

I made one 48 hour run and then a second restart run from 24 hours. I get "nccmp -dgqSfs /gpfs/dell2/ptmp/Shrinivas.Moorthi/CFV3/C384_phyac/gfs.20130701/00/ATM_HISTORY_0/ gfs.t00z.sfcf027.nc gfs.t00z.sfcf027.nc Variable Group Count Sum AbsSum Min Max Range Mean StdDev ulwrf_ave / 72080 109.53 150.607 -0.0213928 0.0229797 0.0443726 0.00151957 0.00309552" "nccmp -dgqSfs /gpfs/dell2/ptmp/Shrinivas.Moorthi/CFV3/C384_phyac/gfs.20130701/00/ATM_HISTORY_0/ gfs.t00z.sfcf030.nc gfs.t00z.sfcf030.nc Files "/gpfs/dell2/ptmp/Shrinivas.Moorthi/CFV3/C384_phyac/gfs.20130701/00/ATM_HISTORY_0/ gfs.t00z.sfcf030.nc" and "gfs.t00z.sfcf030.nc" are identical." "nccmp -dgqSfs /gpfs/dell2/ptmp/Shrinivas.Moorthi/CFV3/C384_phyac/gfs.20130701/00/ATM_HISTORY_0/ gfs.t00z.sfcf048.nc gfs.t00z.sfcf048.nc Files "/gpfs/dell2/ptmp/Shrinivas.Moorthi/CFV3/C384_phyac/gfs.20130701/00/ATM_HISTORY_0/ gfs.t00z.sfcf048.nc" and "gfs.t00z.sfcf048.nc" are identical." This is akin to what I used to get without CA. note that I am using NoahMP and frac-grid=.true. Moorthi

…

On Tue, Sep 14, 2021 at 12:39 PM Denise Worthen ***@***.***> wrote: yes, this test was with noahmp. I can't run w/ noahmp off and the other options on (nsst,ugwpv1) because I don't have a suite file. Trying lsm=1 gave me a seg fault. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#797 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYQ4ZD7KWMBNDS5DDNLUB53ELANCNFSM5D552CEA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[HelinWei-NOAA]

@SMoorthi-emc Can you have another test with RUC lsm? RUC LSM has the same way to deal with surface albedo/emissivity as Noah MP. If we get the same result, then we can conclude that this problem is caused by how we composite surface emissivity with fractional grid for Noah MP/RUC LSMs.

[SMoorthi-emc]

When I tried it, it was a bit tricky. I used "ca_data.tile1.nc" only in checking for the existence, but left the name of the file generic. Otherwise, it could not find all tiles.

…

On Tue, Sep 14, 2021 at 1:40 PM Denise Worthen ***@***.***> wrote: OK, maybe a stupid question, but if you specify "tile1" directly, is it reading all the other CA tiles also or just filling the ca restart fields w/ zeros on other tiles? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#797 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYTUNWGQFABBSW7P7YLUB6CHNANCNFSM5D552CEA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[lisa-bengtsson]

That is correct, only the line I propose above should change.

[SMoorthi-emc]

One question I have is why this restart file can't be read in FV3_GFS_io.F90 instead of under stochastic physics? On Tue, Sep 14, 2021 at 1:44 PM Shrinivas Moorthi - NOAA Federal < ***@***.***> wrote:

…

When I tried it, it was a bit tricky. I used "ca_data.tile1.nc" only in checking for the existence, but left the name of the file generic. Otherwise, it could not find all tiles. On Tue, Sep 14, 2021 at 1:40 PM Denise Worthen ***@***.***> wrote: > OK, maybe a stupid question, but if you specify "tile1" directly, is it > reading all the other CA tiles also or just filling the ca restart fields > w/ zeros on other tiles? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#797 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ALLVRYTUNWGQFABBSW7P7YLUB6CHNANCNFSM5D552CEA> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > > -- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[lisa-bengtsson]

The CA is running on a different resolution than the rest of the model, so I had to create a separate domain specification for io. For this I needed information at the level of stochastic physics call.

[lisa-bengtsson]

I am not 100% satisfied with the hard-coded change, mainly because I don't know exactly what was changed (perhaps FMS?). But since it works, I propose to include it in the PR that addresses the MPI decomposition and GNU debug issue so that we can close all the CA infrastructure issues for P7 in one PR.

Me and Phil are planning on creating a second PR at a later time where we make some additional changes to the scheme (infrastructure only, will not impact performance), and I can take a closer look at this particular update for that future PR to see if it can be done better.

Please let me know if anyone has any comments/concerns regarding this proposed strategy.

[DeniseWorthen]

Let me test w/ the C384 and C96 cases. I expect in the C96 case, we'll get restart repro because no changed lwnet values will get mapped to ocean. I expect the C384 case will give the same result as C192. We'll see!

[DeniseWorthen]

The results are as expected. The C96mx100 case does not map any changed lwnet to ocean and it reproduces. The C384 has 16 points, all in the Arctic (tile3) where changed lwnet values get mapped to the ocean. The C384mx025 case does not reproduce.

[HelinWei-NOAA]

Are all of 16 points fractional grids? So the fractional grid (have both land and water) in the atm will be mapped to the ocean points in the ocean model?

The results are as expected. The C96mx100 case does not map any changed lwnet to ocean and it reproduces. The C384 has 16 points, all in the Arctic (tile3) where changed lwnet values get mapped to the ocean. The C384mx025 case does not reproduce.

[DeniseWorthen]

The ocean does not have a fractional grid. The values of the ATM fields with fractional water points on the ATM side are being conservatively mapped to the ocean. Which points exactly get mapped from ATM to OCN depends on the overlap between the FV3 grid cells and the ocean grid cell. There is not a 1:1 correspondence between a fractional grid point on the ATM and a corresponding ocean grid point. I can see whether non-reproducing ATM lwnet points have land_frac values less than 1.0 (I think they must) but I can't tell you exactly which of these values together get mapped to the OCN and produce a non-reproducing lwnet for the ocean.

From the figure above, it seems clear that the non-reproducing lwnet are associated primarily with lake ice?

[SMoorthi-emc]

Helin, I don’t see the issue Denise sees. I have never run RUC lsm. Moorthi

…

Sent from my iPhone

On Sep 14, 2021, at 4:52 PM, HelinWei-NOAA ***@***.***> wrote: @SMoorthi-emc Can you have another test with RUC lsm? RUC LSM has the same way to deal with surface albedo/emissivity as Noah MP. If we get the same result, then we can conclude that this problem is caused by how we composite surface emissivity with fractional grid for Noah MP/RUC LSMs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[climbfuji]

If you want to try RUC, use the following settings in input.nml: lsoil = 4 lsoil_lsm = 9 kice = 9 lsm = 3 and in the SDF, replace <scheme>lsm_noah</scheme> <scheme>sfc_sice</scheme> with <scheme>lsm_ruc</scheme>

…

On Sep 14, 2021, at 3:02 PM, SMoorthi-emc ***@***.***> wrote: Helin, I don’t see the issue Denise sees. I have never run RUC lsm. Moorthi Sent from my iPhone > On Sep 14, 2021, at 4:52 PM, HelinWei-NOAA ***@***.***> wrote: > > @SMoorthi-emc Can you have another test with RUC lsm? RUC LSM has the same way to deal with surface albedo/emissivity as Noah MP. If we get the same result, then we can conclude that this problem is caused by how we composite surface emissivity with fractional grid for Noah MP/RUC LSMs. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or unsubscribe. > Triage notifications on the go with GitHub Mobile for iOS or Android. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#797 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RJCFV4JQSOPVVNYMGLUB6Z4ZANCNFSM5D552CEA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[DeniseWorthen]

@climbfuji I'm willing to give this a try, but I'm not sure I see the right thing to do in the suite file. For the FV3_GFS_v16_coupled_nsstNoahmpUGWPv1 we're using, I see:

    <!-- Surface iteration loop -->
    <subcycle loop="2">
      <scheme>sfc_diff</scheme>
      <scheme>GFS_surface_loop_control_part1</scheme>
      <scheme>sfc_nst_pre</scheme>
      <scheme>sfc_nst</scheme>
      <scheme>sfc_nst_post</scheme>
      <scheme>noahmpdrv</scheme>
      <scheme>sfc_cice</scheme>
      <scheme>sfc_sice</scheme>
      <scheme>GFS_surface_loop_control_part2</scheme>
    </subcycle>

I don't see any lsm_noah, only noahmpdrv. Am I looking in the right place?

[climbfuji]

You are. You will need to replace

<scheme>noahmpdrv</scheme> <scheme>sfc_sice</scheme>

with the RUC scheme, but I am not sure about the coupled ice code sfc_cice, though. @tanyasmirnova any ideas? We may need to look at the sfc_cice code and see what it does to prep the ice arrays. Maybe it needs to go before the call to RUC, because RUC contains the replacement for sfc_sice.

…

On Sep 14, 2021, at 3:54 PM, Denise Worthen ***@***.***> wrote: @climbfuji <https://github.com/climbfuji> I'm willing to give this a try, but I'm not sure I see the right thing to do in the suite file. For the FV3_GFS_v16_coupled_nsstNoahmpUGWPv1 we're using, I see: <!-- Surface iteration loop --> <subcycle loop="2"> <scheme>sfc_diff</scheme> <scheme>GFS_surface_loop_control_part1</scheme> <scheme>sfc_nst_pre</scheme> <scheme>sfc_nst</scheme> <scheme>sfc_nst_post</scheme> <scheme>noahmpdrv</scheme> <scheme>sfc_cice</scheme> <scheme>sfc_sice</scheme> <scheme>GFS_surface_loop_control_part2</scheme> </subcycle> I don't see any lsm_noah, only noahmpdrv. Am I looking in the right place? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#797 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RMTMQFDQP6H3BCCRK3UB7AC5ANCNFSM5D552CEA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[tanyasmirnova]

Should it be one or another? I mean if the model is coupled to sea ice (sfc_cice.f) then sfc_sice.f is not needed. At least with RUC LSM, this is the case - nothing is done when the model is coupled to a sea ice model.

[DeniseWorthen]

I was able to create a C192mx050 test using RUC instead of NoahMP. I used in the SDF:

      <scheme>sfc_nst_post</scheme>
      <scheme>lsm_ruc</scheme>
      <scheme>sfc_cice</scheme>

and Dom's namelist changes.

It at least runs and the tmpsfc difference at between the NoahMP and the RUC at restart is pretty small (mean of ~2.e-5) so nothing seems to have gone off the rails. The lwnet for the RUC case also does not reproduce on restart, but on many fewer points and none get mapped to the ocean. The RUC version doesn't reproduce because other fields (e.g, Tbot) do not reproduce. This is the same figure I made above for the NoahMP case, the difference on restart for the lwnet imported from ATM.

[SMoorthi-emc]

Just to clarify, "sfc_cice" is used only in the coupled model. It is used to take care of miscellaneous fluxes over sea-ice. In the coupled model "sfc_sice" only operates on lake ice. If RUC LSM takes care of lake ice, I assume "sfc_sice" may not be needed. However, you will need "sfc_cice". Moorthi

…

On Tue, Sep 14, 2021 at 6:05 PM tanyasmirnova ***@***.***> wrote: Should it be one or another? I mean if the model is coupled to sea ice (sfc_cice.f) then sfc_sice.f is not needed. At least with RUC LSM, this is the case - nothing is done when the model is coupled to a sea ice model. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#797 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYSFJB5YCZRBJJXVPPDUB7BKDANCNFSM5D552CEA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[SMoorthi-emc]

In my restart tests, at hour 3 from restart one field had different values. This field is "ulwrf_ave". Attached is the difference between the two fields. You can see that the differences are small and over sea-ice! I wonder what is causing this. For the remaining hours of the forecast all are identical. Moorthi

…

On Tue, Sep 14, 2021 at 7:32 PM HelinWei-NOAA ***@***.***> wrote: Are all of 16 points fractional grids? So the fractional grid (have both land and water) in the atm will be mapped to the ocean points in the ocean model? The results are as expected. The C96mx100 case does not map any changed lwnet to ocean and it reproduces. The C384 has 16 points, all in the Arctic (tile3) where changed lwnet values get mapped to the ocean. The C384mx025 case does not reproduce. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#797 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYS446MACPDQYWCTL2LUB7LR5ANCNFSM5D552CEA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[DeniseWorthen]

There must be some basic difference between your test and what I am testing in the C384mx025 P7 test but I have not yet identified what it is. For the C192mx050, this is sfcf018.tile3.nc difference between the base and restart. There are differences over sea ice itself, but the largest differences appear to be associated with the coast (fractional grid) points. This is 1e2*(ulwrf_ave difference).

[SMoorthi-emc]

[junwang-noaa]

For restart tests,the cpld_restart_bmark_v16 from the develop branch with the new ICs and fixed files reproduces cpld_bmark_v16. So this confirms that:

1. ICs and fixed tile files do not cause restart issues
2. Hardcoded physics updates do not cause restart issues. The updates include:
   Z-bot calculation updates,
   Open-water normalization for the ATM->OCN fluxes
   Compositing method over fractional grid
   Convection and PBL updates
   Memory leak fix in stochastic physics
   Land mask bug fix

[DeniseWorthen]

I made a fix to the CICE cap which allows a run w/ frac_grid off and CA on. This case reproduces. It seems clear to me that the issue for the P7 RTs is the compositing of lwnet and/or ulwrf on fractional grid points. The run directory is /scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/REST192/ca_on_frac_grid_off_stochfix

[DeniseWorthen]

With the stochfix for the restart file name and changing the GFS_surface_composites.F90

if (iemsflg == 2 .and. .not.flag_init ) then
to
if (iemsflg == 2 .and. (.not.flag_init .or. flag_restart)) then

I am able to get restart reproducibility for the C192mx050 sandbox.

[lisa-bengtsson]

Nice work Denise!

[DeniseWorthen]

That was @HelinWei-NOAA 's fix for the GFS_surface_composites.F90.

[SMoorthi-emc]

Why does the CICE cap depend on fractional grid and CA? I do not understand.

…

On Wed, Sep 15, 2021 at 8:25 AM Denise Worthen ***@***.***> wrote: I made a fix to the CICE cap which allows a run w/ frac_grid off and CA on. This case reproduces. It seems clear to me that the issue for the P7 RTs is the compositing of lwnet and/or ulwrf on fractional grid points. The run directory is /scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/REST192/ca_on_frac_grid_off_stochfix — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#797 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYU2CDBFAC5YUZZDLDTUCCGCDANCNFSM5D552CEA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[DeniseWorthen]

see here

[DeniseWorthen]

I now suspect I know the reason for the CICE failure. When I switched to frac_grid=false, I forgot to also change the coupling_mode for CMEPS. Since CMEPS was not doing "nstod" mapping (nearest-to-destination), a 0.0 air temperature was sent to CICE. Fixing the CICE rhoa to have a background value probably resolved that.

What is interesting is that w/ the rhoa-fix (which will be committed in today's CICE PR), it seems that the nstod mapping is no longer required when running frac_grid=false. More testing would need to be done to confirm.

[DeniseWorthen]

I've tested the cpld_control P7 for C96mx100, C192mx050 and C384mx025 in the RT and all restart tests pass. This is using the CA file name fix and the GFS_surface_composit fix. The remaining tests which do not pass are the 2-threads and the decomp.

[junwang-noaa]

Great!