Extend CLM6 PPE simulations to 2100 using anomaly forcing

[olyson]

From @linniahawkins : I would like to extend our CLM6 PPE simulations through 2100, similarly to what Daniel did for the CLM5 PPE.

[olyson]

CTSM branch info:
/glade/u/home/linnia/ctsm6_ppe/gen_ensembles/jobscripts/startup.sh

setup info for my historical sparse grid PPE cases:
/glade/u/home/linnia/ctsm6_ppe/gen_ensembles/jobscripts/setup_basecases.sh

restart:
/glade/campaign/cgd/tss/projects/PPE/ctsm6_lhc/rest/BNF_v2.n01_ctsm5.3.012_transient_lhc0000.clm2.r.2024-01-01-00000.nc

Paramfile:
/glade/work/linnia/CLM6-PPE/ctsm6_lhc/paramfiles/lhc0000.nc

[olyson]

• the cases should be two degrees
• the baseline climate should be CRU-JRA 2005-2014
• we had saved the anomalies here: /glade/campaign/cgd/tss/projects/PPE/forcing
  • you'll be able to find fullgrid 1deg anomalies in those directories
  • the "SUBSET" versions have a coastline bug (were likely not regridded)

[olyson]

I have an SSP370 case setup. It is with ctsm5.3.055, which has the anomaly forcing infrastructure we want.
This tag is more advanced from the tag LInnia used for the historical, ctsm5.3.012.
Looking through the tags between these two tags, I see that most of them are either fates, doc, or b4b-dev tags. However, a few of them implement new options in lnd_in.
One way to approach this might be to take LInnia's tag and just cherry pick the two anomaly forcing tags into that. However, those tags seem to involve some changes to externals and so I'm not sure if that will work well.
The other approach would be to adjust ctsm5.3.055 to behave like ctsm5.3.012. A list of those changes that I've identified so far:

1. Add mimics_fi to Linnia's /glade/work/linnia/CLM6-PPE/ctsm6_lhc/paramfiles/lhc0000.nc (not answer changing but the model fails when trying to read in that parameter).

2. Set snow_thermal_cond_lake_method = 'Sturm1997'. Then everything is using Sturm1997 which is what Linnea is using.

3. Set nfix_method = 'Houlton'

4. Set irrigate = .false. (although I'm not sure why it is showing up as .false. in Linnea's historical, it shows up as .true. in my SSP, need to look into this)

5. The zeroth time step is eliminated in ctsm5.3.019. That would be hard to back out but the impact is small.

6. There is an urban building energy model bug fix in ctsm5.3.024, but I don't think we have to worry about that since it only has minor effects on urban landunits.

A conservative approach would be to repeat at least part of LInnia's historical with this setup to make sure we get very similar answers (likely not bfb).

[olyson]

As pointed out by @linniahawkins , in 3. above, nfix_method should be "Bytnerowicz".
Also, snow_thermal_cond_lake should be set to "Jordan1991", as this was a bug that was fixed later after the CLM6 PPE historical was run.
Also, in 1. above, had to add leafcn_co2_base and leafcn_co2_slope to the parameter file.
Should irrigate be set to .false., as it appears to have been set in the historical? It is .true. by default in the SSP setup.

[linniahawkins]

Thanks Keith, yes I think irrigate should be set to false to match our historical simulations.

[olyson]

I've run two SSP370s with anomaly forcings, one with ctsm5.3.055 and one with BNF_v2.n01_ctsm5.3.012. The case directories are:

/glade/work/oleson/ctsm5.3.055/cime/scripts/Clm60Bgc_ctsm53055_2deg_CRUJRA2024_Control_SSP370
/glade/work/oleson/BNF_v2.n01_ctsm5.3.012/cime/scripts/Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_Control_SSP370

They are identical in atmospheric forcing and answers are not bfb but have very small differences.
They are compared using the standard diagnostics here:
https://webext.cgd.ucar.edu/IRCP/Clm60Bgc_ctsm53055_2deg_CRUJRA2024_Control_SSP370/lnd/Clm60Bgc_ctsm53055_2deg_CRUJRA2024_Control_SSP370_2081_2100-Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_Control_SSP370_2081_2100/setsIndex.html

I've been looking through the timeseries files I created, e.g., for Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_Control_SSP370 here:

/glade/derecho/scratch/oleson/timeseries/Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_Control_SSP370/lnd/proc/tseries/month_1/

There seems to be at least a couple of gridcells where the anomaly forcing doesn't seem to have been applied, e.g, as seen here for the TBOT timeseries for a single gridcell:

The repeating CRUJRA forcing is evident but I don't see any upward trend.

[linniahawkins]

Thanks Keith, I'll poke around in those directories. Two questions:

1. Are these running with the datm subset to the sparse grid meshfile?
2. Are the SSP anomalies regridded? or do we just use the nearest neighbor?

[olyson]

1. The CRUJRA datm data is subset to the sparse grid and mapalgo = 'nn'.
2. The anomaly forcing is still at 1deg and has not been regridded or subset. My impression from @djk2120 was that the subsetted version that was used for CLM5 was causing a problem for some coastal grid cells, so I've not subset the anomaly forcing for now. My notes also indicated that there wasn't much computational improvement in subsetting either: "I thought maybe we could get better computing performance by subsetting the anomaly forcing as well but the improvement is marginal (461 years/day and 5.64 pe-hrs/yr compared to 458 years/day and 5.66 pe-hrs/yr)." By the way, the BNF version of the code I ran for this SSP simulation got 465 years/day and 6.61 pe-hrs/yr.
   I'm using the default setting for the anomaly forcing, mapalgo = 'bilinear'. If we were using subsetted data then we would need to use 'nn'.
   I'm not sure what is causing the problem I noted above, maybe the anomaly forcing needs to be regridded regardless.

[olyson]

I tried 'nn' instead of 'bilinear' for the anomaly forcing mapalgo and got similar results for TBOT for the gridcell plotted above, timeseries file here:

/glade/derecho/scratch/oleson/timeseries/Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_IrrigF-AFnn_SSP370/lnd/proc/tseries/month_1/Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_IrrigF-AFnn_SSP370.clm2.h0.TBOT.202401-210012.nc

I regridded the 1deg anomaly forcing to 2deg, ran a new simulation, and that seemed to fix that gridcell and the other gridcell that was problematic, e.g.,:

(Also turned off irrigation). Case directory is here:

/glade/work/oleson/BNF_v2.n01_ctsm5.3.012/cime/scripts/Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_IrrigF-AFrg_SSP370

Timeseries files are here:

/glade/derecho/scratch/oleson/timeseries/Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_IrrigF-AFnn_SSP370/lnd/proc/tseries/month_1

Regridding of the anomaly forcing was done with:

Regrid the 1deg anomaly forcing data to 2deg.
First, generate a mapping file:
/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/oneapi/2023.2.1/7haa/bin/ESMF_RegridWeightGen --method conserve --source /glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/mappingdata/grids/0.9x1.25_c110307.nc --destination /glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/mappingdata/grids/1.9x2.5_c110308.nc --weight map_fv0.9x1.25_TO_1.9x2.5_conserve.250624.nc
Result is:
/glade/derecho/scratch/oleson/ANALYSIS/map_fv0.9x1.25_TO_1.9x2.5_conserve.250624.nc
Then regrid:
ncremap -t 1 -P clm --sgs_frc=landfrac --sgs_msk=landmask -m map_fv0.9x1.25_TO_1.9x2.5_conserve.250624.nc /glade/campaign/cesm/cesmdata/inputdata/atm/datm7/anomaly_forcing/CMIP6-SSP3-7.0/af.allvars.CESM.SSP3-7.0.2015-2100_c20220628.nc af.allvars.CESM.SSP3-7.0.2015-2100.1.9x2.5_c20250624.nc
Result is:
/glade/derecho/scratch/oleson/ANALYSIS/af.allvars.CESM.SSP3-7.0.2015-2100.1.9x2.5_c20250624.nc

[linniahawkins]

Great news! Thanks for sorting this out @olyson

[olyson]

Diagnostics for this new simulation (Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_IrrigF-AFrg_SSP370) compared to the original simulation (Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_Control_SSP370) are here.
This looks as expected to me.
But I'm happy to entertain other methods of making sure this new simulation is a good baseline for the SSP370 PPE extension.

Also, the script I used to create and run this case is here:

/glade/work/oleson/PPE/setup_run_PPE_BGC_SSP_BNF_v2.n01_ctsm5.3.012.csh

[linniahawkins]

Looks good to me, thank you Keith! I'll use this script to set up our basecases and make sure I can reproduce this.

[olyson]

Updating this discusion per my meeting with @linniahawkins today:

I migrated my regridded anomaly forcing approach to a more modern code base, ctsm5.3.055.
I ran the SSP370 simulation for this code base. Script, case directory, and output are here:

/glade/work/oleson/PPE/setup_run_PPE_BGC_SSP_ctsm5.3.055.csh
/glade/work/oleson/ctsm5.3.055/cime/scripts/Clm60Bgc_ctsm53055_2deg_CRUJRA2024_IrrigF-AFrg_SSP370
/glade/derecho/scratch/oleson/Clm60Bgc_ctsm53055_2deg_CRUJRA2024_IrrigF-AFrg_SSP370/run

Timeseries output is here:

/glade/derecho/scratch/oleson/timeseries/Clm60Bgc_ctsm53055_2deg_CRUJRA2024_IrrigF-AFrg_SSP370

I spot checked TBOT timeseries for the previously problematic gridcells I encountered in the previous 1deg full grid anomaly forcing simulation, and it looks like anomaly forcing is being applied corrrectly.

@linniahawkins and I also looked at the atm log file and verified that the other streams, e.g., ndep, aerosol, co2, were being read in correctly (the correct years).

@linniahawkins and I also had a brief discussion about when and how to transition from historical (CRUJRA) to SSP370 (anomaly forcing). Since @linniahawkins wants to compare to observations, I recommended running the historical through 2023 (CRUJRA forcing only, no anomaly forcing) and then continuing on with SSP370 anomaly forcing beginning in 2024. As @linniahawkins pointed out, there is a potential discontinuity at 2023-2024, but not sure how to avoid that. Maybe others have better ideas, @djk2120 , @slevis-lmwg , @wwieder , @katiedagon

Another thing I was thinking of is that the CRUJRA forcing for the SSP370 simulation is being looped over for 2005-2014 which I think is the baseline for the anomaly forcing. I assume this should remain at 2005-2014.

And another important thing that I just realized is that the anomaly forcing itself that comes out of the box and that is being used here is based on CMIP6 historical and SSP370 simulations. Whereas the simulations I set up in 2022 that I think were adapted for use in Daniel's paper were from the LE2 simulations. Those anomaly forcing files would need to be manipulated to be compatible with the current code as they won't work as currently constructed. Which set of forcing files to use is a science question that the PPE team can weigh in on @linniahawkins , @djk2120 , @katiedagon , @dlawrenncar , @wwieder

[olyson]

I ran diagnostics on this new simulation, comparing it to the BNF tag simulation I had done previously (Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_IrrigF-AFrg_SSP370). The forcings look identical as expected. Differences then can be attributed to differences between the two tags. See:

https://webext.cgd.ucar.edu/IRCP/Clm60Bgc_ctsm53055_2deg_CRUJRA2024_IrrigF-AFrg_SSP370/lnd/Clm60Bgc_ctsm53055_2deg_CRUJRA2024_IrrigF-AFrg_SSP370_2081_2100-Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_IrrigF-AFrg_SSP370_2081_2100/setsIndex.html

[dlawrenncar]

Where do 'we' think the discontinuity will come from? Is it land use or climate forcing or ??? And, I'm not sure what the base period should be for the CRUJRA looping. It was 2005-2014 previously because that was the end of the record. I think it might make sense to move to it to be 2014-2023 since that is the new end of the record, though maybe the argument should be that you use the same period as whatever period one used to generate the baseline for the anomaly SSP forcing. Since the anomaly forcing still comes from CESM2 simulations, which ended in 2014, then probably the anomaly forcing is still relative to the the 2005-2014 period and it makes sense to stick with that period. Happy to talk this through if helpful.

…

On Fri, Sep 19, 2025 at 10:23 AM Keith Oleson ***@***.***> wrote: I ran diagnostics on this new simulation, comparing it to the BNF tag simulation I had done previously (Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_IrrigF-AFrg_SSP370). The forcings look identical as expected. Differences then can be attributed to differences between the two tags. See: https://webext.cgd.ucar.edu/IRCP/Clm60Bgc_ctsm53055_2deg_CRUJRA2024_IrrigF-AFrg_SSP370/lnd/Clm60Bgc_ctsm53055_2deg_CRUJRA2024_IrrigF-AFrg_SSP370_2081_2100-Clm60Bgc_BNFv2n01ctsm53012_2deg_CRUJRA2024_IrrigF-AFrg_SSP370_2081_2100/setsIndex.html — Reply to this email directly, view it on GitHub <#3274 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFABYVASSBY4KYYE7RYWC7T3TQUXBAVCNFSM6AAAAAB76I2UMCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTINBVHEYDKNI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[linniahawkins]

Thanks for setting these runs up @olyson and for summarizing here. I was able to replicate your simulation in our PPE ecosystem and confirm that anomalies are being applied to all gridcells now.

I agree with Dave that we should stick with the 2005-2014 period as the baseline since that is what was used to generate the anomalies. However, I think we need to align the anomalies in 2024 with the baseline year 2014. As currently set up, 2024 is aligned with 2005. I've updated this our PPE simulations.

The discontinuity I was concerned about is related to landuse. I was unclear on whether to switch from hist to SSP in 2015 or in 2024. I tested both for comparison:

The hist landuse file removes less carbon than the SSP from 2015 to 2024. To avoid a discontinuity in 2024 I believe I should switch to SSP in 2015. (?) is this correct and are there other variables I should be looking at?

landuse files:
hist: surfdata_esmf/ctsm5.3.0/landuse.timeseries_1.9x2.5_hist_1850-2023_16pfts_c240926.nc
ssp: surfdata_esmf/ctsm5.3.0/landuse.timeseries_1.9x2.5_SSP3-7.0_2015-2100_16pfts_c250612.nc

[slevis-lmwg]

@linniahawkins as far as I know, you are correct. To remove the discontinuity, you would switch to SSP in 2015 because that's the "harmonization" year that @lawrencepj1 used.

[slevis-lmwg]

Also, I remember encountering the zero _CFLUX in Adrianna's TRENDY simulations last year and confirming that branch/restart simulations did not exhibit that behavior (because they continue the simulation as if it had not stopped). Startup/hybrid cases exhibit the behavior because they do not read that _CFLUX variable from the initial conditions and instead initialize it at zero.

[linniahawkins]

Okay thanks Sam, Then I don't see a way around this because I can't use branch when switching landuse.timeseries (2015) nor when switching to anomaly forcing (2024). Is that correct or am I missing something?

[slevis-lmwg]

I haven't played with that transition much myself recently to say it confidently, but I think you're correct. A question I have though is how you were able to use "branch" in your ctsm5.3.012 simulations. In that case did you keep the same landuse file and did you not switch to anomaly forcing?

[linniahawkins]

Thanks Sam, yes that's correct, I kept the same landuse file through 2023 so I was able to use "branch".

[dlawrenncar]

It just occurred to me that a more complicated solution would be to maintain the meteorology forcing out to 2022, but switch the land use and other forcing at 2015.

…

On Mon, Sep 22, 2025 at 3:14 PM Samuel Levis ***@***.***> wrote: @linniahawkins <https://github.com/linniahawkins> as far as I know, you are correct. To remove the discontinuity, you would switch to SSP in 2015 because that's the "harmonization" year that @lawrencepj1 <https://github.com/lawrencepj1> used. — Reply to this email directly, view it on GitHub <#3274 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFABYVBKGYEF6W2IIF5NPCL3UBREXAVCNFSM6AAAAAB76I2UMCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTINBYGA4DMNA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[linniahawkins]

Yes, I am continuing the CRUJRA met forcing through 2023, but switching to SSP fsurdat and landuse in 2015. I don't switch to anomaly met forcing until 2024.

All of the other forcings (CO2, NDEP, aerosols) are from CMIP simulations that span the entire period (e.g., fndep_clm_SSP370_b.e21.BWSSP370cmip6.f09_g17.CMIP6-SSP3-7.0-WACCM.002_1849-2101_monthly_0.9x1.25_c211216.nc )

[linniahawkins]

I've run into a new issue with the ndep stream.

I set the ndep stream in user_nl_datm_streams as:

presndep.hist:datafiles = /glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/ndepdata/fndep_clm_SSP370_b.e21.BWSSP370cmip6.f09_g17.CMIP6-SSP3-7.0-WACCM.002_1849-2101_monthly_0.9x1.25_c211216.nc
presndep.hist:year_first=1850
presndep.hist:year_align=1850
presndep.hist:year_last = 2024

However, it appears the case is not using this file. In lnd_in I see:

 model_year_align_ndep = 1850
 ndep_taxmode = 'cycle'
 ndep_varlist = 'NDEP_month'
 ndepmapalgo = 'bilinear'
 stream_fldfilename_ndep = '/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/ndepdata/fndep_clm_hist_b.e21.BWHIST.f09_g17.CMIP6-historical-WACCM.ensmean_1849-2015_monthly_0.9x1.25_c180926.nc'
 stream_meshfile_ndep = '/glade/campaign/cesm/cesmdata/inputdata/share/meshes/fv0.9x1.25_141008_polemod_ESMFmesh.nc'
 stream_year_first_ndep = 1850
 stream_year_last_ndep = 2015

Am I setting this incorrectly? @adrifoster Did you have a similar issue with TRENDY?

Here is an example case for reference:

/glade/work/linnia/ctsm5.3.065/cime/scripts/transient/ctsm5.3.065_transient/ctsm5.3.065_transient_lhc0003/

/glade/derecho/scratch/linnia/spinup_tests_ctsm53065_lhc/ctsm5.3.065_transient_lhc0003/

[wwieder]

This always confuses me too, Linnia.

The datm streams are being set in your user_nl_datm_stream, but then there's a seperate ndep file on the clm namelist defaults that gets used in lnd_in. Do we actually need to make changes in user_nl_clm for this to be handled correctly, @slevis-lmwg and @ekluzek? Can we remove what-ever's not needed (related to #1900)

[wwieder]

oh bummer, is that evident on the history files, or just from the logs? Reading your note again it seems like the history output... groan.

[ekluzek]

Yes, this is very confusing for everyone. For usability longer term we want to move so that ndep ONLY comes from the atmosphere. However, doing that switch will take some software work, and a bunch of evaluation to make sure answers are correct scientifically. This would be good to do by the cesm3.0 release -- but we likely don't have the time to make it happen.

#1900 is one of the issues that discusses this.

[ekluzek]

What might be good do though -- is to at least have this problem documented in a few different places as a warning to the user.

And thinking about it -- we should probably require that NDEP from DATM be turned off for CTSM I cases. That way if the user trys to change it in DATM -- it'll die with a warning.

What do people think about that idea? @linniahawkins @wwieder what do you think about that idea?

[linniahawkins]

oh bummer, is that evident on the history files, or just from the logs? Reading your note again it seems like the history output... groan.

Yes - from the history output it looks like NDEP cycles back to 1850 in 2015 (which is when the lnd-in ndep file ends):

[linniahawkins]

What might be good do though -- is to at least have this problem documented in a few different places as a warning to the user.

And thinking about it -- we should probably require that NDEP from DATM be turned off for CTSM I cases. That way if the user trys to change it in DATM -- it'll die with a warning.

What do people think about that idea? @linniahawkins @wwieder what do you think about that idea?

I like this idea - it would have helped to prevent my mistake in this case.