Sporadic NaNs in C13_GPP and C13_NBP in https://github.com/NCAR/LMWG_dev/issues/100

[olyson]

Brief summary of bug

The land diagnostics package reports a couple of NaNs in C13_GPP and C13_NBP in the pSASU simulation for NCAR/LMWG_dev#100

General bug information

CTSM version you are using: See NCAR/LMWG_dev#100

Does this bug cause significantly incorrect results in the model's science? Maybe not significant?

Configurations affected: pSASU

Details of bug

I didn't really want to file this right before I leave on vacation for a week, but

The land diagnostics package screens for NaNs in the history fields and stops the package if any NaN is found (if BLOCK_NAN is set to 1). It searches the ANN_ALL file. For this 160 year simulation it reports:

(0) screening case 2 : C13_GPP
(0) found 2 NaNs
(0) Sample NaN Coords: time = 126 latDim = 155 lonDim 193
(0) Sample NaN Coords: time = 146 latDim = 155 lonDim 193
(0) screening case 2 : C13_HR
(0) screening case 2 : C13_NBP
(0) found 2 NaNs
(0) Sample NaN Coords: time = 126 latDim = 155 lonDim 193
(0) Sample NaN Coords: time = 146 latDim = 155 lonDim 193

So this means that there is, e.g., a NaN in C13_GPP at year 127, at j=155, i=193, and again at year 147, in this pSASU simulation. So there is at least one month in the annual mean for those two years that have a NaN at that gridcell.
I've spot checked the ANN_ALL files from a couple of our recent land-only historical simulations and don't see any NaNs. Also, I haven't run across any NaNs for these fields in any of our recent fully-coupled simulations.
Pretty sparse and transient problem.

[samsrabin]

@olyson Could you figure out the last CTSM tag with a simulation that didn't have this problem and the first tag that does?

[olyson]

@olyson Could you figure out the last CTSM tag with a simulation that didn't have this problem and the first tag that does?

Yes, but probably next week.

[wwieder]

I think LMWG_dev#111 has isotopes on for AD, SASU and pSASU, if it's helpful to see if this feature is still an issue in the latest tag? I turned isotopes off for the HIST case, but can rerun these if it's helpful too.

[olyson]

No NaNs in any of the 1,920 LMWG_dev#111 pSASU history files for variables C13_GPP and C13_NBP, so that's encouraging.
AD and SASU files can't be checked because they don't have isotope output.
I don't think I would necessarily rerun the LMWG_dev#111 historical with isotopes on. Maybe just turn them on for one of the next historicals and we can check then?

[wwieder]

OK, I'll leave isotopes on for our next LMWG_dev historical case, likely_#1115, which should start in the next day or two.