something not right with restart of timeSeriesStats(Monthly)

[xylar]

I'm running the ISOMIP+ test cases and recently turned on timeSeriesStatsMonthly. I run MPAS-Ocean repeatedly for one month, then update the forcing with a python script. Initially, I was also writing out restart files for timeSeriesStatsMonthly. It appears that xtime_startMonthly never gets updated in these circumstances (it is always the start of the simulation) and that none of the variables being written out get reset at the beginning of a month either. Yesterday, I re-ran without timeSeriesStatsMonthly restart files (which are not needed when the restart interval is a multiple of the compute interval) and xtime_startMonthly is now updated each month as expected.

I suspect, but haven't proven, that there is something bigger that is wrong with how alarms work on restart, and that this may related to E3SM-Project/E3SM#2499. My suspicion is that certain alarms are not triggered immediately on restart (or init) that would be triggered if the simulation continued without a restart. In the case of timeSeriesStatsMonthly, I think the alarm is not being triggered to zero out averaging variables and xtime_startMonthly is not being reset. It is not yet clear to me if this should happen immediately before restart files are written or immediately after they are read back in, but one of these would be the expected behavior.

I have a workaround and the same is true of E3SM as currently configured but this bug should be fixed, particularly if it indicates a bigger problem with alarms and how they interact with restart/init.