Error when using restart files

[arbennett]

When trying to run SUMMA using the initial conditions as a restart file from a previous run I get an error:

[andrbenn@n0561 06305-06320]$ /civil/hydro/arbennett/summa/bin/summa_debug -m summa_fileManager.txt -g 6305 16 -p m -r y                                                              
file_master is 'summa_fileManager.txt'.
 GRU-Parallelization run activated. 16 GRUs are selected for simulation.
start at 14:09:13
Name of Model Output control file: /civil/hydro/michaelou/summaProj/summaData/blivneh/summa_run/oc_columbia_v2.txt
forrtl: severe (408): fort: (2): Subscript #1 of the array SNOWDATA has value 6305 which is greater than the upper bound of 16

Image              PC                Routine            Line        Source             
summa_debug        0000000000A5B166  Unknown               Unknown  Unknown
summa_debug        000000000089FA33  read_icond_module          95  read_icond.f90

Stack trace terminated abnormally.

I believe (but have not yet confirmed) that this is due to using the GRU Parallelization mode (ie with the -g 06305 16 option).

[arbennett]

The issue appears to be that the hruID is not being written out correctly. With the current implementation the "correct" way to use a restart would be to specify -g 1 16 since the written HRU index starts at 1. This is not the ideal behavior.

[bartnijssen]

Problems:

• The restart file does not contain a gru dimension or gruId
• The restart file does contain a hru dimension but not a hruId
• The -g --gru Run a subset of countGRU GRUs starting from index startGRU command-line option to SUMMA does not properly parse the GRU. There are two problems:
  • The specified values are not treated as IDs, but simply used as array indices in the code. We should actually provide a lookup that maps the IDs specified on the command-line to the correct elements in the model setup
  • While the option specifies that these are GRUs, the command-line value is used as an index into an array of HRUs. The implicit assumption here is that GRU and HRU IDs are aligned and that there is only one HRU per GRU.

Solutions

• Add gru dimension and gruId to restart file
• Add hruId to restart file
• Parse gruId and hruId when reading the restart file
• Create a lookup function that maps an hruId and gruId to the correct model elements
• Use this same lookup to map the hruId (-h) and gruId (-g) specified on the command-line to the correct model elements

@arbennett: Please comment whether this makes sense

[arbennett]

I'm not sure on the third point in your problems section, but the first two are definitely problems. It's possible that when using -g startGRU nGRU it does look for the right numbered HRU (which would still be wrong) but the restart files are not indexed correctly.

[bartnijssen]

Yeah - the third one is definitely a problem as well. We should keep GRUs and HRUs cleanly separated and not use the former as an index for the latter. It'll be hopelessly confusing (as demonstrated by the discussion).

[bartnijssen]

Note - when performing restart tests for model setups with single HRUs:

• restarts are exact (bit-for-bit) in the absence of snow
• restarts are close (but not exact) when snow is present in the state file

[andywood]

Good to get these issues addressed. I've been using restart files successfully, but working around the lack of functionality from not having comprehensive ID info in the state files (ie one can either run whole domain or single GRU runs). Fortunately there's a lot of existing code in SUMMA to borrow to do this (eg mapping/lookup).

[andywood]

as of PR #430 'gru' and also 'tdh' are new dimensions in the state file. these allow for writing basin hillslope runoff variables.