Adding a checkpoint restart capability

[JessicaMeixner-NOAA]

A requirement for operations, is going to be the ability to restart WW3 from a checkpoint. This means selecting the correct restart file and then continuing to write output files from that point.

A proposed solution is to add an option to not only label the restart files with dates such as
restart.YYYYMMDDHHMMSS.ww3 (or some form of YYYYMMDDHH) and to explore the possibility of having output such as out_pnt.ww3 also be for each time (out_pnt.YYYYMMDDHH.ww3) so that you would not have to over-write anything in the binary file or cycle through to find out in initialization where you should be in the binary file.

We could also consider adding an option to put the output in folders, since this potentially could increase the number of files which is of a concern in coupled models where all models have lots of output files and it could be an issue for the file system.

@aliabdolali @ajhenrique

[ajhenrique]

Planning for the checkpoint feature has started. My initial idea is to keep the restart files as they are for backwards compatibility, but have an option to use the suffix YYYYMMDDHH rather than NNN. A separate checkpoint file (say, snapshot.ww3.YYYYMMDDHH or checkpoint.ww3.YYYYMMDDHH) would be created with an optional flag indicating how many of the latest checkpoint files would be retained as the model runs -- given restart files can be very large, the retain barrier could keep filesystem footprint under control.

In tandem with the checkpoint file, options to write individual out_grd, out_pnt, partition, and nest outputs would be made available.

Finally, log files would be appended with the continuing run std out if found in the runtime directory.