`checkpoints.txt` not checked when restarting

[PrometheusPi]

With openPMD-api automated series read in, only the existence of an (ADIOS) checkpoint file is enough to try a restart from that file, even if it is corrupted due to a failed writing of the checkpoint.
Thus, restarting currently requires manual cleanup.
Using the last entry in checkpoints.txt could avoid that problem and enable automatic restarts.

[psychocoderHPC]

To restart form a checkpoint all plugin checkpoints must be valid too.
To handle invalid restart files we need

• extend the plugin interface with a method where each plugin is validating there checkpoint file
• openPMD files need a way report to the openPMD-api user that a file is valid.

[PrometheusPi]

@franzpoeschel is there are way for openPMD-api to report corrupted iterations in a series?

[franzpoeschel]

The upcoming openPMD-api release (see openPMD/openPMD-api#1391) will recover from errors at parsing time. An iteration that is entirely unreadable will be skipped.
If however only some single records or so are broken, then only those will skipped and the rest of the iteration will be shown (might happen in HDF5).
Note that this covers not yet all kinds of errors that might happen at read time.

[psychocoderHPC]

@PrometheusPi Please keep in mind that every plugin can write checkpoint data, so it is not enough to check the openPMD files only.
@franzpoeschel IMO the more important question is: Is there a call in openPMD where I can pass in a file and ask if this file is guaranteed fully written? Where fully written means also all disk IO is flushed and writing was not canceled during the flush.
Does openPMD files contain an internal flag to mark that all series are successfully flushed which is written after the flush?

[franzpoeschel]

No, openPMD does not explicitly have any atomic writing functionality (yet)

[PrometheusPi]

@franzpoeschel I do not get, why an atomic writing needs to exist for that, could you elaborate?

[franzpoeschel]

Doing this would be an atomic write, it's how such things are commonly implemented. Use a flag to indicate if the iteration is there or not. It's either fully there or not at all.

[PrometheusPi]

@franzpoeschel thanks for the explanation. I naively thought an array of length equal number of writers that is set to a specify (True) value by each writer after they are finished with everything else would be sufficient.

[ax3l]

Does openPMD files contain an internal flag to mark that all series are successfully flushed which is written after the flush?

Originally, the idea was to write the /openPMD attribute last to signal the writing was complete.

At the same time, the request for an atomic completion flag collides in practice with other user requests, which ask to "try to recover and read as much as possible" of a crashed simulation by default.

[franzpoeschel]

At the same time, the request for an atomic completion flag collides in practice with other user requests, which ask to "try to recover and read as much as possible" of a crashed simulation by default.

In practise, can't this be solved by having a call auto Iteration::finalized() -> bool that evaluates such a flag, but the Iteration is still parsed and presented to the user anyway?

[ax3l]

That would work.

I am not sure we get such a guarantee from HDF5 or ADIOS2 though:

IMO the more important question is: Is there a call in openPMD where I can pass in a file and ask if this file is guaranteed fully written? Where fully written means also all disk IO is flushed and writing was not canceled during the flush.

We can try somewhat and rely that closed write steps imply things are on the way to disk ... and hope FS-worlds and MPI-world are magically in sync (they usually are not).

[ax3l]

I think the underlying resolution to this issue in PIConGPU is to delay writing the entry in checkpoint.txt to when all checkpoints from plugins & openPMD-api are written.

Recover only from full checkpoints. Trying to "fix" a corrupted (partial) checkpoint indeed needs manual intervention - it should by default not be taken to avoid surprises and undefined recovery of sim state.

Did I get this right or is there more context to the issue that I missed in the description? From the issue description, it feels to me there is a specific sim crash situation here as a premise that I don't understand.