State corruption when hydra-node storage becomes full during write operations

[DeltaDeFiProtocol]

Summary

When the hydra-node storage persistence becomes full, incomplete write operations can corrupt the node state, potentially leading to data loss and requiring manual intervention to recover.

Description

I encountered a critical issue where my hydra-node became unresponsive due to state corruption. The root cause appears to be insufficient disk space during a state write operation, which resulted in an
incomplete write that corrupted the persisted state.

When attempting to restart the node from the persisted state, I received the following error:
Error when decoding from file /persistence/persistence-alice/state: "Unexpected end-of-input while parsing string literal"

This error indicates that the state file was truncated or corrupted during the write operation when storage became full.

Attached is a copy of the corrupted state.
state.zip

Steps to Reproduce

1. Run hydra-node with limited storage space
2. Allow storage to fill up during normal operation
3. Node attempts to persist state during low disk space conditions
4. Write operation is interrupted/incomplete due to storage constraints
5. Node state becomes corrupted and unrecoverable
6. Restart attempt fails with JSON parsing error due to truncated state file

Expected Behavior

• Hydra-node should handle storage constraints gracefully
• State writes should be atomic or include rollback mechanisms
• Recovery mechanisms should be available for corrupted states

Actual Behavior

• State corruption occurs when write operations are incomplete
• Node becomes unresponsive/unrecoverable
• Restart fails with "Unexpected end-of-input while parsing string literal" error
• No clear recovery path without manual intervention

Environment

• Hydra Node Version: 0.22.3

Questions for the Team

1. Are there any existing safeguards against storage-related state corruption?
2. Is there a planned fix or mitigation strategy for this issue?
3. Are there recovery procedures for corrupted state files?

[noonio]

Thanks for reporting this.

We have a "hack" to address this recently; merged here #2255 (on master, not yet relased).

For longer term options, we will probably work on #1585 in combination with investigating some different statefile backends instead of just file-based.

The best safeguard is to use the persistence-rotate-after command-line argument, and just archive any state-* files (these ones aren't used anymore). This should keep your disk space roughly constant; depending on the details in the UTxO set/snapshot of course.

PS: This issue is basically captured here - #2253 - but I'll leave this open as it captures a bit more detail and feature requests.