Add commit-aware recovery tracing and defer restart

[Amaury Chamayou (achamayou)]

Defers host restart until the recovery protocol's JOINING transaction is globally committed, preventing aborted transactions from triggering a restart. Adds disabled-by-default recovery tracing (CCF_RECOVERY_TRACE), with commit-aware events and causal message IDs in node logs.

Implementation prelude for #8282. The unversioned trace format is internal to CI and coupled to the matching Lean reader.

[Copilot]

🟡 Changes recommended

Address trace ID validation, JOINING timeout handling, and trace-enabled regression coverage.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds commit-aware recovery restart handling and opt-in recovery tracing for deterministic replay validation.

Changes:

• Defers restarts until JOINING commits globally.
• Adds causal, versioned recovery tracing.
• Documents the trace format and adds the build option.

File summaries

File	Description
src/node/rpc/self_healing_open_handlers.h	Validates and records traced recovery messages.
src/node/recovery_decision_protocol.h	Adds tracing state and APIs.
src/node/recovery_decision_protocol.cpp	Implements deferred restart and tracing behavior.
lean/disaster-recovery-trace/TRACE_FORMAT_V1.md	Documents the NDJSON trace contract.
include/ccf/service/tables/self_healing_open.h	Defines trace event storage and serialization.
CMakeLists.txt	Adds the disabled-by-default tracing option.
CHANGELOG.md	Records the recovery restart behavior change.

Review details

Suppressed comments (3)

CMakeLists.txt:473

• The new behavior is behind CCF_RECOVERY_TRACE, but this change does not add or enable any test/build configuration for that branch: the existing recovery e2e cases run with the default (OFF) setting. As a result, the trace-only JSON serialization, commit-hook ordering, and deferred restart path can compile or behave incorrectly without CI detecting it; add a trace-enabled recovery regression/CI build (or land this together with the promised follow-up validation).

option(
  CCF_RECOVERY_TRACE
  "Enable committed recovery-decision-protocol tracing"
  OFF
)
if(CCF_RECOVERY_TRACE)
  add_compile_definitions(CCF_RECOVERY_TRACE)
endif()

src/node/recovery_decision_protocol.cpp:235

• The failover timer can have an in-flight timeout request when an IAmOpen transaction moves the node to JOINING, and advance() is still called for that request while the state is already JOINING. This unconditional check records another join_restart and the trace commit hook queues another host restart for a JOINING->JOINING timeout; the non-trace path now restarts only on the committed state transition, and the trace contract treats this as a one-shot effect. Restrict this observation to the transition into JOINING by also requiring pre != JOINING.

    if (post == recovery_decision_protocol::StateMachine::JOINING)

src/node/recovery_decision_protocol.cpp:390

• Please add a regression scenario that causes the transaction which writes JOINING to abort and verifies that the joiner remains running. The existing recovery e2e cases only cover successful completion, so they would not catch a regression that moved this restart back before global commit.

#ifndef CCF_RECOVERY_TRACE
            restart_after_commit();
#else
        // The trace-event commit hook emits join_restart before restarting.
#endif

• Files reviewed: 7/7 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.