Worker processes every UpdateFlow wrap-up with a hardcoded unrecoverable=true (benign wrap-ups log as errors; success=false wrap-ups skip retry and continue_on_error) #11498

Describe the bug

Every flow wrap-up that travels through the SendResultPayload::UpdateFlow channel is processed by the worker with a hardcoded unrecoverable: true argument. This produces two problems:

1. Benign completions are logged as unrecoverable. A branchone without a default branch, when no branch expression matches, completes its router module as Success { job: null } (via NextFlowTransform::EmptyInnerFlows). The wrap-up then goes through the UpdateFlow channel and the worker logs a line that reads as a severe error:

   success = true, unrecoverable = true, continue_on_error = false, should_continue_flow = true
   

   The run succeeds. We chased this in production as a false alarm.

2. success = false wrap-ups skip retry and continue_on_error. The suspend disapproval/timeout path (worker_flow.rs, the PushNextFlowJob::Done(Some(UpdateFlow { success: false, ... })) sender for SuspendedDisapproved / SuspendedTimedOut) sends a failure through the same channel. At the receiver, unrecoverable = true makes the false if unrecoverable arm of the should_continue_flow match win before the continue_on_error / skip_failure / retry arms, so a failure on that path is never retried and continue_on_error is silently ignored. Whether a suspend timeout should retry is a product question, but the decision currently happens as a side effect of a hardcoded flag whose own comments define it as "worker crash/OOM" semantics, and the sender cannot express the choice.

To Reproduce

1. Create a flow with a branchone module that has no default branch (only branches).
2. Run the flow so that no branch expression matches.
3. The run succeeds, but the worker log shows the success = true, unrecoverable = true, ... line above.

Expected behavior

unrecoverable should be chosen at the send site, or derived, rather than hardcoded at the receiver. The UpdateFlow message already carries success and stop_early_override; an explicit unrecoverable field (defaulting to false) would let the suspend timeout sender decide its own semantics, and would stop benign wrap-ups from logging as unrecoverable.

Screenshots

Not applicable; the log line above is the observable behavior.

Version (payload)

Windmill v1.775.2, self-hosted (deployed via Helm chart). Relevant code at that tag:

• backend/windmill-worker/src/result_processor.rs, in process_jc, the SendResultPayload::UpdateFlow arm: the call to update_flow_status_after_job_completion passes success from the message but hardcodes true for the unrecoverable parameter (line 480).
• backend/windmill-worker/src/worker_flow.rs: the should_continue_flow match, where the false if unrecoverable arm pre-empts the continue_on_error / retry arms, and the suspend timeout/disapprove sender that emits UpdateFlow { success: false, ... }.