Propagate CDDS sub-workflow failures and prevent stalled CDDS runs blocking subsequent CMEW workflows

[mo-nikosbaltas]

Description

CMEW launches a CDDS conversion as a sub-workflow (cdds_CMEW) during the standardise_model_data task and waits for it to complete. If the CDDS workflow stalls or fails, the parent CMEW workflow does not fail fast and continues running indefinitely.
Additionally, while a stalled or failed cdds_CMEW workflow remains present, new CMEW workflows cannot be started successfully without manual intervention (e.g. cylc stop --now --now cdds_CMEW). This creates operational coupling between CMEW runs and requires human cleanup to recover.
This behaviour was observed repeatedly during failures in the standardise_model_data task when CDDS conversion failed.

---

Observed behaviour
• CDDS (cdds_CMEW) fails or stalls.
• The parent CMEW task continues running and waits indefinitely.
• Subsequent CMEW workflows are blocked because an existing cdds_CMEW workflow is still present.
• Manual cleanup (cylc stop --now --now cdds_CMEW) is required before a new CMEW run can proceed.

---

Expected behaviour

1. If the CDDS sub-workflow fails or stalls, the parent CMEW workflow should:

• Detect the failure promptly
• Fail the standardise_model_data task
• Stop or clean up the child CDDS workflow automatically

2. A failed or stalled CDDS run from one CMEW workflow should not block subsequent CMEW workflows.

---

Impact

• CMEW workflows can remain running indefinitely with no progress.
• New CMEW runs are operationally blocked by previous failed runs.
• Manual intervention is required to recover, reducing reliability and automation.
• Makes parallelisation and scaling of CMEW unsafe.

---

Proposed implementation plan

1. Fail-fast propagation from CDDS to CMEW Will be resolved via Investigate the use of external triggers to await completion of the CDDS workflow #158
   • After launching the CDDS workflow, explicitly monitor its state.
   • If the CDDS workflow enters a failed or stopped state, the CMEW task should exit non-zero and fail.
   • Avoid masking CDDS failures by wrapper scripts.

---

2. Introduce a timeout watchdog for CDDS Captured in Investigate the use of external triggers to await completion of the CDDS workflow #158
   • Add a configurable timeout (e.g. CDDS_TIMEOUT_HOURS).
   • If CDDS does not complete within the timeout:
   o Mark the CMEW task as failed
   o Trigger cleanup logic
   • Prevent infinite waiting states.

---

3. Ensure CDDS workflows are uniquely named per CMEW run Completed via Launch cdds_convert with a custom workflow name  #164
   • Avoid using a fixed workflow name (cdds_CMEW) for all runs.
   • Include a unique identifier (e.g. CMEW run ID, suite ID, or timestamp) in the CDDS workflow name.
   • This prevents cross-run interference and allows safe parallel execution.

---

4. Automatic cleanup on failure Will be resolved via Stop and clean CDDS workflows #519
   • On CDDS failure or timeout:
   o Automatically stop the child CDDS workflow
   o Optionally clean its run directory if safe to do so
   • Remove the need for manual cylc stop --now commands.

---

5. Improve logging and status visibility Captured in Investigate the use of external triggers to await completion of the CDDS workflow #158
   • Log the CDDS workflow name, run directory, and elapsed wait time.
   • Periodically report that CMEW is waiting on CDDS and its current state.
   • Make it clear in logs and Cylc UI why CMEW is blocked.

---

Acceptance criteria
• CMEW fails cleanly if CDDS fails.
• A failed CDDS run does not block subsequent CMEW workflows: no manual cleanup is required to start a new CMEW run after a CDDS failure.
• Behaviour is documented and reproducible.

---

Notes
This work is a prerequisite for safely parallelising CDDS standardisation (see #315) and improves overall CMEW robustness and operational reliability.