Skip to content

Publish the pipeline fail-fast decision to logs, metrics, and traces #285

Description

@leynos

Problem

When a pipeline stage fails and Cuprum tears the rest of the pipeline down, the
decision is invisible to every observability surface Cuprum already offers.
PipelineResult.failure_index records which stage lost, but only after the
run returns, and only to the caller holding the result object. An operator
watching logs, a dashboard watching counters, or a trace consumer watching
spans sees a set of stages that all died at once, with nothing saying which one
caused it or that a teardown was deliberate rather than a crash.

Cuprum already publishes a structured ExecEvent stream through sh.observe(),
and already ships logging, metrics, and tracing adapters that project that
stream onto their respective backends. The fail-fast decision should ride those
existing surfaces rather than needing a new one.

Required work

Publish the fail-fast decision as a first-class event and project it through
every shipped adapter.

1. ExecPhase gains pipeline_fail_fast

A new phase reporting a coordinator decision, not a process lifecycle
transition. Emitted at most once per pipeline, on the failing stage's existing
exec_id, and always before termination is requested, so a consumer sees
the intent even if teardown then hangs. The failing stage's own exit event
still follows; no stage's events are skipped, duplicated, or reordered.

ExecEvent gains typed stage_index and stage_count fields. These must be
typed fields rather than tag entries: caller-supplied tags are merged last and
may legitimately shadow a pipeline_stage_index key, and the decision must
report the index the coordinator actually acted on. stage_count is needed to
read stage_index at all — stage 1 of 2 is final, stage 1 of 4 is not.

2. MetricsHook counter

cuprum_pipeline_fail_fast_total, incremented once per torn-down pipeline,
labelled by program and project and nothing else. exec_id is unique
per execution and would leave the series unbounded; stage index and exit code
would multiply series for no aggregate a dashboard needs. Per-incident detail
stays on the event and the span.

3. TracingHook span event

cuprum.pipeline_fail_fast, recorded on the failing stage's already-open
span rather than a new span of its own — the teardown belongs to the trace of
the stage that caused it, and Cuprum has no pipeline-level correlation id to
mint a parent from. The event must go through the same exec_id-keyed,
lock-guarded span lookup the other correlated phases use, so a recycled PID
cannot attach it to the wrong span, and an uncorrelatable event is dropped
rather than guessed at.

4. Structured logging arm

The logging adapter renders the phase at a configurable LogLevels
fail_fast_level, defaulting to WARNING — a pipeline giving up on itself is
not routine INFO traffic.

5. Adapter and protocol carriers

Whatever module structure the above requires, within the project's 400-line
file ceiling: the tracing backend contract (Span / Tracer) may need to move
out of the hook module into its own protocol module, with the documented import
path preserved by re-export.

Acceptance criteria

  • ExecPhase includes pipeline_fail_fast; ExecEvent carries typed
    stage_index and stage_count.
  • The event fires only when a completion newly latches failure_index
    and the failing stage is neither the final stage nor the only stage.
    A failing final stage and a single-stage pipeline emit nothing: there is
    nothing left running to terminate.
  • The event is emitted before termination is requested, carrying the
    failing stage's exec_id, stage_index, stage_count, exit_code, and
    duration_s.
  • MetricsHook increments cuprum_pipeline_fail_fast_total, labelled by
    program and project only.
  • TracingHook adds a cuprum.pipeline_fail_fast event to the failing
    stage's open span, starting no span and ending none; events without a
    resolvable exec_id are dropped.
  • The logging adapter emits the phase at LogLevels.fail_fast_level
    (WARNING by default).
  • Every adapter that matches exhaustively on ExecPhase has an arm for the
    new phase, so none of Cuprum's own hooks fail closed on it.
  • The new phase is recorded as a breaking change in CHANGELOG.md:
    third-party hooks that match exhaustively on phase and reject unknown
    values will raise on it — and thereby fail the run, since Cuprum
    re-raises observe-hook failures rather than swallowing them — until they
    grow an arm.
  • docs/cuprum-design.md, docs/developers-guide.md, and
    docs/users-guide.md document the phase, its firing condition, the
    counter's label set, the span event, and the log level.
  • Covered by tests at the wiring level and end to end, including the
    no-emission cases (final stage, single stage, repeat failures) and the
    span-lifecycle behaviour under interleaved stages.

Notes

This work was specified for, and is delivered by, PR #243 alongside the
completion-transition state-machine tests for #73. The issue records the
requirement that PR satisfies.

Metadata

Metadata

Assignees

No one assigned

    Labels

    IssueconcurrencyConcurrency, parallelism, and synchronization work, including races and deadlocks.documentationImprovements or additions to documentationenhancementNew feature or requestmediumRoadmap items to schedule within the current quarter. Clear scope, normal review cycles.pythontestingTest coverage, test infrastructure, and verification tooling work.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions