You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When a pipeline stage fails and Cuprum tears the rest of the pipeline down, the
decision is invisible to every observability surface Cuprum already offers. PipelineResult.failure_index records which stage lost, but only after the
run returns, and only to the caller holding the result object. An operator
watching logs, a dashboard watching counters, or a trace consumer watching
spans sees a set of stages that all died at once, with nothing saying which one
caused it or that a teardown was deliberate rather than a crash.
Cuprum already publishes a structured ExecEvent stream through sh.observe(),
and already ships logging, metrics, and tracing adapters that project that
stream onto their respective backends. The fail-fast decision should ride those
existing surfaces rather than needing a new one.
Required work
Publish the fail-fast decision as a first-class event and project it through
every shipped adapter.
1. ExecPhase gains pipeline_fail_fast
A new phase reporting a coordinator decision, not a process lifecycle
transition. Emitted at most once per pipeline, on the failing stage's existing exec_id, and always before termination is requested, so a consumer sees
the intent even if teardown then hangs. The failing stage's own exit event
still follows; no stage's events are skipped, duplicated, or reordered.
ExecEvent gains typed stage_index and stage_count fields. These must be
typed fields rather than tag entries: caller-supplied tags are merged last and
may legitimately shadow a pipeline_stage_index key, and the decision must
report the index the coordinator actually acted on. stage_count is needed to
read stage_index at all — stage 1 of 2 is final, stage 1 of 4 is not.
2. MetricsHook counter
cuprum_pipeline_fail_fast_total, incremented once per torn-down pipeline,
labelled by program and projectand nothing else. exec_id is unique
per execution and would leave the series unbounded; stage index and exit code
would multiply series for no aggregate a dashboard needs. Per-incident detail
stays on the event and the span.
3. TracingHook span event
cuprum.pipeline_fail_fast, recorded on the failing stage's already-open
span rather than a new span of its own — the teardown belongs to the trace of
the stage that caused it, and Cuprum has no pipeline-level correlation id to
mint a parent from. The event must go through the same exec_id-keyed,
lock-guarded span lookup the other correlated phases use, so a recycled PID
cannot attach it to the wrong span, and an uncorrelatable event is dropped
rather than guessed at.
4. Structured logging arm
The logging adapter renders the phase at a configurable LogLevels fail_fast_level, defaulting to WARNING — a pipeline giving up on itself is
not routine INFO traffic.
5. Adapter and protocol carriers
Whatever module structure the above requires, within the project's 400-line
file ceiling: the tracing backend contract (Span / Tracer) may need to move
out of the hook module into its own protocol module, with the documented import
path preserved by re-export.
Acceptance criteria
ExecPhase includes pipeline_fail_fast; ExecEvent carries typed stage_index and stage_count.
The event fires only when a completion newly latches failure_index and the failing stage is neither the final stage nor the only stage.
A failing final stage and a single-stage pipeline emit nothing: there is
nothing left running to terminate.
The event is emitted before termination is requested, carrying the
failing stage's exec_id, stage_index, stage_count, exit_code, and duration_s.
MetricsHook increments cuprum_pipeline_fail_fast_total, labelled by program and project only.
TracingHook adds a cuprum.pipeline_fail_fast event to the failing
stage's open span, starting no span and ending none; events without a
resolvable exec_id are dropped.
The logging adapter emits the phase at LogLevels.fail_fast_level
(WARNING by default).
Every adapter that matches exhaustively on ExecPhase has an arm for the
new phase, so none of Cuprum's own hooks fail closed on it.
The new phase is recorded as a breaking change in CHANGELOG.md:
third-party hooks that match exhaustively on phase and reject unknown
values will raise on it — and thereby fail the run, since Cuprum
re-raises observe-hook failures rather than swallowing them — until they
grow an arm.
docs/cuprum-design.md, docs/developers-guide.md, and docs/users-guide.md document the phase, its firing condition, the
counter's label set, the span event, and the log level.
Covered by tests at the wiring level and end to end, including the
no-emission cases (final stage, single stage, repeat failures) and the
span-lifecycle behaviour under interleaved stages.
Notes
This work was specified for, and is delivered by, PR #243 alongside the
completion-transition state-machine tests for #73. The issue records the
requirement that PR satisfies.
Problem
When a pipeline stage fails and Cuprum tears the rest of the pipeline down, the
decision is invisible to every observability surface Cuprum already offers.
PipelineResult.failure_indexrecords which stage lost, but only after therun returns, and only to the caller holding the result object. An operator
watching logs, a dashboard watching counters, or a trace consumer watching
spans sees a set of stages that all died at once, with nothing saying which one
caused it or that a teardown was deliberate rather than a crash.
Cuprum already publishes a structured
ExecEventstream throughsh.observe(),and already ships logging, metrics, and tracing adapters that project that
stream onto their respective backends. The fail-fast decision should ride those
existing surfaces rather than needing a new one.
Required work
Publish the fail-fast decision as a first-class event and project it through
every shipped adapter.
1.
ExecPhasegainspipeline_fail_fastA new phase reporting a coordinator decision, not a process lifecycle
transition. Emitted at most once per pipeline, on the failing stage's existing
exec_id, and always before termination is requested, so a consumer seesthe intent even if teardown then hangs. The failing stage's own
exiteventstill follows; no stage's events are skipped, duplicated, or reordered.
ExecEventgains typedstage_indexandstage_countfields. These must betyped fields rather than tag entries: caller-supplied tags are merged last and
may legitimately shadow a
pipeline_stage_indexkey, and the decision mustreport the index the coordinator actually acted on.
stage_countis needed toread
stage_indexat all — stage 1 of 2 is final, stage 1 of 4 is not.2.
MetricsHookcountercuprum_pipeline_fail_fast_total, incremented once per torn-down pipeline,labelled by
programandprojectand nothing else.exec_idis uniqueper execution and would leave the series unbounded; stage index and exit code
would multiply series for no aggregate a dashboard needs. Per-incident detail
stays on the event and the span.
3.
TracingHookspan eventcuprum.pipeline_fail_fast, recorded on the failing stage's already-openspan rather than a new span of its own — the teardown belongs to the trace of
the stage that caused it, and Cuprum has no pipeline-level correlation id to
mint a parent from. The event must go through the same
exec_id-keyed,lock-guarded span lookup the other correlated phases use, so a recycled PID
cannot attach it to the wrong span, and an uncorrelatable event is dropped
rather than guessed at.
4. Structured logging arm
The logging adapter renders the phase at a configurable
LogLevelsfail_fast_level, defaulting toWARNING— a pipeline giving up on itself isnot routine
INFOtraffic.5. Adapter and protocol carriers
Whatever module structure the above requires, within the project's 400-line
file ceiling: the tracing backend contract (
Span/Tracer) may need to moveout of the hook module into its own protocol module, with the documented import
path preserved by re-export.
Acceptance criteria
ExecPhaseincludespipeline_fail_fast;ExecEventcarries typedstage_indexandstage_count.failure_indexand the failing stage is neither the final stage nor the only stage.
A failing final stage and a single-stage pipeline emit nothing: there is
nothing left running to terminate.
failing stage's
exec_id,stage_index,stage_count,exit_code, andduration_s.MetricsHookincrementscuprum_pipeline_fail_fast_total, labelled byprogramandprojectonly.TracingHookadds acuprum.pipeline_fail_fastevent to the failingstage's open span, starting no span and ending none; events without a
resolvable
exec_idare dropped.LogLevels.fail_fast_level(
WARNINGby default).ExecPhasehas an arm for thenew phase, so none of Cuprum's own hooks fail closed on it.
CHANGELOG.md:third-party hooks that match exhaustively on phase and reject unknown
values will raise on it — and thereby fail the run, since Cuprum
re-raises observe-hook failures rather than swallowing them — until they
grow an arm.
docs/cuprum-design.md,docs/developers-guide.md, anddocs/users-guide.mddocument the phase, its firing condition, thecounter's label set, the span event, and the log level.
no-emission cases (final stage, single stage, repeat failures) and the
span-lifecycle behaviour under interleaved stages.
Notes
This work was specified for, and is delivered by, PR #243 alongside the
completion-transition state-machine tests for #73. The issue records the
requirement that PR satisfies.