Dead letters from local queues are never counted in the per-type/per-tenant DeadLetters metric (every local:// destination is classified System) #4665

Symptom (measured in CritterWatch, WolverineFx 6.41.0)

A message that dead-letters from a local queue never shows up in the per-message-type / per-tenant DeadLetters metric that MessageHandlingMetrics publishes. The rows are in the dead-letter table. On a Fisher-backed monitored service, the published metrics read:

ShareTodoList | globex | UnreachableRecipientException   failures=8   deadLetters=0

while wolverine_dead_letters held both of that tenant's dead-lettered envelopes. Executions and failures for the same message type and tenant are counted.

Mechanism, read from the source

In WolverineRuntime.MovedToErrorQueue (src/Wolverine/Runtime/WolverineRuntime.Tracking.cs, ~:251), the per-type accumulator post is gated:

if (Options.Metrics.Mode != WolverineMetricsMode.SystemDiagnosticsMeter
    && envelope.MessageType.IsNotEmpty()
    && !IsSystemEndpoint(envelope.Destination))
{
    ...Post(new RecordDeadLetter(ex.GetType().FullNameInCode(), envelope.TenantId!));
}

classifyDestination (~:334) marks every local:// URI as DestinationClassification.System, so any dead letter from a local queue skips the post. Failures and executions are recorded on the executor's tracker, which has no such gate, and that is why they are counted.

The System classification exists to keep Wolverine's own traffic (local:// system queues, wolverine.response) out of the metrics. A user's local queue is not system traffic, though. Since GH-3774 (CompletionTrackerFor returns the runtime-global tracker), the Hybrid tracker's unguarded RecordDeadLetter path (HybridMetricsPublishingMessageTracker.cs:103) is no longer reached for these envelopes, so nothing else counts them either. That last point is inferred from the code, not traced at runtime.

Impact

Any consumer of the per-type DeadLetters metric under-reports every local-queue dead letter as zero. For CritterWatch that means the dead-letter-rate alert (MetricsDlqRate) and the per-tenant dead-letter column are blind to them. Local durable queues are a common home for handlers, and they are the default in UseDurableLocalQueues() setups.

Suggested direction (a pointer, not a diagnosis)

Gate on "is this Wolverine's own system traffic" rather than on "is this local". For example, check the endpoint's EndpointRole.System, which the classification could consult, instead of the local scheme alone. CritterWatch can provide a repro service: it now has a Fisher sample (FisherTodoService) whose failing command dead-letters from a local queue.

🤖 Generated with Claude Code