[OTEL] Retries don't correlate to the failed attempt they follow - sibling spans, not linked spans #4398

Summary

When a message fails and is retried (inline, scheduled, or requeued), Wolverine starts a brand
new Activity for the next attempt that is only ever a sibling of the failed attempt's
Activity — both hang off whatever Envelope.ParentId the envelope originally carried on
receipt. There's no trace-level reference from "attempt 2" back to "attempt 1", beyond an
ActivityEvent (wolverine.envelope.retried / wolverine.envelope.rescheduled /
wolverine.envelope.requeued) added to the failed attempt's own span, which does nothing to
help you navigate forward from that failure to what happened next.

In a trace backend (Jaeger, Tempo, Application Insights, etc.) this means retries of the same
logical message show up as unrelated spans scattered across the trace of whatever originally
published the message — you can't click from "attempt 1 failed with TimeoutException" to
"here's attempt 2, which succeeded."

Where this lives in the code

WolverineTracing.StartEnvelopeActivity
parents every new envelope Activity off Envelope.ParentId only.

RetryInlineContinuation.ExecuteAsync, ScheduledRetryContinuation.ExecuteAsync, and
RequeueContinuation.ExecuteAsync all call activity?.AddEvent(...) on the failed attempt and
then move on — none of them touch Envelope.ParentId or anything else that would let the next
attempt's Activity reference this one.

Why this isn't just "add ParentId reparenting"

It's tempting to fix this by setting envelope.ParentId = activity.Id before retrying, and
that's exactly what we did as a local workaround in our own codebase via a custom
IChain/receive middleware. But per the OpenTelemetry messaging semantic
conventions, a
parent/child relationship is supposed to represent temporal containment — the child's duration
should live inside the parent's. A retry doesn't satisfy that: attempt 1 has already finished
(with an exception) by the time attempt 2 starts, and for a ScheduledRetryContinuation there
can be seconds or minutes between them. The documented mechanism for "these two spans are
related, but neither contains the other" is ActivityLink, not reparenting.

Prior art in other .NET messaging libraries

Two other .NET messaging libraries already made this exact call for the same scenario:

• MassTransit propagates an MT-Activity-Propagation: "Link" header on redelivered/retried
  jobs specifically so the consumer links back to the original activity instead of reparenting
  under it (JobStateMachine.cs, LogContextActivityExtensions.cs).
• Brighter's BrighterTracer.GetMessagePumpLinks explicitly builds ActivityLinks for its
  message-pump-to-consume-span relationship, with an in-code comment citing the same OTel
  semantic convention as the reason parent/child isn't used there.

Proposed fix

Add Envelope.PreviousAttemptActivityId, wired through the same serialization and
EnvelopeMapper header-propagation path that ParentId already uses (so it survives a durable
reschedule via the inbox, and round-trips over external transports the same way ParentId
does). The three retry continuations stamp it from the failed attempt's Activity.Id
immediately before retrying/rescheduling/requeuing. WolverineTracing.StartEnvelopeActivity
consumes it as a single-use ActivityLink when starting the next attempt's Activity, in
addition to (not instead of) the existing ParentId-based parent, which still correctly
reflects the original inbound context.

Happy to open this as a PR if the general direction (new envelope field + ActivityLink on
retry, rather than reparenting) looks right. Wanted to raise it as an issue first since it
touches the wire format (EnvelopeConstants, EnvelopeSerializer, EnvelopeMapper) and I'd
rather confirm the approach before asking for review on that.

Environment

• Wolverine: main branch, current HEAD as of this report.
• Discovered while auditing a custom middleware in our own codebase that manually did
  envelope.ParentId = Activity.Current?.Id ?? envelope.ParentId on receive — which "worked"
  for visually connecting attempts in our trace backend, but reparents rather than links, which
  is the wrong shape per the OTel convention above.