Add OpenTelemetry traces and metrics to peer and orderer

[dviejokfs]

What this does

Adds OpenTelemetry traces and metrics to the peer and the orderer, so that a
slow transaction can be attributed to a specific stage rather than inferred from
aggregate metrics.

The existing Prometheus metrics answer how long endorsement takes on this peer.
They cannot answer where the 900ms went for this particular transaction,
because that time is spread across a client, one or more endorsing peers, the
ordering service, and finally a commit on every peer in the channel.

Everything is inert unless an OTLP endpoint is configured. With no OTEL_*
variables set, no provider is installed, all instrumentation falls through to the
OpenTelemetry no-op implementation, and the nodes behave exactly as they do
today.

Spans

Endorsement — preProcess, GetTxSimulator (contention with commit, since
the simulator takes a shared lock on the state DB), ExecuteChaincode,
GetTxSimulationResults, DistributePrivateData, EndorseWithPlugin.

Ordering — ProcessNormalMsg, WaitReady (consenter backpressure), Order.

Commit — Validate, RetrievePvtdata, CommitLegacy.

The design problem worth reviewing

Endorsement and ordering are ordinary gRPC calls, so W3C traceparent
propagates natively via request metadata and those spans join the client's trace
with no changes to transactions.

Commit cannot work that way. A block is cut asynchronously, contains transactions
from many unrelated clients, and commits independently on every peer, so there is
no ambient context to continue and no single parent a block could belong to.
Block commit spans are therefore separate traces related to transactions by
span links, resolved from either:

1. What the peer recorded when it endorsed the transaction — bounded in time and
   size, and lossy by design, since a peer that did not endorse never had the
   context.
2. Trace context the client embedded in the signed transaction. Implemented here
   as protobuf fields 1000/1001 on ChaincodeHeaderExtension — unknown fields
   that stock Fabric preserves and ignores, and which verify normally because the
   client signs the header after adding them.

When neither resolves, the commit is still traced and still carries
fabric.tx_id, so the traces can be joined at query time.

Metrics

Rather than adding new metric definitions, this implements Fabric's existing
metrics.Provider over the OTEL SDK and registers it as a new otel provider
alongside statsd/prometheus/disabled. Every metric Fabric already defines
is then exported over OTLP with no new instrumentation call sites, which is what
keeps the OTLP and Prometheus views from drifting apart.

Note that Fabric's Gauge supports both Set and Add while OTLP carries only
absolute values, so Add is resolved against the previous reading per label set.

Deliberate exclusions

Gossip, the Raft cluster service and health checks are not traced: their volume
scales with cluster size rather than transaction load, and consensus timing is
better served by the existing metrics.

Long-lived streams (Deliver, and AtomicBroadcast at the stream level) are
excluded from automatic instrumentation. A span lasts as long as its RPC, so
wrapping a stream open for hours would hold one span in memory that whole time
and produce a single span covering everything the stream ever carried. Ordering
is traced per message instead.

Status and known limitations

Opening as a draft — this is a cross-cutting change and I would rather agree
the approach before polishing it. Specifically:

• This has not been run against a live OTLP collector. Unit tests confirm
  spans are recorded and metrics aggregate correctly, but the OTLP exporter
  connects lazily, so nothing here proves data lands in a backend.
• The client side of envelope-carried trace context is not implemented. The
  peer side reads it and MarshalTraceContextExtension exists as a reference
  implementation, but emitting the fields needs a change in whichever SDK builds
  transactions.
• Happy to move this to an RFC in hyperledger/fabric-rfcs if that is the
  preferred route for a change of this scope.

Testing

• peer and orderer both build; go vet clean on all changed packages
• Tests pass for telemetry, endorser, broadcast, privdata, operations,
  comm
• 22 tests in the new package, covering the inert path, the enabled path, the
  registry's expiry and size bounds, the metrics provider's Set/Add
  semantics, and a proto round-trip proving a transaction carrying trace context
  still parses and re-serializes correctly on a peer that does not know about it

Notes for reviewers

Design rationale and rebase notes are in docs/source/telemetry.md.

The diff is 316 files, but only 24 are source — the rest is vendor/ from
go mod vendor.

Instrumentation is confined to internal/pkg/telemetry plus seven call sites.
internal/pkg/comm gains a StatsHandlers field because gRPC supports multiple
stats handlers and Fabric only exposed one. context.Context is threaded through
ProcessProposalSuccessfullyOrError, simulateProposal, callChaincode and
ProcessMessage.

[dviejokfs]

@pfi79 thoughts? this is important especially to detect performance issues in the peer/orderer in companies using OTEL in their observability stack.

[pfi79]

@pfi79 thoughts? this is important especially to detect performance issues in the peer/orderer in companies using OTEL in their observability stack.

In my memory, simply inserting telemetry into the code led to serious degradation of the programs.
I would like to understand how much the work will deteriorate, with telemetry enabled, under high load.

[dviejokfs]

@pfi79 thoughts? this is important especially to detect performance issues in the peer/orderer in companies using OTEL in their observability stack.

In my memory, simply inserting telemetry into the code led to serious degradation of the programs. I would like to understand how much the work will deteriorate, with telemetry enabled, under high load.

was OTEl the telemetry inserted into the code? do you have a PR or a reference to not repeat the same mistake?

I think telemetry needs to be enabled per component I agree, and I think the bottleneck can be the network rather than cpu, since the OTEL library sends the traces to the collector, nothing else.

So we could have env vars to enable telemetry in the chaincode, peer, gossip, etc

For the companies using fabric, the most important part is to be able to instrument the chaincode invocations IMHO.