add Custom Trace Attributes to extend observability

[Spherrrical]

Introduces Custom Trace Attributes to allow developers to extend business logic!

Today, traces include request path and system metadata, but not application-specific identifiers. This makes it hard to troubleshoot per-tenant, per-workspace, or per-user issues.

#707

[salmanap]

One quick question - why wouldn't these trace attributes live in the customers application code. Are these attributes part of the LLM call, if so why? Is the idea that these multi-tenant attributes must be sent up for tracking certain customers?

Wouldn't it make more sense to simply accept and x-plano-trace-customer-attribute- prefix and rotate through all those and write them to the trace. Why build a static config that needs to get update all the time?

[Spherrrical]

One quick question - why wouldn't these trace attributes live in the customers application code. Are these attributes part of the LLM call, if so why? Is the idea that these multi-tenant attributes must be sent up for tracking certain customers?

These attributes are intentionally configured at the gateway so we can consistently attach tenant/workspace/user metadata to every trace without requiring each customer app/SDK to know our trace schema or emit spans themselves. They’re not part of the LLM payload; we read them from the incoming request headers and add them to the trace for operational filtering across routing/LLM/agent spans. the goal is multi‑tenant attribution so we can slice traces by customer/tenant/user reliably across all requests per the customers choice.

Wouldn't it make more sense to simply accept and x-plano-trace-customer-attribute- prefix and rotate through all those and write them to the trace. Why build a static config that needs to get update all the time?

We use a gateway allowlist instead of a wildcard prefix so trace schemas stay stable and auditable. The config maps header → attribute key with type parsing (str, int, etc.), which prevents sensitive or arbitary headers from being logged and keeps queries consistent. A prefix would let any client inject new "keys" and leak data/over permissive in data.