mcp: add configured header attribtues to metrics and spans #1342

nacx · 2025-10-12T15:47:32Z

Description

The LLM metrics and trace providers allow for the configuration of custom headers to be included as metric attributes and span attributes. The MCP implementation, however, did not honor those values.

This PR updates the MCP metrics and tracer configurations to also take this configuration into account.

The tracer method to create a span now receives the request headers.
To avoid having to propagate request headers all the way down inside the MCP Proxy so that they can be taken into account when recording metrics, I've opted to instantiate an MCPProxy on each request, and have it set up with the request attributes from the beginning. This allows reusing the metrics implementation in a clean way.

Related Issues/PRs (if applicable)

Fixes #1335
Fixes #1348

Special notes for reviewers (if applicable)

N/A

codecov-commenter · 2025-10-12T15:54:53Z

Codecov Report

❌ Patch coverage is 97.70115% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.13%. Comparing base (3341eb8) to head (296fa40).

Files with missing lines	Patch %	Lines
internal/mcpproxy/mcpproxy.go	77.77%	4 Missing ⚠️

❌ Your project status has failed because the head coverage (77.13%) is below the target coverage (86.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1342      +/-   ##
==========================================
+ Coverage   77.03%   77.13%   +0.09%     
==========================================
  Files         125      126       +1     
  Lines       16136    16204      +68     
==========================================
+ Hits        12431    12499      +68     
  Misses       3080     3080              
  Partials      625      625

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codefromthecrypt

this is "ok" but I think we are misunderstanding the session concept here. So attributes like session.id will need to be propagated as request meta, just like trace IDs are. This change will not be able to be used for all kinds of session because it relies on header which are typically not set in the smaller scope of JSON rpc messages.

codefromthecrypt · 2025-10-13T02:36:29Z

internal/metrics/mcp_metrics.go

+	initializationDuration    metric.Float64Histogram
+	capabilitiesNegotiated    metric.Float64Counter
+	progressNotifications     metric.Float64Counter
+	requestHeaderLabelMapping map[string]string // maps HTTP headers to metric label names.


please use the word attribute as this is otel and not just prom. label is a prom thing and applied at export

Done. I've also added a commit to rename it in the chat completion and embeddings metrics implementations

codefromthecrypt · 2025-10-13T02:37:15Z

internal/mcpproxy/handlers.go

 	}

-	span := m.tracer.StartSpanAndInjectMeta(ctx, req, p)
+	// TODO: headers with multiple values


curious why not pass http.Header type instead of converting it to a map like this?

Mostly to keep a similar interface to the chat completions one: https://github.com/envoyproxy/ai-gateway/blob/main/internal/tracing/api/api.go#L81
But I can change to http.Header if we think that'd be better.

it isn't too important just I must have started a problem with copying in maps? if we don't need to copy we shouldn't especailly as headers are case insentive in http

codefromthecrypt · 2025-10-13T02:48:08Z

another way out would be to do the HTTP tier in case some attribute maps apply, just don't close out MCP mapping until there's another attribute for request meta -> span attributes. Doing things at the HTTP layer will apply attributes to multiple traces (to to the JSON-RPC messaging layer), which is different than normal HTTP where one request is a single span.

modelcontextprotocol/modelcontextprotocol#414 is the details about why trace propagation is at meta level, and so likely should be other things like session.id.

nacx · 2025-10-13T07:59:12Z

I thought about the _meta field as well, but I don't know if it would be useful in practice to achieve the custom attribute thing.
Enriching LLM metrics with custom headers today, is mostly about initializing their agents with the headers they want. Something like:

client = AsyncOpenAI(base_url="...", api_key="...", default_headers=CUSTOM_HEADERS)
provider = OpenAIProvider(openai_client=client)

So, for people already using custom headers for LLM, initializing the MCP client using the same configuration looks like something reasonable that will just work with current setups, as the clients are already accepting custom headers.

mcp = MCPServerStreamableHttp(name="Envoy AI Gateway MCP",
        params={"url": mcp_url, "timeout": 300, "headers": CUSTOM_HEADERS},
        cache_tools_list=True,
)

Having the tracing implementation extract these custom attributes from the _meta field would require agent/mcp SDKs to populate that information there, and AFAIK this is done specifically for the tracing information, but not ina. more general fashion today.

codefromthecrypt

@nacx it is ok to go forward with this, my main point was to highlight that it is solving a different scope than why that feature was added (session.id). That said, the issue you are mentioning is at a higher scope "user.region" which this would solve.

To do session.id properly we need to extract attributes from the json-rpc messages, and that can happen later. I don't think you can predict this will never happen because already in order to try new MCP spec additions people put things in meta.

Anyway, since this change is about "X-Tracing-Enrichment-User-Region"-> "user.region" I recommend using exactly those values in your unit tests. Much better than arbitrary and explains the motivation!

codefromthecrypt · 2025-10-13T08:49:42Z

opened #1348 so we don't have to go around on this one. It is ok to skip if you don't feel it is straightforward to implement and wait for someone to complain. Just it takes several months sometimes for an aigw release. Meanwhile any late request can use the docker images built from main branch anyway!

nacx · 2025-10-13T09:56:58Z

@codefromthecrypt in 1420dbb I'm reading custom attributes from the metadata as well, those taking precedence over the HTTP header ones. WDYT?

nacx · 2025-10-13T10:48:48Z

/retest

codefromthecrypt · 2025-10-13T13:20:10Z

internal/tracing/mcp.go

-		if headerValue, ok := headers[headerName]; ok {
-			attrs = append(attrs, attribute.String(attrName, headerValue))
+	for srcName, targetName := range m.attributeMappings {
+		// Check if the attribute is present in the metadata first, as this is the common place to add custom attributes


great idea!

codefromthecrypt · 2025-10-13T13:20:57Z

fyi I am working on goose to do propagation at the moment and will probably try to get it to propagate its session id when done

mathetake · 2025-10-13T22:36:34Z

is this ready or on hold?

nacx · 2025-10-14T10:30:11Z

This is ready! :)
With the latest changes, it also tackles #1348

mathetake · 2025-10-14T15:49:07Z

internal/mcpproxy/handlers.go

+	headerMap := make(map[string]string, len(headers))
+	for header := range headers {
+		headerMap[header] = headers.Get(header)
+	}
+


non blocking nit comment (as i think the impact is negligible): maybe skip this header copying when the header-to-attribute thingy is not configured at the startup time

codefromthecrypt · 2025-10-14T16:05:33Z

internal/tracing/mcp.go

+		// Check if the attribute is present in the metadata first, as this is the common place to add custom attributes
+		// in MCP requests. Fall back to headers if not found in metadata.
+		// If the attribute is not found there, check if there is any custom header to map.
+		if metaValue, ok := param.GetMeta()[srcName]; ok {


we need to make sure that we do downcase based comparison as unlike http these are case sensitive, but we should clobber anyway.

or any other normal way to do case insensitive map (probably need to scan and decide to pick first or last even though maps aren't ordered in go..)

@nacx could you address this adrian's comment

codefromthecrypt · 2025-10-14T16:57:42Z

fyi for testing block/goose#5165

Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>

nacx · 2025-10-14T23:30:44Z

/retest

…y#1342) **Description** The LLM metrics and trace providers allow for the configuration of custom headers to be included as metric attributes and span attributes. The MCP implementation, however, did not honor those values. This PR updates the MCP metrics and tracer configurations to also take this configuration into account. * The tracer method to create a span now receives the request headers. * To avoid having to propagate request headers all the way down inside the MCP Proxy so that they can be taken into account when recording metrics, I've opted to instantiate an MCPProxy on each request, and have it set up with the request attributes from the beginning. This allows reusing the metrics implementation in a clean way. **Related Issues/PRs (if applicable)** Fixes envoyproxy#1335 Fixes envoyproxy#1348 **Special notes for reviewers (if applicable)** N/A --------- Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>

…y#1342) **Description** The LLM metrics and trace providers allow for the configuration of custom headers to be included as metric attributes and span attributes. The MCP implementation, however, did not honor those values. This PR updates the MCP metrics and tracer configurations to also take this configuration into account. * The tracer method to create a span now receives the request headers. * To avoid having to propagate request headers all the way down inside the MCP Proxy so that they can be taken into account when recording metrics, I've opted to instantiate an MCPProxy on each request, and have it set up with the request attributes from the beginning. This allows reusing the metrics implementation in a clean way. **Related Issues/PRs (if applicable)** Fixes envoyproxy#1335 Fixes envoyproxy#1348 **Special notes for reviewers (if applicable)** N/A --------- Signed-off-by: Ignasi Barrera <ignasi@tetrate.io> Signed-off-by: Hrushikesh Patil <hrushikesh.patil@nutanix.com>

…y#1342) **Description** The LLM metrics and trace providers allow for the configuration of custom headers to be included as metric attributes and span attributes. The MCP implementation, however, did not honor those values. This PR updates the MCP metrics and tracer configurations to also take this configuration into account. * The tracer method to create a span now receives the request headers. * To avoid having to propagate request headers all the way down inside the MCP Proxy so that they can be taken into account when recording metrics, I've opted to instantiate an MCPProxy on each request, and have it set up with the request attributes from the beginning. This allows reusing the metrics implementation in a clean way. **Related Issues/PRs (if applicable)** Fixes envoyproxy#1335 Fixes envoyproxy#1348 **Special notes for reviewers (if applicable)** N/A --------- Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>

nacx changed the title ~~mpc: add configured header attribtues to metrics and spans~~ mcp: add configured header attribtues to metrics and spans Oct 12, 2025

nacx marked this pull request as ready for review October 12, 2025 15:59

nacx requested a review from a team as a code owner October 12, 2025 15:59

codefromthecrypt reviewed Oct 13, 2025

View reviewed changes

codefromthecrypt approved these changes Oct 13, 2025

View reviewed changes

codefromthecrypt mentioned this pull request Oct 13, 2025

MCP request.params._meta support for #1348

Closed

codefromthecrypt reviewed Oct 13, 2025

View reviewed changes

mathetake reviewed Oct 14, 2025

View reviewed changes

mathetake approved these changes Oct 14, 2025

View reviewed changes

codefromthecrypt reviewed Oct 14, 2025

View reviewed changes

mathetake enabled auto-merge (squash) October 14, 2025 16:16

mathetake disabled auto-merge October 14, 2025 17:35

nacx and others added 6 commits October 14, 2025 23:09

mpc: add configured header attribtues to metrics and spans

0510a61

Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>

rename variables to attribute

d870aaa

Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>

propagate custom metric and trace attributes from _meta

5c53798

Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>

add test sfor attribute precedence

7f35ed4

Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>

drift

8ae4f39

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

address review comments

3b448be

Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>

nacx force-pushed the custom-obs branch from dc6f6fd to 3b448be Compare October 14, 2025 22:11

case-insensitive params in metrics as well

296fa40

Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>

mathetake approved these changes Oct 14, 2025

View reviewed changes

mathetake merged commit 220b90c into envoyproxy:main Oct 14, 2025
49 of 51 checks passed

Uh oh!

mcp: add configured header attribtues to metrics and spans #1342

mcp: add configured header attribtues to metrics and spans #1342

Uh oh!

Conversation

nacx commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codefromthecrypt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nacx Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codefromthecrypt commented Oct 13, 2025

Uh oh!

nacx commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codefromthecrypt left a comment

Choose a reason for hiding this comment

Uh oh!

codefromthecrypt commented Oct 13, 2025

Uh oh!

nacx commented Oct 13, 2025

Uh oh!

nacx commented Oct 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codefromthecrypt commented Oct 13, 2025

Uh oh!

mathetake commented Oct 13, 2025

Uh oh!

nacx commented Oct 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codefromthecrypt commented Oct 14, 2025

Uh oh!

nacx commented Oct 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nacx commented Oct 12, 2025 •

edited

Loading

codecov-commenter commented Oct 12, 2025 •

edited

Loading

nacx Oct 13, 2025 •

edited

Loading

nacx commented Oct 13, 2025 •

edited

Loading