Serve vMCP metrics on a separate diagnostics listener

[amirejaz]

Summary

Virtual MCP registers /metrics on the mux that serves MCP traffic — under a comment
that acknowledged it was unauthenticated. That's the vMCP half of Finding E in #6271;
#6296 moved the proxy half. Until both move, the endpoint still shares the port
deployments route publicly.

• Serve /metrics on the dedicated diagnostics listener (pkg/diagnostics), so access
  can be governed by port. NetworkPolicy matches on pods, ports, and protocols and
  cannot filter on HTTP path, so while /metrics shares the MCP port there is no way
  to express "allow MCP traffic, deny metrics scraping".
• Honour the same migration switch as the proxy path, so this is not a breaking
  change on its own. While metricsOnTransportPort is on (the default), /metrics
  stays reachable on the MCP port too and a deprecation warning names it. Once it is
  off, the application mux 404s rather than letting /metrics fall through to the /
  MCP handler.
• Add Server.DiagnosticsAddress() so the resolved port is discoverable.

Stacked on #6370, which introduces the switch. Review that one first; this PR's
diff is against it, and it will be retargeted to main once #6370 merges.

What this does not do: it does not authenticate, rate limit, or audit /metrics.
The diagnostics listener carries no middleware, and the host is inherited from the MCP
listener (0.0.0.0 under the operator), so the endpoint stays reachable from other
pods. Restricting who can reach that port is what protects it — see the NetworkPolicy
example in docs/observability.md.

Part of #6271

Type of change

• Bug fix
• New feature
• Refactoring (no behavior change)
• Dependency update
• Documentation
• Other (describe):

Test plan

• Unit tests (task test)
• E2E tests (task test-e2e)
• Linting (task lint-fix)

New coverage in pkg/vmcp/server/diagnostics_test.go: no listener without telemetry,
metrics bound to a port distinct from the MCP port, configured port honoured, host
defaulting, idempotent stop. TestServeHandlerRegistersMetricsWhenTelemetryEnabled is
renamed to TestServeHandlerDoesNotServeMetrics and now asserts the 404, since its old
assertion described the behaviour this PR removes.

Three telemetry tests scraped /metrics off the MCP address and now go through
DiagnosticsAddress() — necessary rather than cosmetic, since the listener falls back
to an available port when the configured one is taken, so the address cannot be
constructed.

Remaining local failures (TestValidateOCIRegistryHost, TestParseGitReference_*) and
lint findings (cmd/thv/app/upgrade.go, pkg/vmcp/config/crd_cli_roundtrip_test.go)
are pre-existing on main in files this PR does not touch.

Does this introduce a user-facing change?

Not on merge. /metrics becomes available on the diagnostics port
(prometheusPort, default 9464) while remaining on the MCP port, so existing
scrape configurations keep working until the window closes.

When it does close, this is the half that matters more.

The blast radius is larger here than for the proxy half. Nothing shipped enables the
metrics path for MCPServer, but three vMCP artifacts do — examples/vmcp-config.yaml,
docs/operator/virtualmcpserver-api.md, and
docs/operator/virtualmcpserver-observability.md — so more deployments plausibly have
it on. The example and the observability guide are updated to point at the diagnostics
port.

This should ship in the same release as #6296 and #6370, so operators get one
migration event rather than the same feature moving twice. None of it has shipped —
v0.44.0 was cut before #6296 merged.

Special notes for reviewers

DiagnosticsAddress() is worth a look beyond vMCP: the resolved diagnostics port
currently only appears in a startup log, which is why the E2E metrics helpers in #6296
hardcode 9464 (Copilot flagged this, and those threads are still open). This accessor
is the vMCP-side answer; the equivalent for the proxy path — surfacing it in workload
status — is still outstanding.

TestServeHandlerMetricsOnTransportPort pins both ends of the migration — default and
explicitly-on serve on the MCP port, opted-out 404s — so neither the window nor the
cutover can regress unnoticed.

Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 77.04918% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.04%. Comparing base (983c114) to head (c8ff688).

Files with missing lines	Patch %	Lines
pkg/vmcp/server/diagnostics.go	76.74%	5 Missing and 5 partials ⚠️
pkg/vmcp/server/server.go	50.00%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@                          Coverage Diff                           @@
##           metrics-transport-port-deprecation    #6368      +/-   ##
======================================================================
+ Coverage                               72.99%   73.04%   +0.04%     
======================================================================
  Files                                     744      745       +1     
  Lines                                   78623    78676      +53     
======================================================================
+ Hits                                    57393    57471      +78     
+ Misses                                  17230    17180      -50     
- Partials                                 4000     4025      +25     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.