Skip to content

feat(spanner): support user-provided OpenTelemetry for client metrics export - #13741

Merged
rahul2393 merged 2 commits into
mainfrom
feat/spanner-custom-otel-metrics
Jul 30, 2026
Merged

feat(spanner): support user-provided OpenTelemetry for client metrics export#13741
rahul2393 merged 2 commits into
mainfrom
feat/spanner-custom-otel-metrics

Conversation

@rahul2393

Copy link
Copy Markdown
Contributor

Summary

Adds Client Metrics: a new, opt-in feature that exports Spanner's client-side metrics to a caller-provided OpenTelemetry pipeline (OTLP, Prometheus, any exporter), including on Spanner Omni where the existing Cloud Monitoring metrics are unavailable.

Fully opt-in and fully decoupled from the existing built-in (Cloud Monitoring) metrics. Default behavior is unchanged: without a client-metrics provider, nothing new happens, and built-in metrics continue to export to Cloud Monitoring exactly as before.

Scope note: This change was split out of a larger PR into two focused PRs. This PR contains only the Client Metrics export feature. The optional endpoint metric attribute for location-aware routing is a separate follow-up: #13740.

Two independent features

  • Built-in metrics (existing, unchanged): export to Google Cloud Monitoring. Controlled by setBuiltInMetricsEnabled and the SPANNER_DISABLE_BUILTIN_METRICS environment variable, exactly as today. Not available on Spanner Omni.
  • Client metrics (new): export the client instruments to a caller-owned OpenTelemetry. Controlled solely by setClientMetricsProvider(...) — a CustomOpenTelemetryMetricsProvider turns it on; NoopMetricsProvider (or no provider) turns it off. Works on all instance types, Omni included.

The two are decoupled: setBuiltInMetricsEnabled and the env var affect only the built-in Cloud Monitoring sink and have no effect on client metrics; the client-metrics provider affects only the caller-owned sink and has no effect on built-in metrics. Under the hood these are the same client instruments, exported under the distinct spanner/client namespace — the difference is the export path, not the metrics.

Emulator handling

Client metrics are not recorded when Spanner is pointed at the emulator. To make that reliable when the emulator is configured programmatically (via setEmulatorHost(...)) and not only through the SPANNER_EMULATOR_HOST environment variable, this PR broadens emulator detection: SpannerOptions.isEmulatorEnabled() now also recognizes the builder-configured emulator host, and the connection-check error message is generalized to describe both the environment-variable and programmatic configuration paths. This gates both the client and Cloud Monitoring metrics sinks off against the emulator using a single detection predicate. Called out explicitly here for reviewer visibility.

Motivation

Client metrics currently export only to Cloud Monitoring, which is unavailable on Spanner Omni. Customers running on Omni (or who standardize on their own observability stack) had no way to receive these metrics. This lets them route the metrics to any OpenTelemetry exporter, independently of the built-in Cloud Monitoring configuration.

API

SdkMeterProviderBuilder meterProviderBuilder = SdkMeterProvider.builder();
SpannerMetrics.configureMeterProviderBuilder(meterProviderBuilder);
// ... attach your exporter to meterProviderBuilder ...
OpenTelemetry otel =
    OpenTelemetrySdk.builder().setMeterProvider(meterProviderBuilder.build()).build();

SpannerOptions options =
    SpannerOptions.newBuilder()
        .setClientMetricsProvider(new CustomOpenTelemetryMetricsProvider(otel))
        .build();

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for exporting Cloud Spanner client metrics to a caller-owned OpenTelemetry destination using a new MetricsProvider API, allowing users to configure custom pipelines independently of the default Google Cloud Monitoring export. It also updates documentation, adds comprehensive integration tests, and integrates routed endpoint attributes for location-aware routing. The review feedback highlights a bug in the newly added isEmulatorEnabled() method, where a mismatched protocol prefix (e.g., http://) between the configured host and the emulator host can cause the emulator detection to incorrectly return false.

Comment on lines +3052 to +3057
public boolean isEmulatorEnabled() {
return getChannelProvider() == null
&& emulatorHost != null
&& getHost() != null
&& getHost().equals(emulatorHost);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The isEmulatorEnabled() method checks getHost().equals(emulatorHost). However, when setEmulatorHost is called with a value like "localhost:1234", the builder prepends "http://" to this.host (making it "http://localhost:1234"), but builder.emulatorHost remains "localhost:1234". This causes getHost().equals(emulatorHost) to return false because of the mismatched "http://" prefix.

To make this robust and handle both cases (with or without the protocol prefix), consider normalizing the comparison or checking if getHost() ends with emulatorHost or starts with it after stripping the protocol. Note that a null check on getHost() is unnecessary here as the host would have already been validated.

  @InternalApi
  public boolean isEmulatorEnabled() {
    if (getChannelProvider() != null || emulatorHost == null) {
      return false;
    }
    String normalizedHost = getHost().startsWith("http://") ? getHost().substring(7) : getHost();
    String normalizedEmulatorHost = emulatorHost.startsWith("http://") ? emulatorHost.substring(7) : emulatorHost;
    return normalizedHost.equals(normalizedEmulatorHost);
  }
References
  1. A null check on a parsed URL component (like the host) is unnecessary if other parts of the constructor would have already failed with a malformed URL (e.g., one missing a scheme).

… export

Adds the MetricsProvider family (Default/Noop/CustomOpenTelemetry),
SpannerOptions.Builder#setClientMetricsProvider, and
SpannerMetrics#configureMeterProviderBuilder so client metrics can be
recorded on a caller-owned OpenTelemetry in the spanner/client namespace,
decoupled from the Cloud Monitoring built-in metrics flag. Also broadens
the emulator gate (SpannerOptions#isEmulatorEnabled, enablegRPCMetrics
overload) so no client metrics are recorded against the emulator.
The emulator gate normalizes the http(s) scheme away before comparing
the configured host with the emulator host.
@rahul2393
rahul2393 force-pushed the feat/spanner-custom-otel-metrics branch from 818893b to 0d0ab92 Compare July 14, 2026 08:36
@rahul2393 rahul2393 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 30, 2026
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 30, 2026
@rahul2393
rahul2393 requested a review from sakthivelmanii July 30, 2026 04:49
@rahul2393
rahul2393 merged commit da74dee into main Jul 30, 2026
210 checks passed
@rahul2393
rahul2393 deleted the feat/spanner-custom-otel-metrics branch July 30, 2026 15:12
rahul2393 added a commit to googleapis/google-cloud-go that referenced this pull request Aug 24, 2026
lqiu96 pushed a commit that referenced this pull request Aug 24, 2026
🤖 I have created a release *beep* *boop*
---


<details><summary>1.90.0</summary>

##
[1.90.0](v1.89.0...v1.90.0)
(2026-08-24)


### Features

* **bigquery-jdbc:** implement TypeRegistry and TypeDescriptor
([#13947](#13947))
([0557e69](0557e69))
* **bigquery:** add QueryResultsFormat and ArrowSerializationOptions
configurations
([#13942](#13942))
([ff03e19](ff03e19))
* **bigquery:** expose `StatementType` and query execution stats on
`TableResult`
([#14145](#14145))
([7d16de8](7d16de8))
* **bigtable:** enable microsecond timestamps in client
([#14057](#14057))
([57aaf8d](57aaf8d))
* **bigtable:** route single-entry MutateRows through a point-write c…
([#14028](#14028))
([a403703](a403703))
* **datastore:** add support for request tags
([#13732](#13732))
([b1f6186](b1f6186))
* **ftp:** onboard a new library
([#14068](#14068))
([f41b2d9](f41b2d9))
* **gax:** add ResumableUploadCallable and ResumableUploadCallSettings
([#14052](#14052))
([a5e26e8](a5e26e8))
* **google/cloud/biglake/hive/v1:** onboard a new library
([#14130](#14130))
([650c839](650c839))
* **google/maps/mapmanagement/v2:** onboard a new library
([#14131](#14131))
([7d00726](7d00726))
* **spanner:** support user-provided OpenTelemetry for client metrics
export
([#13741](#13741))
([da74dee](da74dee))
* update API sources and regenerate
([#14000](#14000))
([9337a93](9337a93))
* **workloadidentity:** onboard a new library
([#14060](#14060))
([ab226ee](ab226ee))


### Bug Fixes

* add documentation for insertall api that there's no default retry
([#13953](#13953))
([1fdb4f1](1fdb4f1))
* add retry behavior documentation to insertall interface to clarify the
behavior
([#14058](#14058))
([1b8f9e3](1b8f9e3))
* **auth:** fix JSpecify nullability in UserAuthorizer and TokenStore
([#14150](#14150))
([0d5fac0](0d5fac0))
* **auth:** fix remaining nullability in UserAuthorizer and Builder
([#14158](#14158))
([a51bb8d](a51bb8d))
* **auth:** refine JSpecify nullability for ServiceAccountCredentials
and UserCredentials
([#14159](#14159))
([a929250](a929250))
* **bigquery-jdbc:** enable ITOpenTelemetryTest
([#13991](#13991))
([fa6641b](fa6641b))
* **bigquery-jdbc:** pass connection proxy settings to OpenTelemetry
exporters
([#14011](#14011))
([115b9b3](115b9b3))
* **bigquery-jdbc:** session context propagation when session is enabled
([#14161](#14161))
([1e74dda](1e74dda))
* **bigtable:** remove heartbeat miss logging
([#14054](#14054))
([ec17637](ec17637))
* **deps:** update dependency
com.google.apis:google-api-services-bigquery to v2-rev20260731-2.0.0
([#14149](#14149))
([95f6c38](95f6c38))
* **deps:** update dependency com.google.cloud:libraries-bom to v26.86.0
([#14103](#14103))
([cf5697e](cf5697e))
* **gax-httpjson:** reduce Conscrypt fallback error to debug level
([#13962](#13962))
([8236771](8236771))
* **gax-httpjson:** remove unsupported and deprecated PQC named groups
([#14107](#14107))
([7604971](7604971))
* **gax:** register Conscrypt SSLContext SPI classes for GraalVM
reflection
([#14129](#14129))
([73c0243](73c0243))
* **samples:** align native profile junit and surefire versions with
shared config
([#14096](#14096))
([2b84133](2b84133))
* **spanner:** add closeAsync to ReadContext and make transaction
closing non-blocking
([#14076](#14076))
([671f892](671f892))
* **spanner:** scope server-timing metrics per call and guard
interceptor lifecycle callbacks
([#14053](#14053))
([f35c570](f35c570))
* **storage:** use JsonUtils for StorageObject serialization in
resumable writes and read channels
([#13976](#13976))
([d94922f](d94922f))


### Performance Improvements

* **bigquery-jdbc:** eliminate dry run to resolve statement type
([#14156](#14156))
([7109ecd](7109ecd))
* **spanner-jdbc:** cache commonly used query parameter names
([#14036](#14036))
([1eb6aa3](1eb6aa3))
* **spanner-jdbc:** cache JDBC metadata query strings
([#14041](#14041))
([31c628f](31c628f))
* **spanner-jdbc:** cache positional to named param conversion
([#14034](#14034))
([30e031b](30e031b))


### Dependencies

* **gax-httpjson:** upgrade conscrypt-openjdk-uber to 2.6.2
([#14117](#14117))
([2f5481a](2f5481a))
* Update gRPC to v1.82.3
([#13997](#13997))
([a786107](a786107))
* Upgrade gRPC to v1.82.4
([#14088](#14088))
([0c482fe](0c482fe))


### Documentation

* **bigquery-jdbc:** add user guide with connection property and custom
endpoint reference
([#13878](#13878))
([2dde172](2dde172))
* **gax:** update LRO troubleshooting documentation link
([#14108](#14108))
([4c5bbae](4c5bbae))
* **spanner-jdbc:** update connection_properties.md documentation
([#14035](#14035))
([b576fe8](b576fe8))
* **spanner:** update CHANGELOG.md for releases 6.117.0 through 6.120.0
([#13970](#13970))
([2413811](2413811))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants