Metrics exporter does not support Cloud Run

[joao-melo-ingka]

If we attempt to use this with cloud run, it doesn't work:

shadow.com.google.api.gax.rpc.InvalidArgumentException: shadow.io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.
	at shadow.com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
	at shadow.com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
	at shadow.com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
	at shadow.com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
	at shadow.com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67)
	at shadow.com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
	at shadow.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at shadow.com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
	at shadow.com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
	at shadow.com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
	at shadow.io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:574)
	at shadow.io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:544)
	at shadow.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at shadow.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at shadow.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at shadow.com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:541)
	at shadow.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:576)
	at shadow.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
	at shadow.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:757)
	at shadow.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:736)
	at shadow.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at shadow.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: shadow.com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
		at shadow.com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
		at shadow.com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
		at shadow.com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1729)
		at shadow.com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1661)
		at shadow.com.google.cloud.opentelemetry.metric.CloudMetricClientImpl.createTimeSeries(CloudMetricClientImpl.java:40)
		at shadow.com.google.cloud.opentelemetry.metric.InternalMetricExporter.createTimeSeriesBatch(InternalMetricExporter.java:219)
		at shadow.com.google.cloud.opentelemetry.metric.InternalMetricExporter.export(InternalMetricExporter.java:204)
		at shadow.com.google.cloud.opentelemetry.metric.GoogleCloudMetricExporter.export(GoogleCloudMetricExporter.java:90)
		at io.opentelemetry.sdk.metrics.export.PeriodicMetricReader$Scheduled.doRun(PeriodicMetricReader.java:162)
		at io.opentelemetry.sdk.metrics.export.PeriodicMetricReader$Scheduled.run(PeriodicMetricReader.java:148)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
		at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
		... 3 more
Caused by: shadow.io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.; Metrics cannot be written to cloud_run_revision. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.
	at shadow.io.grpc.Status.asRuntimeException(Status.java:539)
	... 14 more

[psx95]

Hi @joao-melo-ingka,
Thank you for bringing this up, this seems to be a bug in our metrics exporter which needs to be fixed.

Root Cause:

The reason for this is that cloud_run_revision is not writable and the metrics exporter currently does not account for this.
By default the metrics exporter uses ResourceDetector to determine the monitored resource, which in your case is cloud_run_revision.

Possible Workaround:

As a workaround for this, in your SdkMeterProvider setup, you can set up the io.opentelemetry.sdk.resources.Resource manually so that it is not auto-detected. For example:

SdkMeterProvider.builder()
                .setResource(Resource.create(Attributes.of(
                        AttributeKey.stringKey("cloud.availability_zone"), "global",
                        AttributeKey.stringKey("service.namespace"), "test-service",
                        AttributeKey.stringKey("service.name"), "service-name",
                        AttributeKey.stringKey("service.instance.id"), "some-id"
                        )))
            .registerMetricReader(
                PeriodicMetricReader.builder(metricExporter)
                    .setInterval(java.time.Duration.ofSeconds(30))
                    .build())
            .build();

The provided list of Attributes in the above example will cause the resource to be detected as generic_task.

Let me know if this approach works for you.

[joao-melo-ingka]

Understood, thanks. Once I do that, I get #247

[punya]

In team discussion, we wondered why this wasn't caught by existing integration tests. It turns out that our integration tests only cover the Trace APIs (and we won't prioritize Metric API coverage any time soon). So for this issue, we should

• Fix the problem in the Java metrics exporter by using generic_task for Cloud Run custom metrics
• Ensure that our metrics exporter spec covers this behavior
• Verify that other language exporters also do the right thing

[psx95]

In team discussion, we wondered why this wasn't caught by existing integration tests. It turns out that our integration tests only cover the Trace APIs (and we won't prioritize Metric API coverage any time soon). So for this issue, we should

• Fix the problem in the Java metrics exporter by using generic_task for Cloud Run custom metrics
• Ensure that our metrics exporter spec covers this behavior
• Verify that other language exporters also do the right thing

For the other language exporters - Python & Go seem compliant. JavaScript needs to fix this same issue. A bug has been opened against JS repo to track this - GoogleCloudPlatform/opentelemetry-operations-js#553