Support for different metric prefixes in StackdriverMeterRegistry

[rafal-dudek]

Please describe the feature request.
Add support for different prefixes in StackdriverMeterRegistry. Currently it is hardcoded with "custom.googleapis.com/" here:
https://github.com/micrometer-metrics/micrometer/blob/main/implementations/micrometer-registry-stackdriver/src/main/java/io/micrometer/stackdriver/StackdriverMeterRegistry.java#L429

It could be either property with any value provided by user (with default to "custom.googleapis.com/") or just property to switch from the old prefix to the new one - "external.googleapis.com/user/".

Rationale
The new prefix, "external.googleapis.com/user", become available for custom metrics: https://cloud.google.com/monitoring/custom-metrics#identifier
It is recommended especially when using same metrics in different project as it has some optimization and amenities for that case:
https://cloud.google.com/monitoring/api/metrics_other#user

[shakuzen]

I would generally prefer to have less configuration options for users to potentially configure things incorrectly or suboptimally if there isn't a good reason. Is there a reason we shouldn't change the hard-coded prefix to external.googleapis.com/user/ rather than make it configurable?

Also, we default to the global resource type currently, but I see the documentation you linked mentions:

Cloud Monitoring treats external metrics the same as custom metrics, with one exception. For external metrics, a resource_type of global is invalid and results in the metric data being discarded.

We could switch the default to generic_task but that has some required resource labels, I think. https://cloud.google.com/monitoring/api/resources#tag_generic_task

[rafal-dudek]

The basic problem is the need to rework alerts and dashboards when you change the metric prefix.
It is not a hard change but it could stop some application from upgrading micrometer (and e.g. spring-boot-starter-parent with dependencies) until they have time for such update. What's worse, the unnoticed change of the prefix with upgrade will cause the problem with reading metrics by not aware users. In addition, alerts will stop monitoring the application, which will prevent incidents from triggering.

About resource type. Changing the default type global could be problematic because of the labels. Some of the labels has values validation so cannot be e.g. empty when there are not relevant (location is verifies if is proper region, at least for k8s_container type).
We are using k8s_container in our applications so the change of default type is not affecting us. We are using additionally MonitoredResourceUtil from https://github.com/googleapis/java-logging/tree/main to get the values for labels when running in GKE and we are adding some empty values when running locally.

Due to these 2 problems, changing the default prefix and recourse type could be problematic and easier to just make it optional.

[shakuzen]

The basic problem is the need to rework alerts and dashboards when you change the metric prefix.

That's a very good point. We don't want to break things for users or create unnecessary friction in upgrading versions.

Thanks for confirming about the resource type.

Seeing as the prefix has multiple valid options and different users may have different preferences/needs, it probably makes sense for it to be arbitrarily configurable. It's a shame we probably can't do much validation for users, but it doesn't seem like there is a fixed set of valid values - after all the mentioned values didn't used to be valid.

Would you like to contribute the change to allow configuring the prefix?