Allow resetting the global tracer provider

[garry-cairns]

There are circumstances in which I'd like to be able to "reset" the global tracer provider at runtime. For example, there are some features I disallow in my app unless an environment variable is set, and I'd like to be able to update the tracer provider to reflect that change if the variable gets set or unset during a run (a typical use-case here is unit testing).

This is currently prevented in the python implementation, and I believe in others also. In python this is achieved with opentelemetry.util._once.Once.do_once. While I understand why this is generally desirable, I think it would be useful to allow a force override, which could let people delete and re-initialize the global provider.

[MrAlias]

Similarly: Global MeterProvider Get/Set API is under-defined

[jack-berg]

Can you clarify whether you want to:

1. Reconfigure the global tracer provider in place, such that existing instrumentation using the tracer provider receives updated configuration.
2. Update the global tracer provider to be a different instance. Instrumentation that had previously acquired a tracer from the global tracer provider would continue to report to the old instance. New calls to global tracer provider would get the new instance.

If your goal is the first option, the specification for MeterProvider, TracerProvider, and LoggerProvider each have language allowing updating configuration, though details are missing as to how exactly this works.

If your goal is the second option, each language ecosystem should ensure that it has utilities required for testing. In java, for example, we allow the global to be reset for test.

[garry-cairns]

@jack-berg Apologies for the slow response. I wanted option two I think. For example, in Python the BatchSpanProcessor is not fork safe. In an environment where at process start time I don't know whether an application will fork I might want to set up the tracer/meter provider but register fork handlers to shut them down and reinitialize them in the child process(es) in the event of a fork. Right now in the Python implementation Once will prevent that without me doing a bunch of weird stuff. Is it fair to say this might be one to raise in Python rather than at the spec given your response as it relates to option two?

[jack-berg]

Is it fair to say this might be one to raise in Python rather than at the spec given your response as it relates to option two?

If your use case is around testing, then yes, I would raise it with the python community since its reasonable for SDKs to have tools to accommodate testing. If there are use cases outside of testing, I think we need to discuss more and possibly change the specification.

[garry-cairns]

My use-case extends beyond testing yes. Basically I'm working on a large platform using a common entrypoint for a large number of services. There's no way to distinguish between a service that will fork and one that won't, and my ideal solution would be

os.register_at_fork(before=shutdown_telemetry, after_in_child=enable_telemetry)

where the two args there are callables, the latter of which is doing all the stuff we'd expect of instantiating a batch processor and registering an exporter with it and so on. Right now I'm prevented from doing so in Python and, based on what I can tell from the go implementation, elsewhere as well. I'm aware my situation may be rare but I think this would be a good pattern nonetheless.

[jack-berg]

Hmmm.. yes this does seem to be both rare but valid. 🙂

[JohnPaton]

I've just encountered this issue but for meters in a testing use case and basically solved it in a pytest fixture by doing something like

import importlib

import pytest
import opentelemetry.metrics._internal
from opentelemetry.metrics import set_meter_provider


@pytest.fixture
def configure_meter_provider():
    ... # create the provider
    set_meter_provider(provider)

    yield

    # teardown - reset the provider
    importlib.reload(opentelemetry.metrics._internal)


def test_whatever(configure_meter_provider):
    ... # run some tests on emitted metrics

importlib.reload causes these lines to be re-run which resets the global provider.

I would not recommend doing this in production but for tests it should be okay for now. But the fact that this is required suggests some room for improvement here.

[svrnm]

@open-telemetry/technical-committee please take another look, is this something that needs to be addressed?

[yurishkuro]

It is always possible to implement a wrapper provider that can still be set only once but internally allow to swap the delegate if the app requires it. So I don't think this requires a spec change. Having said that, many instrumentation implementations may retrieve Tracer/Meter etc at initialization time, so typically solving this at provider level is not enough.

[jsuereth]

We discussed in the TC:

We will not be adding a reset to global tracer provider as a general specification. There's a lot of issues that this would cause globally across all languages.

We recommend the following:

• All languages should support testing with global providers. Please raise an issue with opentelemetry-python if you have issues.
• Dynamic/changing configuration is already supported in opentelemetry (e.g. jaeger-remote-sampler). We do not want to reset and SDK globally but would entertain specific things to reset with known use cases. Please open new issues if you have such a use case.
• The major concern raised around handling forking is something that needs to be addressed. This issue may also affect, e.g. C++. An issue should be opened directly against opentelemetry-python asking for support against this issue. If specification work is necessary to enable python to fix the forking issue, we can entertain a much smaller/narrow proposal to support forking related issues with SDK initialization.

Closing the current issue as wontfix. Please open a new issue (or retitle + rephrase this one) if you have a more narrow suggestion.