advice for dealing with tracing where the export endpoint might not be present

[rbtcollins]

So, when the export endpoint isn't present, which happens e.g:

• during development, if e.g. jaeger isn't running
• during testing - e.g. CI - if no tracing stack is present

then running tracing enabled code results in:

OpenTelemetry trace error occurred. Exporter otlp encountered the following error(s): the grpc server returns error (The service is currently unavailable): , detailed error message: error trying to connect: tcp connect error: No connection could be made because the target machine actively refused it. (os error 10061)  

when the tracer provider is shutdown (which is desirable in general to ensure that the spans for short tools are exported), (or for longer tasks/servers, when the internal buffer is filled, this happens without the shutdown being invoked at all.

which is understandable - the trace can't be exported, and certainly in prod a case can be made that failing silently is wrong.

However, I'm finding - in multiple contexts: at work in our microservices, in rustup when instrumenting its internals, and in other personal projects - that this works quite poorly. We end up with terabytes of logs messages, saturating our logging system when the telemetry export stack is down, or unit tests fail because extra text has been written to std channels.

Does anyone else run into this problem? Is there some feature to disable the error reporting, or replace it with a metric that can be used to detect the situation instead of spamming log output?

[cijothomas]

Can you disable/not-add OTLP Exporters in development env, right?