Multiple metric_readers causes deadlock

[ekquasar]

Summary

Setting up multiple metrics readers like this:

mprovider = MeterProvider(resource=resource, metric_readers=[console_metric_reader, otlp_metric_reader])

causes a deadlock.

Caveat

It happens rarely, so this may not be tremendously high priority - but I'm reporting so if someone else hits this, they may add to the body of evidence pointing to the underlying issue.

Environment

Python 3.11.6 on macOS Sonoma v14.1.1

Steps to reproduce

1. Run a simple Flask web server with otel instrumentation:

console_metric_reader = PeriodicExportingMetricReader(ConsoleMetricExporter())
otlp_metric_reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint=OTLP_ENDPOINT))
mprovider = MeterProvider(resource=resource, metric_readers=[console_metric_reader, otlp_metric_reader])
metrics.set_meter_provider(mprovider)
meter = metrics.get_meter("demo-meter")

2. Send requests to the endpoint

while true
  curl localhost:8080
  sleep 1
end

3. Wait for the deadlock

Unfortunately, it is not always reproducible. Sometimes the app must be running for >5 minutes. Sometimes it happens immediately.

Expected behavior

1. Console exporter prints metrics to stdout
2. OTLP exporter errors out print to stdout

What is the actual behavior?

When the deadlock occurs, the traceback looks like this:

...
  File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1123, in join
    self._wait_for_tstate_lock(timeout=max(timeout, 0))
  File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1139, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):

[ocelotl]

If this happens again, please include the complete traceback, do not leave out any part since that information can actually show us the where in our code is the failure coming from.