Deadlock in exposition.py

[ochedru]

We are using prometheus-client 0.5.0.

Our server application regularly stops reporting prometheus metrics. A bunch of threads are stuck with the same stack:

Thread 0x00007fe2a3aba700 (most recent call first):
  File "/usr/lib/python3.6/site-packages/prometheus_client/values.py", line 27 in get
  File "/usr/lib/python3.6/site-packages/prometheus_client/metrics.py", line 546 in _child_samples
  File "/usr/lib/python3.6/site-packages/prometheus_client/metrics.py", line 175 in _samples
  File "/usr/lib/python3.6/site-packages/prometheus_client/metrics.py", line 182 in _multi_samples
  File "/usr/lib/python3.6/site-packages/prometheus_client/metrics.py", line 68 in collect
  File "/usr/lib/python3.6/site-packages/prometheus_client/registry.py", line 75 in collect
  File "/usr/lib/python3.6/site-packages/prometheus_client/exposition.py", line 89 in generate_latest
  File "/usr/lib/python3.6/site-packages/prometheus_client/exposition.py", line 146 in do_GET
  File "/usr/lib/python3.6/http/server.py", line 406 in handle_one_request
  File "/usr/lib/python3.6/http/server.py", line 418 in handle
  File "/usr/lib/python3.6/socketserver.py", line 724 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 364 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 654 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

(We are exposing prometheus metrics with start_http_server()).
This looks like a thread died with the lock acquired.
I searched for something related to GC (#371), but there is no thread GCing at that time.
Any idea? Thank you.

[brian-brazil]

I can't see anything that would cause that offhand. Are there any other stack traces going via the client?

[ochedru]

Additional context: sometimes we are updating prometheus metrics from within forked processes. My understanding is that this could block the forked process (if it gets forked with the lock acquired), not the main process where the prometheus http server is running.

[ochedru]

I can't see anything that would cause that offhand. Are there any other stack traces going via the client?

No, this is always the same stack. This is reproducible, but it takes some time (a few days).

[brian-brazil]

My understanding is that this could block the forked process (if it gets forked with the lock acquired), not the main process where the prometheus http server is running.

Yes, that's correct. Mixing forking and threading is generally unwise in Python. If you're doing this you should probably use multi-processing mode.

[haizaar]

@ochedru Please try v0.6.0 to see if it helps.
I had possibly similar issue with v0.5.0 and Python 3.6.8 where I ended up with 1000 threads for my single-threaded app and metrics collection got stuck after some time (couldn't get the stack trace unfortunately).

Now I deployed the same app with the same Python version and p8s client v0.6.0. So far there are no more runaway threads. Time will tell, but for the last hour it's all good so far.

[ochedru]

@haizaar Thanks for the tip, I will try this.

[brian-brazil]

I'm presuming that worked, if not let us know.