wsgi daemon mode

[thatcher]

I run my stateless flask apps with mod_wsgi/apache using daemon mode like:

WSGIDaemonProcess foo-services python-home=/opt/my_org/foo-services/_env processes=8 threads=48 maximum-requests=10000 display-name=%{GROUP}
WSGIApplicationGroup %{GLOBAL}
WSGISocketPrefix /var/run/wsgi


Alias /image-services "/opt/my_org/foo-services/wsgi.py"
<Location "/for-services">
SetHandler wsgi-script
Options +ExecCGI
FileETag None
ExpiresActive On
ExpiresDefault "access plus 1 year"
WSGIProcessGroup image-services
</Location>

Which means when a request gets to the service it could be hitting 1 of 8 daemon processes each of which have their own memory in isolation of the others. Does the metrics endpoint store the prometheus data in a way that is shared across these daemons?

I can create some tests to verify if thats the case or not, just curious if the answer is already known.

Thanks,
Thatcher

[rycus86]

Hi Thatcher,

I think I've seen both thread and process based aggregation in the official Prometheus Python library that's used here, so I think it might work, but I'll have to check.
If you beat me to finding the answer, it would be great if you could share it on this issue! :)

Thanks,
Viktor

[thatcher]

I can definitely verify i can start my micro-service, run a handful of request, and when i continue to refresh my metrics endpoints i get different responses from each daemon, which makes sense.

It looks like promethues_client added support for a multi process collectors as in the examples provided here:

prometheus/client_python#122

I think we could patch this by looking for if os.environ.get('prometheus_multiproc_dir') at configuration time and instead of using the default registry do something like:

from prometheus_client import multiprocess
from prometheus_client import generate_latest, CollectorRegistry

...
registry = DEFAULT_REGISTRY
if os.environ.get('prometheus_multiproc_dir'):
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)

...

@app.route('/metrics')
def metrics():
    data = generate_latest(registry)
    return data

It looks like I can test this out just by passing my own registry to PrometheusMetrics. Ill let you know.

[thatcher]

Yup it worked! heres the gist:

wsgy.py

import os
...
os.environ["prometheus_multiproc_dir"] = "/tmp/my_app.stats"
...
from my_app import app as application

in my flask app:

import os
...
from prometheus_client import multiprocess
from prometheus_client import CollectorRegistry
from prometheus_flask_exporter import PrometheusMetrics
from prometheus_flask_exporter import DEFAULT_REGISTRY
...
app = Flask(__name__)

registry = DEFAULT_REGISTRY
if os.environ.get('prometheus_multiproc_dir'):
    stats_dir = os.environ.get('prometheus_multiproc_dir')
    if not os.path.exists(stats_dir):
        os.makedirs(stats_dir)
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)

metrics = PrometheusMetrics(app, registry=registry)

Thats it! You could add this as an introspected feature or just document the recipe for others.

Thanks rycus86!

Thatcher

[rycus86]

Awesome, thanks a lot!
I'll give it a go to see if I can add this as a configurable feature.
If that fails, I'll add this info to README or some docs.
Very useful to know, thanks for the investigation!

[snegivulcan]

@rycus86 & @thatcher

I was playing with this library and even with a single instance, when i hit the /metrics endpoint i get 3 different results.

I am following the steps outlined in the conversation here.

registry = CollectorRegistry()
if not os.path.exists(config.PROMETHEUS_MULTIPROC_DIR):
    os.makedirs(config.PROMETHEUS_MULTIPROC_DIR)
multiprocess.MultiProcessCollector(registry, path=config.PROMETHEUS_MULTIPROC_DIR)
metrics = PrometheusMetrics(app, registry=registry)

@app.route('/metrics')
def metrics():
    data = generate_latest(registry)
    return data

So why "3" different responses ? And how to resolve the issue ? I am using 0.2.2 version of prometheus_flask_exporter

[thatcher]

@snegivulcan I discovered the same issue and realized the issue is related to some of the code in the prometheus_flask_exporter and I didnt have time to dig into it. I'd like to eventually figure out how to get it to play nicely with prometheus_flask_exporter. I can confirm that the issue does not occur if I use the prometheus client directly. Here is how i am using it for now in production:

from flask import Flask
from flask import request
from prometheus_client import CONTENT_TYPE_LATEST
from prometheus_client import CollectorRegistry
from prometheus_client import Counter
from prometheus_client import Histogram
from prometheus_client import generate_latest
from prometheus_client import multiprocess

app = Flask(__name__)

stats_dir = os.environ.get('prometheus_multiproc_dir')
if not os.path.exists(stats_dir):
    os.makedirs(stats_dir)
registry = CollectorRegistry()
multiprocess.MultiProcessCollector(registry)
histogram = Histogram(
    'flask_http_request_duration_seconds',
    'Flask HTTP request duration in seconds',
    ('method', 'endpoint', 'status'),
)
counter = Counter(
    'flask_http_request_total',
    'Total number of HTTP requests',
    ('method', 'status')
)

def before_request():
    request.start_time = default_timer()


def after_request(response):
    total_time = max(default_timer() - request.start_time, 0)
    histogram.labels(
        request.method,
        request.endpoint,
        response.status_code
    ).observe(total_time)

    counter.labels(request.method, response.status_code).inc()

    return response


app.before_request(before_request)
app.after_request(after_request)


@app.route('/metrics')
def metrics():
    headers = {'Content-Type': CONTENT_TYPE_LATEST}
    return generate_latest(registry), 200, headers

[rycus86]

Hi @snegivulcan and @thatcher ,

Have you had a look at the small example in https://github.com/rycus86/prometheus_flask_exporter/tree/master/examples/wsgi ?
I don't have much experience with running wsgi apps, but the results seem to have indicated it's working as intented.
Please let me know if that's not the case and I'll try to have another look.

Thanks!

[jpds]

I'm using the same code as @snegivulcan from his Flask example and I noticed from the comment headers that what's being exposed relates to one of the workers and the other belongs to the multiprocess metrics:

# HELP flask_http_request_duration_seconds Flask HTTP request duration in seconds
# TYPE flask_http_request_duration_seconds histogram
flask_http_request_duration_seconds_bucket{le="0.005",method="GET",path="/_healthcheck/",status="200"} 17.0
flask_http_request_duration_seconds_bucket{le="0.01",method="GET",path="/_healthcheck/",status="200"} 17.0
flask_http_request_duration_seconds_bucket{le="0.025",method="GET",path="/_healthcheck/",status="200"} 18.0
flask_http_request_duration_seconds_bucket{le="0.05",method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_bucket{le="0.075",method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_bucket{le="0.1",method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_bucket{le="0.25",method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_bucket{le="0.5",method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_bucket{le="0.75",method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_bucket{le="1.0",method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_bucket{le="2.5",method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_bucket{le="5.0",method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_bucket{le="7.5",method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_bucket{le="10.0",method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_bucket{le="+Inf",method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_count{method="GET",path="/_healthcheck/",status="200"} 19.0
flask_http_request_duration_seconds_sum{method="GET",path="/_healthcheck/",status="200"} 0.07988429069519043
# HELP flask_http_request_duration_seconds Multiprocess metric
# TYPE flask_http_request_duration_seconds histogram
flask_http_request_duration_seconds_bucket{le="0.01",method="GET",path="/_healthcheck/",status="200"} 67.0
flask_http_request_duration_seconds_bucket{le="0.1",method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_bucket{le="5.0",method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_count{method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_bucket{le="0.075",method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_bucket{le="1.0",method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_bucket{le="0.5",method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_sum{method="GET",path="/_healthcheck/",status="200"} 0.375255823135376
flask_http_request_duration_seconds_bucket{le="0.25",method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_bucket{le="0.75",method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_bucket{le="+Inf",method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_bucket{le="2.5",method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_bucket{le="0.05",method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_bucket{le="7.5",method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_bucket{le="10.0",method="GET",path="/_healthcheck/",status="200"} 80.0
flask_http_request_duration_seconds_bucket{le="0.005",method="GET",path="/_healthcheck/",status="200"} 63.0
flask_http_request_duration_seconds_bucket{le="0.025",method="GET",path="/_healthcheck/",status="200"} 76.0

[jpds]

I found a fix from:

https://github.com/jonashaag/prometheus-multiprocessing-example/blob/master/yourapp.py

Where I define the following as a /metrics endpoint:

    @app.route('/metrics')
    def metrics():
        registry = CollectorRegistry()
        multiprocess.MultiProcessCollector(registry)
        data = generate_latest(registry)
        return Response(data, mimetype=CONTENT_TYPE_LATEST)

[rycus86]

Hi @jpds,

Thanks for sharing your findings! The project you mentioned also links the multiprocessing section of the official Prometheus Python client library: https://github.com/prometheus/client_python#multiprocess-mode-gunicorn

I'll try to have a look if this could be supported somehow better with this library.

Thanks!

[rycus86]

Hi,

There's a new release with some multiprocessing changes from @elephantum - https://pypi.org/project/prometheus-flask-exporter/0.2.3/

It would be great if you could give it a look if the previous problem still happens using this version.

Thanks!

[rycus86]

I've added some more multiprocessing changes to version 0.5.0, mainly targeted for Gunicorn and uWSGI, but should work in a generic way - see the README for more info.