Warning: Silently ignoring app.run() because the application is run from the flask command line executable

[raxod502]

An awkward situation. Code:

metrics = prometheus_flask_exporter.PrometheusMetrics(app)
metrics.start_http_server(8081, host=config.METRICS_HOST)

Result:

website_1   |  * Serving Flask app 'src.app:app' (lazy loading)
website_1   |  * Environment: development
website_1   |  * Debug mode: on
website_1   | /usr/local/lib/python3.8/dist-packages/prometheus_flask_exporter/__init__.py:322: Warning: Silently ignoring app.run() because the application is run from the flask command line executable. Consider putting app.run() behind an if __name__ == "__main__" guard to silence this warning.

And the metrics server does not listen on port 8081. It seems that Flask notices that I have invoked the main application using flask run, and has elected to disable the ability to run new Flask applications, which obviously breaks start_http_server from prometheus_flask_exporter.

Not sure what the best way to handle this is, but wanted to flag.

[rycus86]

Hey, thanks for flagging this!
Do you have a more complete (but minimal) example I could turn into an integration test to ensure this works as expected?

[raxod502]

Sure, I will put something together and get back to you soon.

[raxod502]

Here's a working example: https://paste.sr.ht/~raxod502/e7527b5868cdee04160f457870299ff33ce0c1ed

% poetry run flask run             
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000 (Press CTRL+C to quit)
/home/raxod502/.cache/pypoetry/virtualenvs/prometheus-flask-exporter-issue-135-_O3CvxEA-py3.10/lib/python3.10/site-packages/prometheus_flask_exporter/__init__.py:322: Warning: Silently ignoring app.run() because the application is run from the flask command line executable. Consider putting app.run() behind an if __name__ == "__main__" guard to silence this warning.
  app.run(host=host, port=port)

[raxod502]

Now that I've had a moment to review the source, I can confirm that simply adding this code before invoking metrics.start_http_server works around the problem, and allows the metrics server to be started without issue:

os.environ.pop("FLASK_RUN_FROM_CLI")

[raxod502]

However, when running in development mode (with the live reloader) we encounter an additional problem, because the metrics Flask app tries to start its own live reloader from inside the first live reloader, which does not work since the live reloader establishes signal handlers and therefore can only run from the main thread:

% FLASK_ENV=development poetry run flask run
 * Environment: development
 * Debug mode: on
 * Serving Flask app 'prometheus-flask-exporter-8081' (lazy loading)
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:8081 (Press CTRL+C to quit)
 * Restarting with stat
Exception in thread Thread-1 (run_app):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "/home/raxod502/.cache/pypoetry/virtualenvs/prometheus-flask-exporter-issue-135-_O3CvxEA-py3.10/lib/python3.10/site-packages/prometheus_flask_exporter/__init__.py", line 322, in run_app
    app.run(host=host, port=port)
  File "/home/raxod502/.cache/pypoetry/virtualenvs/prometheus-flask-exporter-issue-135-_O3CvxEA-py3.10/lib/python3.10/site-packages/flask/app.py", line 920, in run
    run_simple(t.cast(str, host), port, self, **options)
  File "/home/raxod502/.cache/pypoetry/virtualenvs/prometheus-flask-exporter-issue-135-_O3CvxEA-py3.10/lib/python3.10/site-packages/werkzeug/serving.py", line 1083, in run_simple
    run_with_reloader(
  File "/home/raxod502/.cache/pypoetry/virtualenvs/prometheus-flask-exporter-issue-135-_O3CvxEA-py3.10/lib/python3.10/site-packages/werkzeug/_reloader.py", line 427, in run_with_reloader
    signal.signal(signal.SIGTERM, lambda *args: sys.exit(0))
  File "/usr/lib/python3.10/signal.py", line 56, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread of the main interpreter
 * Debugger is active!
 * Debugger PIN: 788-973-912

[raxod502]

It appears to be possible to work around this issue by unsetting FLASK_ENV from Python as well, before starting the metrics server. However, this does have the disadvantage of producing relatively confusing log output:

% FLASK_ENV=development poetry run flask run
 * Environment: development
 * Debug mode: on
 * Serving Flask app 'prometheus-flask-exporter-8081' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:8081 (Press CTRL+C to quit)
 * Running on http://127.0.0.1:5000 (Press CTRL+C to quit)
 * Restarting with stat

Working code:

import os

import flask
import prometheus_flask_exporter

app = flask.Flask(__name__)

os.environ.pop("FLASK_RUN_FROM_CLI")
os.environ.pop("FLASK_ENV")

metrics = prometheus_flask_exporter.PrometheusMetrics(app)
metrics.start_http_server(8081, host="127.0.0.1")

[rycus86]

Thanks a lot for the investigation @raxod502 ! 🙇
Do you have a proposed way forward based on that? I expect if we unset those environment variables then we'd interfere with other parts of Flask that would handle those?

[raxod502]

Yeah... and environment variables are shared globally for the whole process, so messing with them is really nasty. Let's file an issue upstream: pallets/flask#4616

If a keyword argument is added to app.run() as I suggest in that issue report, then we can set that keyword argument from prometheus_flask_exporter, and if we additionally set debug=False in prometheus_flask_exporter, that should fully resolve this problem.

[davidism]

This metrics server does not appear to be part of the application under development. It is its own, separate application. Therefore, it should be run using a production server, even locally. Production servers include Gunicorn, Waitress, uWSGI, mod_wsgi, Eventlet, and Gevent.

[raxod502]

OK, sounds like the way to resolve this issue would be to modify prometheus_flask_exporter to use Gunicorn or an equivalent framework by default, since the official upstream recommendation is that prometheus_flask_exporter is not an appropriate use case for the Flask development server. Does that seem like an appropriate course of action to you, @rycus86?

[rycus86]

I'll have a look at this more deeply when I get a chance. I wouldn't necessarily want Gunicorn or other dependencies to be mandatory, so I'll need to dig deeper into what am I doing wrong in the code that surfaces this problem (sounds like maybe I use the wrong app somewhere perhaps).

[rycus86]

OK, so I had a chance now to look at the code.
So flask run passes an environment variable to the app so that we don't actually start an app from within, because the cli machinery is going to set that development server up for us.

If this is intended for development mode only, I think it should be OK to just have the metrics endpoint exposed on the main Flask app, which happens by default.

import flask
import prometheus_flask_exporter

app = flask.Flask(__name__)
metrics = prometheus_flask_exporter.PrometheusMetrics(app)

@app.route('/test')
def index():
    return 'Hello world'

If you flask run this, then you'll get the app endpoints and the metrics endpoint exposed on the same app that belongs to the (single) development server, e.g.:

• curl localhost:5000/test returns Hello world, and
• curl localhost:5000/metrics returns the Prometheus metrics

Does this work for you @raxod502 ?
I think the best course of action from here is document this fact and/or make this library raise an exception when we're in flask run mode and someone calls the start_http_server that we (now) know won't work.

[raxod502]

That works for development mode, but what about in production? It sounds like the Flask folks are saying it is not appropriate to use the Flask development server in production, yet it seems like prometheus_flask_exporter does not have an option to use anything other than the Flask development server.

What if we add an API to prometheus_flask_exporter that simply returns the Flask app for exposing metrics, so that the user may choose how they wish to serve that application (e.g. by adding a dependency on Gunicorn or a similar WGSI package)?

[rycus86]

We do have examples on how to integrate with "production ready" servers, for example:

• Gunicorn
  • https://github.com/rycus86/prometheus_flask_exporter/tree/master/examples/gunicorn
  • https://github.com/rycus86/prometheus_flask_exporter/tree/master/examples/gunicorn-internal (metrics endpoint exposed on the same app)
  • https://github.com/rycus86/prometheus_flask_exporter/tree/master/examples/uwsgi
  • https://github.com/rycus86/prometheus_flask_exporter/tree/master/examples/wsgi

One thing to note: for the non-internal multiprocessing enabled Prometheus exporter wrappers, we use the prometheus_client.start_http_server function from the Prometheus client library, see https://github.com/rycus86/prometheus_flask_exporter/blob/master/prometheus_flask_exporter/multiprocess.py#L78

What if we add an API to prometheus_flask_exporter that simply returns the Flask app for exposing metrics, so that the user may choose how they wish to serve that application (e.g. by adding a dependency on Gunicorn or a similar WGSI package)?

We could perhaps expose the contents of https://github.com/rycus86/prometheus_flask_exporter/blob/master/prometheus_flask_exporter/__init__.py#L272 as a function that you can then call from an endpoint you supply in whatever way is appropriate - if that's desirable, we can look into that.