UI API CLI and Breeze integration

[pierrejeambrun]

Add the airflow ui-api command. Include it in airflow standalone and also in breeze to have that at the startup.

Everything is working locally port binding from host to docker api port, standalone command, airflow ui-api command.

Note: We use gunicorn special uvicorn worker to run the FastAPI app in production. This is what is recommended by the uvicorn project itself doc here

⚠️
I just have one thing I need to investigate which is that there are some errors on the TTIN TTOUT signals of the gunicorn server. That does not appear when I run the gunicorn server manually, that seems to come from our GunicornMonitor. I need to dive deeper into the GunicornMonitor loop to see if this is 'expected or not', probably not.
cf screenshot, workers are being brought up and down, and when closing apparently a filedesriptor is already closed or improperly closed.(solved), why are workers terminated in the first place.

[pierrejeambrun]

Ok, I was able to get rid of the bad file descriptor error when shutting down the worker. Cf this commit 76b653e

Now I need to understand why are workers first incrementated (most likely by the monitoring loop), to then yield a sigterm error. That does not seem normal.

[jscheffl]

I can confirm it is working as described. Code looks good.

I am just a bit sceptic that we need to run another server process whereas in the other webserver we also host an API. As the new FastAPI also is running in gunicorn... can this not directly be integrated into webserver and we save another port forward, URL routing and (potential) complexity when we go further?

[potiuk]

LGTM.

I think for Airflow 3 development, it's quite ok to keep things separate, but agree with @jscheffl that it should be unified eventually. I think we should get rid of all the old connexion thing - having all the experience with connexion 3 and convert all our APIs and websever endpoints to use FastAPI - looking at how much it handles for us, and the fact that we can make some incompatibility decisions made in Airflow 2 for the APIs (error handling for one, base_url handling etc.) - having a single, unified FastAPI (even if opinionated and with different decieions) is a great opportunity to "standardising" our UI work - that would need some work on converting existing REST endpoints (and especially tests) to use Fast API, but other that potential slight incompatibilities, I do not see any problems with it. Fast API can generate OpenAPI spec from the code, get swagger conected and so on and we can make it works "slightly differently" for airlfow 3 - and migrate the code for our current REST API gradually.

But it does not have to be "now".

[potiuk]

I'd say - migrating existing REST APIs to the new "ui-api" command is the way to go.

[pierrejeambrun]

I think for Airflow 3 development, it's quite ok to keep things separate, but agree with @jscheffl that it should be unified eventually. I think we should get rid of all the old connexion thing - having all the experience with connexion 3 and convert all our APIs and websever endpoints to use FastAPI - looking at how much it handles for us, and the fact that we can make some incompatibility decisions made in Airflow 2 for the APIs (error handling for one, base_url handling etc.) - having a single, unified FastAPI (even if opinionated and with different decieions) is a great opportunity to "standardising" our UI work - that would need some work on converting existing REST endpoints (and especially tests) to use Fast API, but other that potential slight incompatibilities, I do not see any problems with it. Fast API can generate OpenAPI spec from the code, get swagger conected and so on and we can make it works "slightly differently" for airlfow 3 - and migrate the code for our current REST API gradually.

But it does not have to be "now".

I'd say - migrating existing REST APIs to the new "ui-api" command is the way to go.

Totally agree. At the end of the day we want a standardized way of handling our Rest APIs, code first spec, most likely all FastAPI. And we will migrate the existing for airflow 3 development. Then the new RestAPI can be a drop in replacement for the existing plublic API that will be completely removed I suppose.

[pierrejeambrun]

Ok, the TTIN followed by TTOU seems perfectly normal, worker_refresh_interval and worker_refresh_batch_size handle this and is expected. (every 30 seconds a new worker is refreshed).

Merging when CI is green

[potiuk]

Ok, the TTIN followed by TTOU seems perfectly normal, worker_refresh_interval and worker_refresh_batch_size handle this and is expected. (every 30 seconds a new worker is refreshed).

correct