Airflow CLI to list roles is slow

[malthe]

Apache Airflow version

2.5.0

What happened

We're currently running a suboptimal setup where database connectivity is laggy, 125ms roundtrip.

This has interesting consequences. For example, airflow roles list is really slow. Turns out that it's doing a lot of individual queries.

What you think should happen instead

Ideally, listing roles should be a single (perhaps complex) query.

How to reproduce

We're using py-spy to sample program execution:

$ py-spy record -o spy.svg -i --rate 250 --nonblocking airflow roles list

Now, to see the bad behavior, the database should incur significant latency.

Operating System

Linux

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[malthe]

Flamegraph generated using py-spy.

Click to view in a new window, it's interactive (hover to see samples). All the little squares are SQLAlchemy code.

[potiuk]

I believe there is not much we can do - when you look at the flamegraph it initializes flask and flask-app-builder. Until we replace the user model with somethiing else thanf Flass App Builder, there is not much we can do I am afraid.

[potiuk]

Correction: We could likely decrease it a bit by removing connexion (REST API) intialization- it seems it contributes to > 50% of the time

[potiuk]

Or maybe we could actually not initialize views ...