[SPIKE] Work out a way for API server to load providers declaratively

[amoghrajesh]

Body

Right now, API server loads all provider code just for UI metadata(static data), the intention I have in mind is that API server can start without loading provider code for connections, hooks etc

To do this, I am exploring some options where UI metadata comes from static sources like yaml.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[amoghrajesh]

This is being worked on by: #60410

[jason810496]

Or another approach, similar to how we introduced a ‎⁠cli⁠ section in the provider metadata: #59805

We could separate the UI metadata into a new, clean module.

For example, if you search for ‎⁠cli/definition.py⁠, that’s where all the provider‑level CLI definitions live. Then we could still use the existing ProviderManager approach (loading objects via ‎⁠importlib.metadata⁠). For community providers, we could add a prek to avoid importing heavy modules in the new UI metadata module, as shown here: https://github.com/apache/airflow/pull/59805/changes#diff-832133982e269231a55df6a2e72b5ac560f7b828b38b3cff2b904c0d96405ab1R145

Which should significantly reduce the loading time as we just import some dicts via ProviderManager.

[jason810496]

The static YAML approach also makes sense to me! Even if we go with the static YAML approach, I think we still require providers’ ‎⁠pip⁠ dependencies on the API server, and it is unavoidable to call ‎⁠ProviderManager⁠ for the end-to-end process.

[amoghrajesh]

Or another approach, similar to how we introduced a ‎⁠cli⁠ section in the provider metadata: #59805

We could separate the UI metadata into a new, clean module.

For example, if you search for ‎⁠cli/definition.py⁠, that’s where all the provider‑level CLI definitions live. Then we could still use the existing ProviderManager approach (loading objects via ‎⁠importlib.metadata⁠). For community providers, we could add a prek to avoid importing heavy modules in the new UI metadata module, as shown here: https://github.com/apache/airflow/pull/59805/changes#diff-832133982e269231a55df6a2e72b5ac560f7b828b38b3cff2b904c0d96405ab1R145

Which should significantly reduce the loading time as we just import some dicts via ProviderManager.

Maybe, yes, but trying to have a declarative way instead. This would require loading python code provided by users, which could also pose a security risk for example. As of today, nothing can stop a user from writing some malicious code in get_ui_field_behaviour

[amoghrajesh]

The static YAML approach also makes sense to me! Even if we go with the static YAML approach, I think we still require providers’ ‎⁠pip⁠ dependencies on the API server, and it is unavoidable to call ‎⁠ProviderManager⁠ for the end-to-end process.

Right now yes, but I am proposing a change soon on that which will let API server read connection schemas truly from yaml without having the provider installed at all. A peek into the solution would be something along the lines of having pre packaged bundles for provider.yml in api server