Add SQLAlchemy to the dev‑dependency group in providers

[Prab-27]

Related : #59895
Related discussion : here and here

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[Prab-27]

Do I need to update this

airflow/providers/postgres/pyproject.toml

Line 108 in 39e52f5

"psycopg[binary]>=3.2.9",

and

airflow/providers/amazon/pyproject.toml

Line 179 in 39e52f5

"aiobotocore[boto3]>=2.21.1",

like apache-airflow-providers-postgres[psycopg[binary]] since we applied here

[potiuk]

Nice! Thanks

Do I need to update this:
"psycopg[binary]>=3.2.9",

This one should be actually removed -> psycopg2 is already added as required dependency - so we can remove psycopg from dev deps.

"aiobotocore[boto3]>=2.21.1",

This one needs to be changed anyway - because we keep on getting warning about aiobotocore not having boto3 extra.

I have some other small fixes to fix so I will fix those. Thanks for notifying @Prab-27 !

[Prab-27]

@potiuk Can I remove psycopg2 , I want to learn more about UV

I tried to find the answe but I couldn't

1 - When we run uv sync or uv run, we only install the dev dependency group. How will the required and optional dependencies be upgraded in that case?

2 - I have read this - here
Why do we do this? Do we group the optional SQLAlchemy dependencies together with task-sdk, and the required SQLAlchemy dependency with airflow-core, or is it not possible to separate them since SQLAlchemy is required?

3 What’s the meaning of workspace = true? Does it mean we can install it from PyPI? (I noticed )Most modules (like task-sdk and providers) have workspace = true, but airflowctl does not, since we don’t install it from PyPI

4 I don’t exactly remember, but when we changed airflow-core/src/airflow, you mentioned that we could only achieve this by using UV and workspaces. Is this related to the workspace = true option?

[potiuk]

1 - When we run uv sync or uv run, we only install the dev dependency group. How will the required and optional dependencies be upgraded in that case?

No - we install required dependencies and dev group. No optional deps unless you also specify --extra flag. This is what we are doing here, when we want some optional devs to by also added for development always - we just add them as current-distribution[extra] to dev group so that always when dev group is updated by uv sync extra deps are also updated.

2 - I have read this - Why do we do this? Do we group the optional SQLAlchemy dependencies together with task-sdk, and the required SQLAlchemy dependency with airflow-core, or is it not possible to separate them since SQLAlchemy is required?

So far everything we install requires airflow-core - which bings sqlalchemy in, but shortly - for Airflow 3.2, airflow-core and airflow-task-sdk will be completely independent from each other and task-sdk will not have sqlalchemy as dependency. So far sqlalchemy was simply always installed through provider -> task-sdk-> airflow-core -> sqlalchemy. Therefor each provider that expects sqlalchmy will have to bring it in as a dependnecy (either optional or required - depending how they use. There are few other dependency "knots" that we will have to unentangle - with executors, and ui plugins - in order to make airflow distributions more isolated, independent and only have dependencies they needed - mostly because in the future, different components of airflow will be installable separately - with different sets of dependencies - different image will be needed for airlfow-core, different (possbibly) for airflow-ui, different for task-sdk, also multi-team feature will allow different teams to have different set of providers and dependencies installed in their workers. This is a "north" star of our isolation that I also explained in more detail in this discussion at the devlist https://lists.apache.org/thread/xwspbh8wfqcyovpznqh415mbs227qdoq

3. What’s the meaning of workspace = true? Does it mean we can install it from PyPI? (I noticed )Most modules (like task-sdk and providers) have workspace = true, but airflowctl does not, since we don’t install it from PyPI

Also airflow-ctl has "workspace=true" - whrere do you get it does not have it ? Maybe we missed it somewhere? Some older version did not have it yes - but I think we added it since.

Look at https://docs.astral.sh/uv/concepts/projects/workspaces/. Workspace is a way to bring together several distributions that are otherwise separate distributions - to make sure that they all can be installed together with common set of dependencies. Apart of the isolation above we also have to support the case where airflow is installed from a single image - where you can run all components: scheduier, api-server, workers (with providers), triggerer etc. -> and we have to be able to make sure that all those distributions that we want to install together potentially - can in fact be installed - i.e. they have no conflicting dependencies. This is what uv sync in the root airflow directory does (this is the workspace root) - it actually takes all the distributions and attempts to install all of them and find the right set of dependencies for all of them that are non-conflicting. Also when distributions are in the same workspace, you can refer to them by their name as dependency, and rather than installing them from PyPI when you run uv sync during local development, you install them from sources in the same workspace. This way when provider-airbyte uses task-sdk as dependency, when you go to the folder of the provider-airbyte and you run uv sync, then uv will install task-sdk from the local task-sdk source folder in airflow repo that is part of the workspace, but it will install airbyte-api from PyPI because it cannot find it in local workspace.

4. I don’t exactly remember, but when we changed airflow-core/src/airflow, you mentioned that we could only achieve this by using UV and workspaces. Is this related to the workspace = true option?

Not sure what you are asking for here and what changing airflow-core/src/airflow might mean - but I recommend you to read the workspace docs in uv. You can also read my series of blogs about the monorepo/workspace - there are 4 parts starting with https://medium.com/apache-airflow/modern-python-monorepo-for-apache-airflow-part-1-1fe84863e1e1 - link to remaining 3 parts in there, I am also (this Saturday) giving a talk about it at FOSDEM https://fosdem.org/2026/schedule/event/WE7NHM-modern-python-monorepo-apache-airflow/ which should be publishded in a few weeks. Also we are recording a "Talk Python To Me" podcast with @amoghrajesh on 10th of Feb about it https://talkpython.fm/ and there will be podcast recording available (but also livestream will likely be there - more info closer to the time on slack and devlist).

[Prab-27]

1 - When we run uv sync or uv run, we only install the dev dependency group. How will the required and optional dependencies be upgraded in that case?

No - we install required dependencies and dev group. No optional deps unless you also specify --extra flag. This is what we are doing here, when we want some optional devs to by also added for development always - we just add them as current-distribution[extra] to dev group so that always when dev group is updated by uv sync extra deps are also updated.

2 - I have read this - Why do we do this? Do we group the optional SQLAlchemy dependencies together with task-sdk, and the required SQLAlchemy dependency with airflow-core, or is it not possible to separate them since SQLAlchemy is required?

So far everything we install requires airflow-core - which bings sqlalchemy in, but shortly - for Airflow 3.2, airflow-core and airflow-task-sdk will be completely independent from each other and task-sdk will not have sqlalchemy as dependency. So far sqlalchemy was simply always installed through provider -> task-sdk-> airflow-core -> sqlalchemy. Therefor each provider that expects sqlalchmy will have to bring it in as a dependnecy (either optional or required - depending how they use. There are few other dependency "knots" that we will have to unentangle - with executors, and ui plugins - in order to make airflow distributions more isolated, independent and only have dependencies they needed - mostly because in the future, different components of airflow will be installable separately - with different sets of dependencies - different image will be needed for airlfow-core, different (possbibly) for airflow-ui, different for task-sdk, also multi-team feature will allow different teams to have different set of providers and dependencies installed in their workers. This is a "north" star of our isolation that I also explained in more detail in this discussion at the devlist https://lists.apache.org/thread/xwspbh8wfqcyovpznqh415mbs227qdoq

3. What’s the meaning of workspace = true? Does it mean we can install it from PyPI? (I noticed )Most modules (like task-sdk and providers) have workspace = true, but airflowctl does not, since we don’t install it from PyPI

Also airflow-ctl has "workspace=true" - whrere do you get it does not have it ? Maybe we missed it somewhere? Some older version did not have it yes - but I think we added it since.

Look at https://docs.astral.sh/uv/concepts/projects/workspaces/. Workspace is a way to bring together several distributions that are otherwise separate distributions - to make sure that they all can be installed together with common set of dependencies. Apart of the isolation above we also have to support the case where airflow is installed from a single image - where you can run all components: scheduier, api-server, workers (with providers), triggerer etc. -> and we have to be able to make sure that all those distributions that we want to install together potentially - can in fact be installed - i.e. they have no conflicting dependencies. This is what uv sync in the root airflow directory does (this is the workspace root) - it actually takes all the distributions and attempts to install all of them and find the right set of dependencies for all of them that are non-conflicting. Also when distributions are in the same workspace, you can refer to them by their name as dependency, and rather than installing them from PyPI when you run uv sync during local development, you install them from sources in the same workspace. This way when provider-airbyte uses task-sdk as dependency, when you go to the folder of the provider-airbyte and you run uv sync, then uv will install task-sdk from the local task-sdk source folder in airflow repo that is part of the workspace, but it will install airbyte-api from PyPI because it cannot find it in local workspace.

4. I don’t exactly remember, but when we changed airflow-core/src/airflow, you mentioned that we could only achieve this by using UV and workspaces. Is this related to the workspace = true option?

Not sure what you are asking for here and what changing airflow-core/src/airflow might mean - but I recommend you to read the workspace docs in uv. You can also read my series of blogs about the monorepo/workspace - there are 4 parts starting with https://medium.com/apache-airflow/modern-python-monorepo-for-apache-airflow-part-1-1fe84863e1e1 - link to remaining 3 parts in there, I am also (this Saturday) giving a talk about it at FOSDEM https://fosdem.org/2026/schedule/event/WE7NHM-modern-python-monorepo-apache-airflow/ which should be publishded in a few weeks. Also we are recording a "Talk Python To Me" podcast with @amoghrajesh on 10th of Feb about it https://talkpython.fm/ and there will be podcast recording available (but also livestream will likely be there - more info closer to the time on slack and devlist).

All concepts are explained very well !! I really appreciate you !!