Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clarify that bundle extras should not be used for PyPi installs #23697

Merged
merged 1 commit into from
May 13, 2022
Merged

Clarify that bundle extras should not be used for PyPi installs #23697

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/apache-airflow/extra-packages-ref.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,12 @@ Those are extras that provide support for integration with external systems via
Bundle extras
-------------

Those are extras that install one ore more extras as a bundle.
Those are extras that install one ore more extras as a bundle. Note that those extras should only be used for "development" version
of Airflow - i.e. when Airflow is installed from sources. Because of the way how bundle extras are constructed they might not
work when airflow is installed from 'PyPI`.

If you want to install Airflow from PyPI with "all" extras (which should basically be never needed - you almost never need all extras from Airflow),
you need to list explicitly all the non-bundle extras that you want to install.
Comment on lines +301 to +302
Copy link
Contributor

@eladkal eladkal May 13, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really relevant here but since we are talking about dev extras...
One of the things that troubles me is that the packages that we bring with devel are also part of the constraints that we release to the users - I believe that we should not do that.

For example in https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0/constraints-3.7.txt you can find freezegun.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, yes

Copy link
Member Author

@potiuk potiuk May 13, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a problem and removing those devel-only requirements is extremely hard and borderline impossible in automated way.

Why this is not a problem?

Constraints are just constraints not requirements. it's merely a meta-data tellng "if you are installing freezegun, you should be installing this and that version". They have 0 influence on whether freezegun will be installed at all.

Why this is difficutt or even impossible?

The problem is that It might be very difficult to separate those out automatically (in a future-safe way) actually because we have no idea if dependency is not used in some of our dependencies beyond our devel deps.

With Freezegun, this is probably fine as you know what freezegun is typically used for, but there were already a few cases where something that "seem" to be only used for devel, was actually used (wrongly or not but there was an implicit dependency) for some packages we installed.

We could posisbly try to figure it out by traversing the dependency tree, but this is doomed because of the way extras are treated (and not even Airflow extras but other packages extras). The way pip treats extras is a bit unexpected (but there is no other good way) - they are optional and only valid during installation and the relation betwen the packages that installed them is gone right after the installation. The fact that there is a dependency between the package and whatever is declared as extra basically disappears at the moment pip install completes.

Unfiortunately when a package declares dependency it MIGHT add extra and that leads to untraceable transitive dependencies. And that dependency - MIGHT declare other dependencies WITH EXTRAS in THEIR extras, and it can go on and on.

For example:

if we ourselves (and this is not real case but might happen) add apache-beam[pandas] as dependency and apache -beam[pandas] adds pandas[freeze] as dependency and pandas[freeze] adds freezegun<1.4 as dependency, then we have an implicit, transitive dependency on freezgun<1.4 from airflow[apache.beam].

And what's worse beyond the pip resolution process, we have no idea that:

a) freezegun is actually our non-devel transitive dependency (throuhgh beam and pandas)
b) we require freezegun < 1.4

This information is not available anywhere out-of-the-box. In other to know that - we would have to repeat the pip resolver process and basically make a snapshot of the in-memory information that teh resolver keeps to find out which packages should be installed. This is the only way to find out that freezegun is our non-devel depenedency as well.

If we remove freezegun from our constraints, it might be, that someone installs it in the version 1.5 precisely because we do not have it in constraints (and that will be wrong because apache-airflow[apache.beam] requires freezegun|<1.4.

Long story short - freezegun and the above example are pretty artifficial, but there is basically no way (other than repeating what the pip resolver does to generate the list of dependencies that are devel-only.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And yes. Dependencies are HARD 🤯


+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
| extra | install command | enables |
Expand Down