Skip to content

Commit

Permalink
Add notes about PIP 20.3 breaking Airflow installation (#12840)
Browse files Browse the repository at this point in the history
Part of #12838
  • Loading branch information
potiuk authored Dec 5, 2020
1 parent 12c1fe2 commit e9b2ff5
Show file tree
Hide file tree
Showing 11 changed files with 129 additions and 6 deletions.
20 changes: 19 additions & 1 deletion CONTRIBUTING.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.. Licensed to the Apache Software Foundation (ASF) under one
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
Expand Down Expand Up @@ -532,6 +532,15 @@ than production performance. The production images are not yet officially publis
Airflow dependencies
====================

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.


Extras
------

Expand Down Expand Up @@ -666,6 +675,15 @@ as of airflow 1.10.10 and further improved with 1.10.12 (moved to separate orpha
Pinned constraint files
=======================

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.


By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
still allowing the apache-airflow package to install. This means that ``apache-airflow`` package might fail to
install in case a direct or transitive dependency is released that breaks the installation. In such case
Expand Down
10 changes: 10 additions & 0 deletions IMAGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,16 @@ parameter to Breeze:
./breeze build-image --python 3.7 --additional-extras=presto \
--production-image --install-airflow-version=1.10.13
.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.


This will build the image using command similar to:

.. code-block:: bash
Expand Down
8 changes: 8 additions & 0 deletions INSTALL
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,14 @@ java -jar apache-rat.jar -E ./.rat-excludes -d .
python3 -m venv PATH_TO_YOUR_VENV
source PATH_TO_YOUR_VENV/bin/activate

NOTE!!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.

# [required] building and installing by pip (preferred)
pip install .

Expand Down
19 changes: 19 additions & 0 deletions LOCAL_VIRTUALENV.rst
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,16 @@ of required packages.
Extra Packages
--------------

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.



You can also install extra packages (like ``[ssh]``, etc) via
``pip install -e [EXTRA1,EXTRA2 ...]``. However, some of them may
have additional install and setup requirements for your local system.
Expand Down Expand Up @@ -114,6 +124,15 @@ To create and initialize the local virtualenv:
2. Install Python PIP requirements:

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.


.. code-block:: bash
pip install -U -e ".[devel,<OTHER EXTRAS>]" # for example: pip install -U -e ".[devel,google,postgres]"
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,15 @@ correct Airflow tag/version/branch and python versions in the URL.

1. Installing just Airflow:

NOTE!!!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
`pip upgrade --pip==20.2.4` or, in case you use Pip 20.3, you need to add option
`--use-deprecated legacy-resolver` to your pip install command.


```bash
pip install apache-airflow==1.10.13 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.13/constraints-3.7.txt"
Expand Down
10 changes: 10 additions & 0 deletions UPDATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -1668,6 +1668,16 @@ For example:
If you want to install integration for Microsoft Azure, then instead of `pip install apache-airflow[atlas]`
you should use `pip install apache-airflow[apache.atlas]`.


NOTE!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
`pip upgrade --pip==20.2.4` or, in case you use Pip 20.3, you need to add option
`--use-deprecated legacy-resolver` to your pip install command.


If you want to install integration for Microsoft Azure, then instead of

```
Expand Down
18 changes: 18 additions & 0 deletions UPGRADING_TO_2.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,15 @@ assists users migrating to a new version.

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

NOTE!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
`pip upgrade --pip==20.2.4` or, in case you use Pip 20.3, you need to add option
`--use-deprecated legacy-resolver` to your pip install command.


## Step 1: Upgrade to Python 3

Airflow 1.10 will be the last release series to support Python 2. Airflow 2.0.0 will require Python 3.6+.
Expand Down Expand Up @@ -102,6 +111,15 @@ The most important step in this transition is also the easiest step to do in pie
using the [backport providers](./backport-providers.rst) service. In your own time, you can transition to using these backport-providers
by pip installing the provider via `pypi` and changing the import path.


NOTE!!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
`pip upgrade --pip==20.2.4` or, in case you use Pip 20.3, you need to add option
`--use-deprecated legacy-resolver` to your pip install command.

For example: While historically you might have imported the DockerOperator in this fashion:

```python
Expand Down
1 change: 1 addition & 0 deletions airflow/providers/discord/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ are in `airflow.providers.discord` python package.

## Installation


You can install this package on top of an existing airflow 2.* installation via
`pip install apache-airflow-providers-discord`

Expand Down
8 changes: 8 additions & 0 deletions dev/provider_packages/PROVIDER_README_TEMPLATE.md.jinja2
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,14 @@ are in `{{FULL_PACKAGE_NAME}}` python package.

## Installation

NOTE!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
`pip upgrade --pip==20.2.4` or, in case you use Pip 20.3, you need to add option
`--use-deprecated legacy-resolver` to your pip install command.

You can install this package on top of an existing airflow 2.* installation via
`pip install {{PACKAGE_PIP_NAME}}`

Expand Down
23 changes: 18 additions & 5 deletions docs/apache-airflow/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,15 @@ and python versions in the URL.
1. Installing just Airflow

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.


.. code-block:: bash
AIRFLOW_VERSION=1.10.13
Expand All @@ -61,6 +70,15 @@ and python versions in the URL.
2. Installing with extras (for example postgres, google)

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.


.. code-block:: bash
AIRFLOW_VERSION=1.10.13
Expand All @@ -74,11 +92,6 @@ Airflow with such extras, the necessary provider packages are installed automati
PyPI for those packages). However you can freely upgrade and install provider packages independently from
the main Airflow installation.

.. note:: Automated installation of Provider packages does not work in Airflow 2.0.0b1 - for this version
you have to install provider packages manually. As of Airflow 2.0.0b2 the corresponding
provider packages are installed together with the extras.

Read more about it in the :ref:`Provider Packages <installation:provider_packages>` section.

Requirements
''''''''''''
Expand Down
9 changes: 9 additions & 0 deletions docs/apache-airflow/start.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,15 @@
Quick Start
-----------

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.


The installation is quick and straightforward.

.. code-block:: bash
Expand Down

0 comments on commit e9b2ff5

Please sign in to comment.