Skip to content

Commit c4476d2

Browse files
potiukpotiuk
committed
Get rid of Airflow 1.10 in Breeze (#15712)
Gets rid of Airflow 1.10 in Breeze and script/configuration. We were still using occasionally the master version of Breeze to run 1.10 version of Airflow, but this madness should end now when we are approaching 2.1 release. All changes in Breeze were so far ported to 1.10 but this is about the time to finish it. (cherry picked from commit 57ad4af)
1 parent af06c9c commit c4476d2

File tree

23 files changed

+80
-344
lines changed

23 files changed

+80
-344
lines changed

‎BREEZE.rst

Lines changed: 17 additions & 95 deletions
Original file line numberDiff line numberDiff line change
@@ -541,7 +541,7 @@ For all development tasks, unit tests, integration tests, and static code checks
541541
**CI image** maintained on the DockerHub in the ``apache/airflow`` repository.
542542
This Docker image contains a lot of test-related packages (size of ~1GB).
543543
Its tag follows the pattern of ``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>-ci``
544-
(for example, ``apache/airflow:master-python3.6-ci`` or ``apache/airflow:v1-10-test-python3.6-ci``).
544+
(for example, ``apache/airflow:master-python3.6-ci`` or ``apache/airflow:v2-0-test-python3.6-ci``).
545545
The image is built using the `<Dockerfile.ci>`_ Dockerfile.
546546

547547
The CI image is built automatically as needed, however it can be rebuilt manually with
@@ -638,7 +638,7 @@ The **Production image** is also maintained on the DockerHub in the
638638
``apache/airflow`` repository. This Docker image (and Dockerfile) contains size-optimised Airflow
639639
installation with selected extras and dependencies. Its tag follows the pattern of
640640
``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>`` (for example, ``apache/airflow:master-python3.6``
641-
or ``apache/airflow:v1-10-test-python3.6``).
641+
or ``apache/airflow:v2-0-test-python3.6``).
642642

643643
However in many cases you want to add your own custom version of the image - with added apt dependencies,
644644
python dependencies, additional Airflow extras. Breeze's ``build-image`` command helps to build your own,
@@ -693,56 +693,6 @@ Same as above but uses python 3.7.
693693
</a>
694694
</div>
695695

696-
Building Production images for 1.10 Airflow versions
697-
----------------------------------------------------
698-
699-
With Breeze you can also use the master Dockerfile to build custom images for released Airflow versions.
700-
This works in the same way as building production image from master, but you need to add additional switch
701-
``--install-airflow-version``. You should pass version of airflow (as released in PyPI). It can be used
702-
to install both released versions and release candidates. Similarly as in case of master images,
703-
we can pass additional extras/dependencies to install via the additional flags.
704-
705-
.. code-block:: bash
706-
707-
./breeze build-image --production-image --additional-extras "jira" --install-airflow-version="1.10.11"
708-
709-
Builds airflow image with released Airflow version 1.10.11 and additional extra "jira" added.
710-
711-
.. code-block:: bash
712-
713-
./breeze build-image --production-image --install-airflow-version="1.10.11rc2"
714-
715-
Builds airflow image with released Airflow version 1.10.11rc2.
716-
717-
718-
You can also build airflow directly from GitHub source code - by providing Git Reference via
719-
``--install-airflow-reference``. The reference can be a branch name, tag name, or commit hash. This
720-
is useful mostly for testing.
721-
722-
.. code-block:: bash
723-
724-
./breeze build-image --production-image --install-airflow-reference="v1-10-test"
725-
726-
This Builds airflow image from the current ``v1-10-test`` branch of Airflow.
727-
728-
.. code-block:: bash
729-
730-
./breeze build-image --production-image \
731-
--install-airflow-reference="0d91fcf725f69e10f0969ca36f9e38e1d74110d0"
732-
733-
This Builds airflow image from the ``0d91fcf725f69e10f0969ca36f9e38e1d74110d0`` commit hash on
734-
GitHub.
735-
736-
.. raw:: html
737-
738-
<div align="center">
739-
<a href="https://youtu.be/4MCTXq-oF68?t=1586">
740-
<img src="images/breeze/overlayed_breeze_build_images_released_versions.png" width="640"
741-
alt="Airflow Breeze - Building Production images for 1.10 Airflow versions">
742-
</a>
743-
</div>
744-
745-
746696
Running static checks
747697
---------------------
748698

@@ -823,8 +773,7 @@ Generating constraints
823773
----------------------
824774

825775
Whenever setup.py gets modified, the CI master job will re-generate constraint files. Those constraint
826-
files are stored in separated orphan branches: ``constraints-master``, ``constraints-2-0``
827-
and ``constraints-1-10``.
776+
files are stored in separated orphan branches: ``constraints-master``, ``constraints-2-0``.
828777

829778
Those are constraint files as described in detail in the
830779
`<CONTRIBUTING.rst#pinned-constraint-files>`_ contributing documentation.
@@ -1319,21 +1268,18 @@ This is the current syntax for `./breeze <./breeze>`_:
13191268
-p, --python PYTHON_MAJOR_MINOR_VERSION
13201269
Python version used for the image. This is always major/minor version.
13211270
1322-
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
1323-
--install-airflow-version or --install-airflow-reference flags.
1324-
13251271
One of:
13261272
1327-
2.7 3.5 3.6 3.7 3.8
1273+
3.6 3.7 3.8
13281274
13291275
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
13301276
Uses differen version of Airflow when building PROD image.
13311277
1332-
2.0.3 2.0.2 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist
1278+
2.0.3 2.0.2 2.0.1 2.0.0 wheel sdist
13331279
13341280
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
13351281
Installs Airflow directly from reference in GitHub when building PROD image.
1336-
This can be a GitHub branch like master or v1-10-test, or a tag like 2.0.0a1.
1282+
This can be a GitHub branch like master or v2-0-test, or a tag like 2.0.0a1.
13371283
13381284
--installation-method INSTALLATION_METHOD
13391285
Method of installing Airflow in PROD image - either from the sources ('.')
@@ -1564,12 +1510,9 @@ This is the current syntax for `./breeze <./breeze>`_:
15641510
-p, --python PYTHON_MAJOR_MINOR_VERSION
15651511
Python version used for the image. This is always major/minor version.
15661512
1567-
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
1568-
--install-airflow-version or --install-airflow-reference flags.
1569-
15701513
One of:
15711514
1572-
2.7 3.5 3.6 3.7 3.8
1515+
3.6 3.7 3.8
15731516
15741517
-I, --production-image
15751518
Use production image for entering the environment and builds (not for tests).
@@ -1611,7 +1554,7 @@ This is the current syntax for `./breeze <./breeze>`_:
16111554
16121555
Generates pinned constraint files with all extras from setup.py. Those files are generated in
16131556
files folder - separate files for different python version. Those constraint files when
1614-
pushed to orphan constraints-master, constraints-2-0 and constraints-1-10 branches are used
1557+
pushed to orphan constraints-master, constraints-2-0 branches are used
16151558
to generate repeatable CI builds as well as run repeatable production image builds and
16161559
upgrades when you want to include installing or updating some of the released providers
16171560
released at the time particular airflow version was released. You can use those
@@ -1634,12 +1577,9 @@ This is the current syntax for `./breeze <./breeze>`_:
16341577
-p, --python PYTHON_MAJOR_MINOR_VERSION
16351578
Python version used for the image. This is always major/minor version.
16361579
1637-
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
1638-
--install-airflow-version or --install-airflow-reference flags.
1639-
16401580
One of:
16411581
1642-
2.7 3.5 3.6 3.7 3.8
1582+
3.6 3.7 3.8
16431583
16441584
-v, --verbose
16451585
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
@@ -1760,12 +1700,9 @@ This is the current syntax for `./breeze <./breeze>`_:
17601700
-p, --python PYTHON_MAJOR_MINOR_VERSION
17611701
Python version used for the image. This is always major/minor version.
17621702
1763-
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
1764-
--install-airflow-version or --install-airflow-reference flags.
1765-
17661703
One of:
17671704
1768-
2.7 3.5 3.6 3.7 3.8
1705+
3.6 3.7 3.8
17691706
17701707
17711708
####################################################################################################
@@ -1848,16 +1785,13 @@ This is the current syntax for `./breeze <./breeze>`_:
18481785
airflow is just removed. In this case airflow package should be added to dist folder
18491786
and --use-packages-from-dist flag should be used.
18501787
1851-
2.0.3 2.0.2 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist none
1788+
2.0.3 2.0.2 2.0.1 2.0.0 wheel sdist none
18521789
18531790
--use-packages-from-dist
18541791
In CI image, if specified it will look for packages placed in dist folder and
18551792
it will install the packages after entering the image.
18561793
This is useful for testing provider packages.
18571794
1858-
--no-rbac-ui
1859-
Disables RBAC UI when Airflow 1.10.* is installed.
1860-
18611795
--load-example-dags
18621796
Include Airflow example dags.
18631797
@@ -1950,12 +1884,9 @@ This is the current syntax for `./breeze <./breeze>`_:
19501884
-p, --python PYTHON_MAJOR_MINOR_VERSION
19511885
Python version used for the image. This is always major/minor version.
19521886
1953-
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
1954-
--install-airflow-version or --install-airflow-reference flags.
1955-
19561887
One of:
19571888
1958-
2.7 3.5 3.6 3.7 3.8
1889+
3.6 3.7 3.8
19591890
19601891
-b, --backend BACKEND
19611892
Backend to use for tests - it determines which database is used.
@@ -2017,12 +1948,9 @@ This is the current syntax for `./breeze <./breeze>`_:
20171948
-p, --python PYTHON_MAJOR_MINOR_VERSION
20181949
Python version used for the image. This is always major/minor version.
20191950
2020-
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
2021-
--install-airflow-version or --install-airflow-reference flags.
2022-
20231951
One of:
20241952
2025-
2.7 3.5 3.6 3.7 3.8
1953+
3.6 3.7 3.8
20261954
20271955
-F, --force-build-images
20281956
Forces building of the local docker images. The images are rebuilt
@@ -2419,12 +2347,9 @@ This is the current syntax for `./breeze <./breeze>`_:
24192347
-p, --python PYTHON_MAJOR_MINOR_VERSION
24202348
Python version used for the image. This is always major/minor version.
24212349
2422-
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
2423-
--install-airflow-version or --install-airflow-reference flags.
2424-
24252350
One of:
24262351
2427-
2.7 3.5 3.6 3.7 3.8
2352+
3.6 3.7 3.8
24282353
24292354
****************************************************************************************************
24302355
Choose backend to run for Airflow
@@ -2556,11 +2481,11 @@ This is the current syntax for `./breeze <./breeze>`_:
25562481
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
25572482
Uses differen version of Airflow when building PROD image.
25582483
2559-
2.0.3 2.0.2 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist
2484+
2.0.3 2.0.2 2.0.1 2.0.0 wheel sdist
25602485
25612486
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
25622487
Installs Airflow directly from reference in GitHub when building PROD image.
2563-
This can be a GitHub branch like master or v1-10-test, or a tag like 2.0.0a1.
2488+
This can be a GitHub branch like master or v2-0-test, or a tag like 2.0.0a1.
25642489
25652490
--installation-method INSTALLATION_METHOD
25662491
Method of installing Airflow in PROD image - either from the sources ('.')
@@ -2584,16 +2509,13 @@ This is the current syntax for `./breeze <./breeze>`_:
25842509
airflow is just removed. In this case airflow package should be added to dist folder
25852510
and --use-packages-from-dist flag should be used.
25862511
2587-
2.0.3 2.0.2 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist none
2512+
2.0.3 2.0.2 2.0.1 2.0.0 wheel sdist none
25882513
25892514
--use-packages-from-dist
25902515
In CI image, if specified it will look for packages placed in dist folder and
25912516
it will install the packages after entering the image.
25922517
This is useful for testing provider packages.
25932518
2594-
--no-rbac-ui
2595-
Disables RBAC UI when Airflow 1.10.* is installed.
2596-
25972519
****************************************************************************************************
25982520
Credentials
25992521

‎CI.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ CI Environment
2121
==============
2222

2323
Continuous Integration is important component of making Apache Airflow robust and stable. We are running
24-
a lot of tests for every pull request, for master and v1-10-test branches and regularly as CRON jobs.
24+
a lot of tests for every pull request, for master and v2-0-test branches and regularly as CRON jobs.
2525

2626
Our execution environment for CI is `GitHub Actions <https://github.com/features/actions>`_. GitHub Actions
2727
(GA) are very well integrated with GitHub code and Workflow and it has evolved fast in 2019/202 to become
@@ -307,7 +307,7 @@ You can use those variables when you try to reproduce the build locally.
307307
| | | | | tested set of dependency constraints |
308308
| | | | | stored in separated "orphan" branches |
309309
| | | | | of the airflow repository |
310-
| | | | | ("constraints-master, "constraints-1-10") |
310+
| | | | | ("constraints-master, "constraints-2-0") |
311311
| | | | | but when this flag is set to anything but false |
312312
| | | | | (for example commit SHA), they are not used |
313313
| | | | | used and "eager" upgrade strategy is used |

‎IMAGES.rst

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -69,8 +69,8 @@ The images are named as follows:
6969
where:
7070

7171
* ``BRANCH_OR_TAG`` - branch or tag used when creating the image. Examples: ``master``,
72-
``v2-0-test``, ``v1-10-test``, ``2.0.0``. The ``master``, ``v1-10-test`` ``v2-0-test`` labels are
73-
built from branches so they change over time. The ``1.10.*`` and ``2.*`` labels are built from git tags
72+
``v2-0-test``, ``2.0.0``. The ``master``, ``v2-0-test`` labels are
73+
built from branches so they change over time. The ````2.*`` labels are built from git tags
7474
and they are "fixed" once built.
7575
* ``PYTHON_MAJOR_MINOR_VERSION`` - version of python used to build the image. Examples: ``3.6``, ``3.7``,
7676
``3.8``
@@ -562,7 +562,6 @@ The following build arguments (``--build-arg`` in docker build command) can be u
562562
| | | used. By default it is set to |
563563
| | | ``constraints-master`` but can be |
564564
| | | ``constraints-2-0`` for 2.0.* versions |
565-
| | | ``constraints-1-10`` for 1.10.* versions |
566565
| | | or it could point to specific version |
567566
| | | for example ``constraints-2.0.0`` |
568567
| | | is empty, it is auto-detected |

‎README.md

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -137,9 +137,8 @@ our dependencies as open as possible (in `setup.py`) so users can install differ
137137
if needed. This means that from time to time plain `pip install apache-airflow` will not work or will
138138
produce unusable Airflow installation.
139139

140-
In order to have repeatable installation, however, introduced in **Airflow 1.10.10** and updated in
141-
**Airflow 1.10.12** we also keep a set of "known-to-be-working" constraint files in the
142-
orphan `constraints-master`, `constraints-2-0` and `constraints-1-10` branches. We keep those "known-to-be-working"
140+
In order to have repeatable installation, however, we also keep a set of "known-to-be-working" constraint
141+
files in the orphan `constraints-master`, `constraints-2-0` branches. We keep those "known-to-be-working"
143142
constraints files separately per major/minor Python version.
144143
You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify
145144
correct Airflow tag/version/branch and Python versions in the URL.

‎airflow/kubernetes/pod_template_file_examples/git_sync_template.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ spec:
3131
- name: GIT_SYNC_REV
3232
value: "HEAD"
3333
- name: GIT_SYNC_BRANCH
34-
value: "v1-10-stable"
34+
value: "v2-0-stable"
3535
- name: GIT_SYNC_REPO
3636
value: "https://github.com/apache/airflow.git"
3737
- name: GIT_SYNC_DEPTH

0 commit comments

Comments
 (0)