Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions contributing-docs/11_provider_packages.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ Airflow 2.0 is split into core and providers. They are delivered as separate pac
Where providers are kept in our repository
------------------------------------------

Airflow Providers are stored in the same source tree as Airflow Core (under ``airflow.providers``) package. This
means that Airflow's repository is a monorepo, that keeps multiple packages in a single repository. This has a number
Airflow Providers are stored in a separate tree other than the Airflow Core (under ``providers`` directory).
Airflow's repository is a monorepo, that keeps multiple packages in a single repository. This has a number
of advantages, because code and CI infrastructure and tests can be shared. Also contributions are happening to a
single repository - so no matter if you contribute to Airflow or Providers, you are contributing to the same
repository and project.
Expand All @@ -49,7 +49,7 @@ dependencies including runtime dependencies. See `local virtualenv <../07_local_
for more information.

Therefore, until we can introduce multiple ``pyproject.toml`` for providers information/meta-data about the providers
is kept in ``provider.yaml`` file in the right sub-directory of ``airflow\providers``. This file contains:
is kept in ``provider.yaml`` file in the right sub-directory of ``providers``. This file contains:

* package name (``apache-airflow-provider-*``)
* user-facing name of the provider package
Expand Down Expand Up @@ -89,6 +89,19 @@ and pre-commit will generate new entry in ``generated/provider_dependencies.json
``pyproject.toml`` so that the package extra dependencies are properly handled when package
might be installed when breeze is restarted or by your IDE or by running ``pip install -e ".[devel]"``.

Chicken-egg providers
---------------------

Sometimes, when a provider depends on another provider, and you want to add a new feature that spans across
two providers, you might need to add a new feature to the "dependent" provider, you need
to add a new feature to the "dependency" provider as well. This is a chicken-egg problem and by default
some CI jobs (like generating PyPI constraints) will fail because they cannot use the source version of
the provider package. This is handled by adding the "dependent" provider to the chicken-egg list of
"providers" in ``dev/breeze/src/airflow_breeze/global_constants.py``. By doing this, the provider is build
locally from sources rather than downloaded from PyPI when generating constraints.

More information about the chicken-egg providers and how release is handled can be found in
the `Release Provider Packages documentation <../dev/README_RELEASE_PROVIDER_PACKAGES.md#chicken-egg-providers>`_

Developing community managed provider packages
----------------------------------------------
Expand Down
9 changes: 9 additions & 0 deletions dev/README_RELEASE_PROVIDER_PACKAGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,15 @@ release should get it changed to `>= x.y.z`. This is a rare case and should be h
We call such case chicken-egg providers as it's not clear who should be released first - the provider or
the Airflow.

Similar case is when provider depends on another provider (usually `common.*`) that is not yet released
because you it contains new feature and you want to release the providers together.

In such case the "common" provider should be added to the list of "chicken-egg" providers in the
`./dev/breeze/src/airflow_breeze/global_constants.py` file and version of the provider should be bumped
in the PR by author of the PR that adds both the functionality. This will make sure that the
provider is build in CI from sources, when CI jobs are run rather than latest version downloaded from PyPI
when constraints are generated.

# Prepare Regular Provider packages (RC)

## Move provider into remove state
Expand Down