Skip to content

Enable back iterative development of latest providers with old airflows #43617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 3 commits into from
Nov 4, 2024
+17 −2
Merged

Enable back iterative development of latest providers with old airflows #43617

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bd49e7c
Enable back iterative development of latest providers with old airflows
potiuk Nov 3, 2024
07bf348
Update Dockerfile.ci
potiuk Nov 4, 2024
8c7fd56
Update scripts/docker/entrypoint_ci.sh
potiuk Nov 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions Dockerfile.ci
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1017,6 +1017,13 @@ function determine_airflow_to_use() {
--constraint https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt
# Some packages might leave legacy typing module which causes test issues
pip uninstall -y typing || true
if [[ ${LINK_PROVIDERS_TO_AIRFLOW_PACKAGE=} == "true" ]]; then
echo
echo "${COLOR_BLUE}Linking providers to airflow package as we are using them from mounted sources.${COLOR_RESET}"
echo
rm -rf /usr/local/lib/python${PYTHON_MAJOR_MINOR_VERSION}/site-packages/airflow/providers
ln -s "${AIRFLOW_SOURCES}/providers/src/airflow/providers" "/usr/local/lib/python${PYTHON_MAJOR_MINOR_VERSION}/site-packages/airflow/providers"
fi
fi

if [[ "${USE_AIRFLOW_VERSION}" =~ ^2\.2\..*|^2\.1\..*|^2\.0\..* && "${AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=}" != "" ]]; then
Expand Down
5 changes: 3 additions & 2 deletions scripts/ci/docker-compose/providers-and-tests-sources.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ services:
tty: true # docker run -t
environment:
- AIRFLOW__CORE__PLUGINS_FOLDER=/files/plugins
- LINK_PROVIDERS_TO_AIRFLOW_PACKAGE=true
# We only mount tests folder
volumes:
- ../../../.bash_aliases:/root/.bash_aliases:cached
Expand All @@ -30,8 +31,8 @@ services:
- ../../../empty:/opt/airflow/airflow
# but keep tests
- ../../../tests/:/opt/airflow/tests:cached
# and providers
- ../../../providers/src/airflow/providers:/opt/airflow/airflow/providers:cached
# Mount providers to make sure that we have the latest providers - both tests and sources
- ../../../providers/:/opt/airflow/providers:cached
# and entrypoint and in_container scripts for testing
- ../../../scripts/docker/entrypoint_ci.sh:/entrypoint
- ../../../scripts/in_container/:/opt/airflow/scripts/in_container
7 changes: 7 additions & 0 deletions scripts/docker/entrypoint_ci.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,13 @@ function determine_airflow_to_use() {
--constraint https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt
# Some packages might leave legacy typing module which causes test issues
pip uninstall -y typing || true
if [[ ${LINK_PROVIDERS_TO_AIRFLOW_PACKAGE=} == "true" ]]; then
echo
echo "${COLOR_BLUE}Linking providers to airflow package as we are using them from mounted sources.${COLOR_RESET}"
echo
rm -rf /usr/local/lib/python${PYTHON_MAJOR_MINOR_VERSION}/site-packages/airflow/providers
ln -s "${AIRFLOW_SOURCES}/providers/src/airflow/providers" "/usr/local/lib/python${PYTHON_MAJOR_MINOR_VERSION}/site-packages/airflow/providers"
fi
fi

if [[ "${USE_AIRFLOW_VERSION}" =~ ^2\.2\..*|^2\.1\..*|^2\.0\..* && "${AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=}" != "" ]]; then
Expand Down