Skip to content

Move GitDagBundle to providers #47636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ephraimbuddy merged 14 commits into from
Mar 26, 2025
Merged

Move GitDagBundle to providers #47636

ephraimbuddy merged 14 commits into from
Mar 26, 2025

Conversation

@ephraimbuddy
Copy link
Contributor

@ephraimbuddy ephraimbuddy commented Mar 11, 2025

This commit moves GitDagBundle into the providers package. As part of the move, a new GIT provider is created and like operators, transfers etc, we now have bundles package. The bundle package is where the GitDagBundles lives.

@ephraimbuddy ephraimbuddy force-pushed the add-gitbundle-provider branch 2 times, most recently from c33cd18 to c123c9b Compare March 12, 2025 12:49
@ephraimbuddy ephraimbuddy force-pushed the add-gitbundle-provider branch 3 times, most recently from 23fa0ba to 6bc6572 Compare March 13, 2025 12:43
@potiuk
Copy link
Member

potiuk commented Mar 13, 2025

I dismissed the alerts from CodeQL as false positive.

For the compat tests - git provider should only be for airflow 3 so you should also add git to the list of excluded providers in src/airflow_breeze/global_constants.py (PROVIDERS_COMPATIBILITY_TESTS_MATRIX)

@ephraimbuddy
Copy link
Contributor Author

ephraimbuddy commented Mar 13, 2025

I dismissed the alerts from CodeQL as false positive.

For the compat tests - git provider should only be for airflow 3 so you should also add git to the list of excluded providers in src/airflow_breeze/global_constants.py (PROVIDERS_COMPATIBILITY_TESTS_MATRIX)

Thanks! Almost got stuck at that! @dstandish is also fixing false positive at #47685

@ephraimbuddy ephraimbuddy force-pushed the add-gitbundle-provider branch 2 times, most recently from 0af257c to 927c584 Compare March 13, 2025 16:19
@jscheffl
Copy link
Contributor

jscheffl commented Mar 15, 2025

Looks good in my view.

Shall we add some config (don't know where) to ensure this provider is installed per default in Airflow Docker images released?

@ephraimbuddy ephraimbuddy force-pushed the add-gitbundle-provider branch from 927c584 to d29890f Compare March 18, 2025 11:31
eladkal
eladkal reviewed Mar 18, 2025
View reviewed changes
Copy link
Contributor

@eladkal eladkal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want this provider to be listed in the PRE_INSTALLED_PROVIDERS ?

@jscheffl
Copy link
Contributor

jscheffl commented Mar 18, 2025 •
edited
Loading

Do we want this provider to be listed in the PRE_INSTALLED_PROVIDERS ?

Strong YES. I assume very many people will use this as git sync is used often as standard in helm chart as well...

(which contradicts a bit with #47927 :-D)

@potiuk
Copy link
Member

potiuk commented Mar 20, 2025

Strong YES. I assume very many people will use this as git sync is used often as standard in helm chart as well...

Hmm. I think it should be rather in https://github.com/apache/airflow/blob/main/prod_image_installed_providers.txt

@jscheffl
Copy link
Contributor

jscheffl commented Mar 21, 2025

Strong YES. I assume very many people will use this as git sync is used often as standard in helm chart as well...

Hmm. I think it should be rather in https://github.com/apache/airflow/blob/main/prod_image_installed_providers.txt

Ups, okay, sorry, mis-conception on my side. Strong YES so that the install is in the Docker, don't need to be in the PIP Distribution directly.

@ephraimbuddy ephraimbuddy force-pushed the add-gitbundle-provider branch from d29890f to df69535 Compare March 21, 2025 10:14
@ephraimbuddy ephraimbuddy marked this pull request as ready for review March 21, 2025 10:17
@ephraimbuddy ephraimbuddy force-pushed the add-gitbundle-provider branch from df69535 to fb44fda Compare March 21, 2025 11:45
@potiuk
Copy link
Member

potiuk commented Mar 22, 2025

I think you need to rebase now after airflow-core move and run this:

breeze release-management prepare-provider-documentation --reapply-templates-only git

That should regenerate all the "build" documentation in provider package using the latest templates.

@ephraimbuddy ephraimbuddy force-pushed the add-gitbundle-provider branch from fb44fda to 595c3c2 Compare March 24, 2025 06:41
@ephraimbuddy ephraimbuddy force-pushed the add-gitbundle-provider branch 2 times, most recently from fcf91e9 to 92eed23 Compare March 24, 2025 21:31
@ephraimbuddy ephraimbuddy force-pushed the add-gitbundle-provider branch 2 times, most recently from 007040d to bb75e96 Compare March 25, 2025 21:36
potiuk
potiuk reviewed Mar 25, 2025
View reviewed changes
@potiuk
Copy link
Member

potiuk commented Mar 25, 2025

One small thing to fix.

@ephraimbuddy ephraimbuddy force-pushed the add-gitbundle-provider branch from bb75e96 to abef0e9 Compare March 25, 2025 22:31
eladkal
eladkal reviewed Mar 26, 2025
View reviewed changes
ephraimbuddy and others added 14 commits March 26, 2025 14:56
@ephraimbuddy
Move GitDagBundle to providers
44dc9a9 
This commit moves GitDagBundle into the providers package. As part of
the move, a new `GIT` provider is created and like `operators`, `transfers`
etc, we now have `bundles` package. The bundle package is where the `GitDagBundles`
lives.
@ephraimbuddy
fixup! Move GitDagBundle to providers
854dcb0
@ephraimbuddy
Update documentation
3cbbf70
@ephraimbuddy
Fix static checks
dcfbb43
@ephraimbuddy
add git to compatibility tests matrix
b27f8b9
@ephraimbuddy
Update doctree
4bc0fc0
@ephraimbuddy
Update with recent changes
b6db357
@ephraimbuddy
Fix doc, update version and add provider to prod image preinstalled p…
a3e564d 
…roviders
@ephraimbuddy
fixup! Fix doc, update version and add provider to prod image preinst…
f03b18e 
…alled providers
@ephraimbuddy
Fix doc
8ffd273
@ephraimbuddy
update breeze command hash
5804c7b
@ephraimbuddy
Fix incompatible providers regex removal syntax
5415db9
@ephraimbuddy
Remove providers mistakenly added in pyproject.toml
55b7c2c
@ephraimbuddy @eladkal
Update providers/git/pyproject.toml
a617218 
Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>
@ephraimbuddy ephraimbuddy force-pushed the add-gitbundle-provider branch from 5045074 to a617218 Compare March 26, 2025 13:56
@ephraimbuddy ephraimbuddy merged commit 08d9dc2 into apache:main Mar 26, 2025
148 checks passed
@ephraimbuddy ephraimbuddy deleted the add-gitbundle-provider branch March 26, 2025 15:29
@github-project-automation github-project-automation bot moved this from In Progress to Done in AIP-66: DAG Bundles & Parsing Mar 26, 2025
pankajkoti pushed a commit to astronomer/airflow that referenced this pull request Mar 28, 2025
@ephraimbuddy @pankajkoti
Move GitDagBundle to providers (apache#47636)
ee493dc 
* Move `GitDagBundle` to providers

This commit moves GitDagBundle into the providers package. As part of
the move, a new `GIT` provider is created and like `operators`, `transfers`
etc, we now have `bundles` package. The bundle package is where the `GitDagBundles`
lives.

* fixup! Move `GitDagBundle` to providers

* Update documentation

* Fix static checks

* add git to compatibility tests matrix

* Update doctree

* Update with recent changes

* Fix doc, update version and add provider to prod image preinstalled providers

* fixup! Fix doc, update version and add provider to prod image preinstalled providers

* Fix doc

* update breeze command hash

* Fix incompatible providers regex removal syntax

* Remove providers mistakenly added in pyproject.toml

* Update providers/git/pyproject.toml
shubham-pyc pushed a commit to shubham-pyc/airflow that referenced this pull request Apr 2, 2025
@ephraimbuddy @shubham-pyc
Move GitDagBundle to providers (apache#47636)
d708077 
* Move `GitDagBundle` to providers

This commit moves GitDagBundle into the providers package. As part of
the move, a new `GIT` provider is created and like `operators`, `transfers`
etc, we now have `bundles` package. The bundle package is where the `GitDagBundles`
lives.

* fixup! Move `GitDagBundle` to providers

* Update documentation

* Fix static checks

* add git to compatibility tests matrix

* Update doctree

* Update with recent changes

* Fix doc, update version and add provider to prod image preinstalled providers

* fixup! Fix doc, update version and add provider to prod image preinstalled providers

* Fix doc

* update breeze command hash

* Fix incompatible providers regex removal syntax

* Remove providers mistakenly added in pyproject.toml

* Update providers/git/pyproject.toml
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Apr 4, 2025
@ephraimbuddy @nailo2c
Move GitDagBundle to providers (apache#47636)
44af58d 
* Move `GitDagBundle` to providers

This commit moves GitDagBundle into the providers package. As part of
the move, a new `GIT` provider is created and like `operators`, `transfers`
etc, we now have `bundles` package. The bundle package is where the `GitDagBundles`
lives.

* fixup! Move `GitDagBundle` to providers

* Update documentation

* Fix static checks

* add git to compatibility tests matrix

* Update doctree

* Update with recent changes

* Fix doc, update version and add provider to prod image preinstalled providers

* fixup! Fix doc, update version and add provider to prod image preinstalled providers

* Fix doc

* update breeze command hash

* Fix incompatible providers regex removal syntax

* Remove providers mistakenly added in pyproject.toml

* Update providers/git/pyproject.toml
@eladkal eladkal mentioned this pull request Apr 10, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiuk potiuk potiuk approved these changes

@eladkal eladkal eladkal left review comments

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@gopidesupavan gopidesupavan Awaiting requested review from gopidesupavan gopidesupavan is a code owner

Assignees

No one assigned

Labels

AIP-66: DAG Bundle/Manifest area:dev-tools area:providers provider:git

Projects

AIP-66: DAG Bundles & Parsing
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ephraimbuddy @potiuk @jscheffl @eladkal @jedcunningham