Skip to content

Commit

Permalink
Improve cross-provider dependency detection (apache#45784)
Browse files Browse the repository at this point in the history
When another provider is only used in example dags, we should not
make it "cross-provider" dependency - this happened to "edge" provider
having "common.compat" as cross-provider dependency.

It has been previously only implemented for "standard" provider but
this is a universal rule that could be applied to any other provider.
  • Loading branch information
potiuk authored Jan 18, 2025
1 parent 3e8449f commit d414ff9
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 7 deletions.
4 changes: 1 addition & 3 deletions generated/provider_dependencies.json
Original file line number Diff line number Diff line change
Expand Up @@ -535,9 +535,7 @@
"plugin-class": "airflow.providers.edge.plugins.edge_executor_plugin.EdgeExecutorPlugin"
}
],
"cross-providers-deps": [
"common.compat"
],
"cross-providers-deps": [],
"excluded-python-versions": [],
"state": "not-ready"
},
Expand Down
12 changes: 8 additions & 4 deletions scripts/ci/pre_commit/update_providers_dependencies.py
Original file line number Diff line number Diff line change
Expand Up @@ -206,11 +206,15 @@ def check_if_different_provider_used(file_path: Path) -> None:
warnings.append(f"The provider {imported_provider} from {file_path} cannot be found.")
continue

if imported_provider == "standard":
# Standard -- i.e. BashOperator is used in a lot of example dags, but we don't want to mark this
if "/example_dags/" in file_path.as_posix():
# If provider is used in a example dags, we don't want to mark this
# as a provider cross dependency
if file_path.name == "celery_executor_utils.py" or "/example_dags/" in file_path.as_posix():
continue
continue
if imported_provider == "standard" and file_path.name == "celery_executor_utils.py":
# some common standard operators are pre-imported in celery when it starts in order to speed
# up the task startup time - but it does not mean that standard provider is a cross-provider
# dependency of the celery executor
continue
if imported_provider:
if file_provider != imported_provider:
ALL_DEPENDENCIES[file_provider]["cross-providers-deps"].append(imported_provider)
Expand Down

0 comments on commit d414ff9

Please sign in to comment.