Skip to content

Conversation

@kaxil
Copy link
Member

@kaxil kaxil commented Jan 19, 2026

When DAG files are loaded from paths that change between deployments (e.g., versioned directories), the python_callable_name in serialized DAGs changes even when the actual DAG code is unchanged.

This happens because qualname() includes the module name, which contains a hash derived from the file path via get_unique_dag_module_name(). When the path changes, the module name changes, causing:

  1. Different python_callable_name values
  2. Different dag_hash
  3. New SerializedDagModel entries on every deployment

Solution

Add exclude_module parameter to qualname() that returns only the qualified name without the module prefix:

# Before: "unusual_prefix_abc123_dag_file.my_function"
qualname(func)

# After: "my_function"  
qualname(func, exclude_module=True)

Use this in operator serialization to ensure python_callable_name is stable across deployments.

Add exclude_module parameter to qualname() that returns only the
qualified name without module prefix. This prevents unnecessary DAG
version churn when bundle paths change (e.g., version or timestamp-based folders).

The module name contains a path-derived hash that changes with each
deployment, causing new `SerializedDagModel` entries even when DAG code
is unchanged.
@kaxil kaxil force-pushed the fix-python-callable-name-stability branch from 01f2900 to d7c6c22 Compare January 19, 2026 22:56
@kaxil kaxil requested a review from vatsrahul1001 January 20, 2026 00:55
@kaxil kaxil merged commit fbb200e into apache:main Jan 20, 2026
71 checks passed
@kaxil kaxil deleted the fix-python-callable-name-stability branch January 20, 2026 13:23
jason810496 pushed a commit to jason810496/airflow that referenced this pull request Jan 22, 2026
amoghrajesh pushed a commit to astronomer/airflow that referenced this pull request Jan 22, 2026
suii2210 pushed a commit to suii2210/airflow that referenced this pull request Jan 26, 2026
ephraimbuddy pushed a commit that referenced this pull request Jan 27, 2026
kaxil added a commit that referenced this pull request Jan 27, 2026
@ephraimbuddy ephraimbuddy added the type:bug-fix Changelog: Bug Fixes label Jan 28, 2026
shreyas-dev pushed a commit to shreyas-dev/airflow that referenced this pull request Jan 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants