Skip to content

Conversation

@kacpermuda
Copy link
Contributor

@kacpermuda kacpermuda commented Nov 25, 2025

TLDR

Automatically inject OpenLineage lineage metadata into triggered DAG run conf when OpenLineage is available and enabled, improving lineage tracking across DAG boundaries.. The implementation never modifies non-OpenLineage parts of conf, gracefully handles all error cases, and preserves original conf if anything goes wrong. Users can opt out by setting openlineage_inject_parent_info=False. Because it's safe and only works for users with enabled OL we can keep default=True.

Built on top of OL feature implemented in #57809

User Impact by Scenario

All Users

  • Non-OpenLineage configuration: Never modified under any circumstances
  • Error handling: If any exception occurs during injection (import errors, unexpected errors), the original conf is returned and a warning is logged
  • Opt-out: Set openlineage_inject_parent_info=False to completely disable this feature

Users with OpenLineage disabled or not installed

  • Conf: DAG run conf is never modified
  • Overhead: Only a lightweight check to detect if OpenLineage is available (debug-level log message)
  • Behavior: Function returns immediately with original configuration unchanged

Users on older OpenLineage Provider versions (< 2.8.0)

  • Conf: DAG run conf is never modified
  • Logging: An informational log message indicates that injection is not supported and suggests upgrading to apache-airflow-providers-openlineage>=2.8.0
  • Behavior: Function catches version incompatibility and returns original configuration unchanged

Users on compatible OpenLineage Provider versions (>= 2.8.0)

  • Conf: OpenLineage metadata is added to DagRun conf under the openlineage key
  • Existing OpenLineage keys: If you already have openlineage part in your conf, they are preserved and merged (parent/root identifiers are not overwritten if they already exist)
  • Behavior: Metadata about the parent task (run ID, job name, namespace, root run info) is injected to enable lineage tracking

Testing

Added comprehensive test coverage for both Airflow 2 and Airflow 3:

  • Injection disabled when flag is False
  • Provider not accessible scenarios
  • Version compatibility checks (older vs newer versions)
  • Exception handling during injection
  • Successful injection with various configuration types (None, empty dict, dict with existing keys)

All tests follow consistent patterns with parametrization and helper functions for maintainability.

Also saved trigger_run_id as TriggerDagRunOperator's attribute to be used by OL listener.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@kacpermuda
Copy link
Contributor Author

The deferral mode may not work until this is merged.

@kacpermuda
Copy link
Contributor Author

Sorry for the noise, apparently any change to dev/breeze/tests/test_selective_checks.py triggers a lot of codeowners.

@kacpermuda kacpermuda force-pushed the feat-ol-trigger-dagrun-op branch from caf8166 to 8496661 Compare November 25, 2025 16:02
…conf

When OpenLineage is enabled, add injection of OpenLineage parent task metadata into triggered DAG run conf to enable improved lineage tracking across DAG boundaries.
@mobuchowski mobuchowski merged commit 80cb4c6 into apache:main Nov 26, 2025
120 checks passed
@kacpermuda kacpermuda deleted the feat-ol-trigger-dagrun-op branch November 26, 2025 12:20
Copilot AI pushed a commit to jason810496/airflow that referenced this pull request Dec 5, 2025
…conf (apache#58672)

When OpenLineage is enabled, add injection of OpenLineage parent task metadata into triggered DAG run conf to enable improved lineage tracking across DAG boundaries.
itayweb pushed a commit to itayweb/airflow that referenced this pull request Dec 6, 2025
…conf (apache#58672)

When OpenLineage is enabled, add injection of OpenLineage parent task metadata into triggered DAG run conf to enable improved lineage tracking across DAG boundaries.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants