Skip to content

Conversation

@Lee-W
Copy link
Member

@Lee-W Lee-W commented Nov 12, 2025

Why

Closes: #56451

Back in 3.0.6, we use to have LazyDeserializedDAG.get_run_data_interval

def get_run_data_interval(self, run: DagRun) -> DataInterval | None:
"""Get the data interval of this run."""
if run.dag_id is not None and run.dag_id != self.dag_id:
raise ValueError(f"Arguments refer to different DAGs: {self.dag_id} != {run.dag_id}")
data_interval = _get_model_data_interval(run, "data_interval_start", "data_interval_end")
if data_interval is None and run.logical_date is not None:
data_interval = self._real_dag.timetable.infer_manual_data_interval(run_after=run.logical_date)
return data_interval

and DAG.get_run_data_interval

def get_run_data_interval(self, run: DagRun) -> DataInterval:
"""
Get the data interval of this run.
For compatibility, this method infers the data interval from the DAG's
schedule if the run does not have an explicit one set, which is possible for
runs created prior to AIP-39.
This function is private to Airflow core and should not be depended on as a
part of the Python API.
:meta private:
"""
if run.dag_id is not None and run.dag_id != self.dag_id:
raise ValueError(f"Arguments refer to different DAGs: {self.dag_id} != {run.dag_id}")
data_interval = _get_model_data_interval(run, "data_interval_start", "data_interval_end")
if data_interval is not None:
return data_interval
# Compatibility: runs created before AIP-39 implementation don't have an
# explicit data interval. Try to infer from the logical date.
return self.infer_automated_data_interval(run.logical_date)

They called when we collect Dags.

last_automated_data_interval = dag.get_run_data_interval(last_automated_run)

The dag here is typed as MaybeSerializedDAG which is

MaybeSerializedDAG = Union[DAG, "LazyDeserializedDAG"]

Thus, both LazyDeserializedDAG.get_run_data_interval and DAG.get_run_data_interval will be called back in 3.0.6

However, in 3.1.0, we removed the MaybeSerializedDAG in https://github.com/apache/airflow/pull/54383/files#diff-2f03f28d3201c630dfbae758caf95ada61ab10328de789fc464f6e27c4afb2d0L2687 which make

last_automated_data_interval = get_run_data_interval(dag.timetable, last_automated_run)

runs only the logic in DAG.get_run_data_interval.

def get_run_data_interval(timetable: Timetable, run: DagRun) -> DataInterval:
"""
Get the data interval of this run.
For compatibility, this method infers the data interval from the DAG's
schedule if the run does not have an explicit one set, which is possible for
runs created prior to AIP-39.
This function is private to Airflow core and should not be depended on as a
part of the Python API.
:meta private:
"""
data_interval = _get_model_data_interval(run, "data_interval_start", "data_interval_end")
if data_interval is not None:
return data_interval
# Compatibility: runs created before AIP-39 implementation don't have an
# explicit data interval. Try to infer from the logical date.
return infer_automated_data_interval(timetable, run.logical_date)

What

Since we now only have get_run_data-interval, we should consider both LazyDeserializedDAG.get_run_data_interval and DAG.get_run_data_interval logic


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@Lee-W Lee-W force-pushed the handle-pre-AIP-39-dagruns branch from 53677c9 to 91472c8 Compare November 12, 2025 11:16
@Lee-W Lee-W marked this pull request as ready for review November 12, 2025 11:16
@Lee-W Lee-W requested review from XD-DENG and ashb as code owners November 12, 2025 11:16
@Lee-W Lee-W force-pushed the handle-pre-AIP-39-dagruns branch from 91472c8 to 9741703 Compare November 12, 2025 13:07
@Lee-W
Copy link
Member Author

Lee-W commented Nov 12, 2025

will add some test tomorrow but would appreciate some early review

@Lee-W Lee-W force-pushed the handle-pre-AIP-39-dagruns branch from 9741703 to 67ce18f Compare November 13, 2025 09:45
@Lee-W Lee-W requested a review from uranusjr November 13, 2025 10:49
@Lee-W Lee-W added this to the Airflow 3.1.4 milestone Nov 14, 2025
@Lee-W Lee-W force-pushed the handle-pre-AIP-39-dagruns branch from 67ce18f to 8f25634 Compare November 16, 2025 03:20
@Lee-W Lee-W force-pushed the handle-pre-AIP-39-dagruns branch from 8f25634 to 414f7a8 Compare November 25, 2025 08:58
@Lee-W Lee-W force-pushed the handle-pre-AIP-39-dagruns branch from 414f7a8 to 1f9cbb4 Compare November 26, 2025 09:52
@Lee-W Lee-W force-pushed the handle-pre-AIP-39-dagruns branch from 1f9cbb4 to 67af8ed Compare November 26, 2025 10:15
@Lee-W Lee-W requested a review from ephraimbuddy November 27, 2025 01:58
@Lee-W Lee-W added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Nov 27, 2025
@Lee-W Lee-W merged commit 1ce068b into apache:main Nov 27, 2025
65 checks passed
@Lee-W Lee-W deleted the handle-pre-AIP-39-dagruns branch November 27, 2025 13:58
github-actions bot pushed a commit that referenced this pull request Nov 27, 2025
(cherry picked from commit 1ce068b)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@github-actions
Copy link

Backport successfully created: v3-1-test

Status Branch Result
v3-1-test PR Link

Lee-W added a commit that referenced this pull request Nov 27, 2025
(cherry picked from commit 1ce068b)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Lee-W added a commit that referenced this pull request Nov 28, 2025
(cherry picked from commit 1ce068b)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Lee-W added a commit that referenced this pull request Nov 28, 2025
ephraimbuddy pushed a commit that referenced this pull request Dec 3, 2025
RoyLee1224 pushed a commit to RoyLee1224/airflow that referenced this pull request Dec 3, 2025
Copilot AI pushed a commit to jason810496/airflow that referenced this pull request Dec 5, 2025
itayweb pushed a commit to itayweb/airflow that referenced this pull request Dec 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dag Processor fails with "ValueError: Not a valid timetable"

2 participants