Skip to content

Partition enhancements for peripheral features#62463

Open
uranusjr wants to merge 9 commits intoapache:mainfrom
astronomer:dag-model-next-partition-key
Open

Partition enhancements for peripheral features#62463
uranusjr wants to merge 9 commits intoapache:mainfrom
astronomer:dag-model-next-partition-key

Conversation

@uranusjr
Copy link
Member

@uranusjr uranusjr commented Feb 25, 2026 •
edited
Loading

This adds multiple fields on DagModel to support various introspection features on the dag’s next scheduled run. Those fields are populated when the scheduler calculates when the dag should be run (time-based), and can be retrieved via a CLI similar to how logical-date-based scheduling can be introspected.

A new flag partitioned is added to timetable classes, and transferred to DagModel during dag-processing, so we can cheaply determine how a dag is being scheduled.

I also added partition_date on DagRun since this seems to fit everything else. I’ll remove it if that’s not the right approach.

Fix #58445. Fix #61086. Fix #61076. Close #61319 (this includes the message change contained in the PR).

@boring-cyborg boring-cyborg bot added area:CLI area:DAG-processing area:db-migrations PRs with DB migration area:Scheduler including HA (high availability) scheduler kind:documentation labels Feb 25, 2026
@uranusjr uranusjr requested a review from Lee-W February 25, 2026 09:24
Lee-W
Lee-W reviewed Feb 25, 2026
View reviewed changes
airflow-core/src/airflow/cli/commands/dag_command.py
from airflow.models.serialized_dag import SerializedDagModel

if args.table and args.field:
raise SystemExit("Cannot use --table and --field together")
Copy link
Member

@Lee-W Lee-W Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit surprised we don't support mutually exclusive by default, and noticed the messages are quite different for such cases. not something we should change in the PR though

Copy link
Member Author

@uranusjr uranusjr Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was surprised too

@Lee-W
Copy link
Member

Lee-W commented Feb 25, 2026

This PR also closes #58445

@uranusjr uranusjr force-pushed the dag-model-next-partition-key branch 2 times, most recently from d1122a1 to 4b73560 Compare February 26, 2026 05:52
@uranusjr uranusjr force-pushed the dag-model-next-partition-key branch from 4b73560 to 78b1022 Compare February 26, 2026 07:52
@uranusjr uranusjr marked this pull request as ready for review February 26, 2026 09:28
@uranusjr uranusjr force-pushed the dag-model-next-partition-key branch from 78b1022 to 3a8d1b3 Compare February 26, 2026 09:29
@uranusjr
Copy link
Member Author

uranusjr commented Feb 26, 2026

Alright I think this is going to work (or at least close). Not sure what other tests we can add, suggestions welcome.

@uranusjr uranusjr force-pushed the dag-model-next-partition-key branch from 3a8d1b3 to 0bd9e87 Compare February 26, 2026 11:23
@Lee-W
Copy link
Member

Lee-W commented Feb 26, 2026

cc @guan404ming with this change, I think we can improve some things in the API layer as well

@uranusjr uranusjr force-pushed the dag-model-next-partition-key branch from 0bd9e87 to 67703ed Compare February 26, 2026 14:23
@uranusjr uranusjr requested a review from guan404ming February 26, 2026 14:24
@guan404ming
Copy link
Member

guan404ming commented Feb 26, 2026

cc @guan404ming with this change, I think we can improve some things in the API layer as well

Thanks, will take a look tomorrow!

guan404ming
guan404ming reviewed Feb 27, 2026
View reviewed changes
Copy link
Member

@guan404ming guan404ming left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some nit comments, overall looks good!

@uranusjr
Copy link
Member Author

uranusjr commented Feb 28, 2026 •
edited
Loading

Hm, while removing the logic from CronPartitionTimetable, somehow key_format is no longer used anywhere? That doesn’t feel right—somewhere we are supposed to convert partition_date to patition_key (or the other way around). Something is amiss here.

@uranusjr
Copy link
Member Author

uranusjr commented Feb 28, 2026

Ah nevermind, I found one.

@uranusjr uranusjr force-pushed the dag-model-next-partition-key branch from 0fd67ba to 20e1097 Compare February 28, 2026 07:45
@uranusjr uranusjr force-pushed the dag-model-next-partition-key branch from 20e1097 to 5bbb170 Compare February 28, 2026 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Lee-W Lee-W Lee-W left review comments

@guan404ming guan404ming guan404ming left review comments

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner

@bugraoz93 bugraoz93 Awaiting requested review from bugraoz93 bugraoz93 is a code owner

@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner

@dheerajturaga dheerajturaga Awaiting requested review from dheerajturaga dheerajturaga is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin bolkedebruin is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

@kaxil kaxil Awaiting requested review from kaxil kaxil is a code owner

@amoghrajesh amoghrajesh Awaiting requested review from amoghrajesh amoghrajesh is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

area:CLI area:DAG-processing area:db-migrations PRs with DB migration area:Scheduler including HA (high availability) scheduler kind:documentation

Projects

None yet

Milestone

No milestone

3 participants

@uranusjr @Lee-W @guan404ming