Automation and Freshness when the asset's update schedule doesn't match the time window partition grain #24648
-
Say we have an asset that's partitioned by day (asset A), and a downstream asset partitioned by month (asset B); This model seems to stop working as soon as we expect the monthly partitioned asset to be updated every day, progressively aggregating data data from the last day(s). Ideally I'd expect the following code block to work (possibly with the addition of an import dagster as dg
@dg.asset(
automation_condition=dg.AutomationCondition.on_cron("0 0 * * *"),
partitions_def=dg.DailyPartitionsDefinition(start_date="2024-01-01"),
)
def asset_a(): ...
@dg.asset(
automation_condition=dg.AutomationCondition.on_cron("0 0 * * *"),
partitions_def=dg.MonthlyPartitionsDefinition(start_date="2024-01-01", end_offset=1),
)
def asset_b(asset_a): ...
dg.build_time_partition_freshness_checks(assets=[asset_a, asset_b], deadline_cron="0 1 * * *") As the freshness check for partitioned assets doesn't support a delta argument, it doesn't know when the materialization was supposed to arrive, only having the partition creation to go by. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
cc @OwenKephart is a "delta" param for freshness checks like ^^ something we've considered? |
Beta Was this translation helpful? Give feedback.
Hi @askvinni! Unfortunately there are no current plans to implement what you've described along the freshness check angle, although I would expect the automation conditions you've laid out to kick off materializations properly.