Automation and Freshness when the asset's update schedule doesn't match the time window partition grain

[askvinni]

Say we have an asset that's partitioned by day (asset A), and a downstream asset partitioned by month (asset B); AutomationConditions and freshness_checks work wonderfully as long as the assets run only once; the monthly partition in asset B only being created after the month is over, aggregating all days of the upstream daily partitioned asset.

This model seems to stop working as soon as we expect the monthly partitioned asset to be updated every day, progressively aggregating data data from the last day(s).

Ideally I'd expect the following code block to work (possibly with the addition of an AssetDep and PartitionMapping to support this type of use case.

import dagster as dg


@dg.asset(
    automation_condition=dg.AutomationCondition.on_cron("0 0 * * *"),
    partitions_def=dg.DailyPartitionsDefinition(start_date="2024-01-01"),
)
def asset_a(): ...


@dg.asset(
    automation_condition=dg.AutomationCondition.on_cron("0 0 * * *"),
    partitions_def=dg.MonthlyPartitionsDefinition(start_date="2024-01-01", end_offset=1),
)
def asset_b(asset_a): ...


dg.build_time_partition_freshness_checks(assets=[asset_a, asset_b], deadline_cron="0 1 * * *")

As the freshness check for partitioned assets doesn't support a delta argument, it doesn't know when the materialization was supposed to arrive, only having the partition creation to go by.

[smackesey]

cc @OwenKephart is a "delta" param for freshness checks like ^^ something we've considered?

[askvinni]

Would be great to hear more about this. Otherwise we'll develop and vendor our own in-house solution.

[OwenKephart]

Hi @askvinni! Unfortunately there are no current plans to implement what you've described along the freshness check angle, although I would expect the automation conditions you've laid out to kick off materializations properly.

[askvinni]

Appreciate the response Owen! The automation conditions do indeed work as expected with the snippet above, but I was mostly concerned with the freshness checks. Sorry for the confusion.