Question about AutomationCondition.eager() conditions

[ssillaots-boku]

Hey!

Using dagster version 1.8.5

This is the default eager AutomationCondition

def eager() -> "AutomationCondition":
        with disable_dagster_warnings():
            return (
                AutomationCondition.in_latest_time_window()
                & (
                    AutomationCondition.newly_missing() | AutomationCondition.any_deps_updated()
                ).since_last_handled()
                & ~AutomationCondition.any_deps_missing()
                & ~AutomationCondition.any_deps_in_progress()
                & ~AutomationCondition.in_progress()
            ).with_label("eager")

I have a question about this condition specifically

(
  AutomationCondition.newly_missing() | AutomationCondition.any_deps_updated()
).since_last_handled()

• Why AutomationCondition.newly_missing().since_last_handled() is used here and not just AutomationCondition.missing().

I guess there's some kind of an edge case but I can't figure it out.

It would be great if you could provide me some context.

[OwenKephart]

Hi @ssillaots-boku ! Good question, and the reason is indeed a bit of an edge case.

For simplicity, imagine you just have a single unpartitioned asset that has never been materialized, and has some error in its definition that will cause it to fail whenever the system attempts to materialize it.

In this case, the eager() condition would become true, and a run would be launched. While that run is going, no further work would be launched (as the asset would still be in progress). However, that run will eventually terminate with an error, at which point, the asset will still be missing, and no longer be in progress. This would cause another run to be kicked off, resulting in an "infinite loop" of errors.

The newly_missing().since_last_handled() structure avoids this by "remembering" that a run was already requested for this partition (since_last_handled is just shorthand for since(updated() | requested()).

[ssillaots-boku]

Thank you, @OwenKephart! Great explanation.

[ssillaots-boku]

Follow-up question @OwenKephart
What's the difference between newly_missing().since_last_handled() and newly_missing()?
Only edge case I can think of is when upstream is materialized and downstream gets wiped.
Is that correct?
Are there more cases that it covers?

[OwenKephart]

Hi @ssillaots-boku! newly_missing() is only true for the single evaluation after an asset partition becomes missing. Adding since_last_handled() makes this stay true for every evaluation until a run is requested. This is useful in cases where a run cannot be immediately requested in response to a partition being missing (e.g. if a parent asset is missing). In these cases, the system needs to "remember" that there's something that needs to be responded to.