How do I limit the concurrency for an asset backfill?

[sryza]

An asset backfill is typically composed of a set of runs that each correspond to one of the partitions being backfilled.

You can limit the number of runs that execute concurrently by configuring a run queue:

1. How to set up a run queue for Dagster Cloud
2. How to set up a run queue for Dagster OSS deployments

Once you've done so, you can use tag_concurrency_limits in the run queue configuration to control concurrency.

Example: set a limit of 4 concurrent runs across all backfills

    tag_concurrency_limits:
      - key: "dagster/backfill"
        limit: 4

Example: set a limit of 4 concurrent runs within each backfill

    tag_concurrency_limits:
      - key: "dagster/backfill"
        limit: 4
        applyLimitPerUniqueValue: true

Example: set a limit of 4 concurrent runs for backfills with a particular tag

    tag_concurrency_limits:
      - key: "my_limited_backfill_tag"
        limit: 4

When launching a backfill from the Dagster UI, you can supply a set of tags in the backfill dialog.

[ytoast]

Hi, thank you for this write up.

When launching a backfill from the Dagster UI, you can supply a set of tags in the backfill dialog.

Do you mean this dialog?

[ytoast]

thank you!

[Smoss]

@sryza What should tag key and tag value be set to?

[sryza]

In the first two examples in my original answer, you don't need to set them to anything. You'd set them if some subset of your backfills are a special kind of backfill that you want to set specific limits on. E.g. tag key = "big_backfill" and tag value = "true".

[the4thamigo-uk]

@sryza Is there a way to do this programmatically. Ideally I want to be able to limit the number of concurrenct backfills to a backend API, but relying on the user to remember to set a custom tag is problematic.

[sryza]

Would the first example work for you?

[n-elia]

When launching a backfill from the Dagster UI, you can supply a set of tags in the backfill dialog.

Hi! Is there any way to make Backfill Runs inherit the Asset's tags?
E.g. I would like that Backfill Runs inherit the tags of an Asset defined like this, when triggering Backfills run from the UI:

@asset(
    partitions_def=time_partition,
    tags={"data_type": "historical"},
)

Is that possible?

[sryza]

This isn't currently possible, but here's an issue where we're tracking similar requests: #8979.

[n-elia]

Thank you! That's exactly what I imagined!
May I ask you to clarify when the tag_concurrency_limits are actually applied to assets materialization?

[sryza]

Are you able to say a little more about what you mean by that?

[n-elia]

For sure!
Let's suppose that I want to limit the concurrency of the Runs based on their assets' tags.
Is that possible, using the Asset's tags defined as in the above example?
In other words, I want to do what the docs say here, but I can't manage to do it because the runs that are spawned by manually triggering the materialization of the Asset do not inherit the Asset's tags.
Edit: if that is not possible, I think that it is worth mentioning it in the documentation. The docs states "Limits can be specified for all ops/assets with a specific tag key or key-value pair. If any limit would be exceeded by launching an op/asset, then the op/asset will be queued.": I would explicitly warn the user about edge cases , e.g. when triggering backfills.

[ClaytonSmith]

This feature gap needs made extremely clear in the documentation. So many people expect mechanics to limit recourse usage, doubly so for the expectation that tag_concurrency_limits would use asset level tags.

I've spent waaayyy too much time trying to find a solution for this. At this time you have to manually regulate your backfills.

[n-elia]

I agree. Assets should work like jobs. Jobs indeed propagate tags.

[shenrun]

Hi everyone, is there any progress?