Run the same Dagster job with different run_configs in parallel

[bruk0]

Hi, I need to create a Dagster pipeline with embedded-elt(dlt). I am looking for an information about how to run the same Dagster pipeline (Dagster job) with different run_configs and schedules.

And how to run that with different run_configs in parallel

[garethbrickman]

Check out this section in our docs for Specifying runtime configuration

[bruk0]

Thank you, @garethbrickman for your response. As far as I understood, the execute_in_process() works for Ops and materialize() - for Assets.

In that case, how to define materialize() in jobs=[] in Definitions class?

Sorry, for asking stupid questions, I need to adapt Dagster for our needs, and there is no specific info or tutorials in docs.

[garethbrickman]

Jobs are distinct between ops and assets. See here for defining asset jobs.

Once your jobs are defined you can define schedules to run them.

[bruk0]

@garethbrickman here's a picture what use case I have and what I want to implement