Is using jinja2 for dynamically generating user code an antipattern?

[C-Loftus]

Hi all,

I am using dagster for crawling/harvesting data from many URLs for constructing a knowledge graph

• I want each website to be able to be harvested individually if needed
• The extraction operation is standardized and the same for each website but
  I don't know the list of websites to crawl until I get the list from an API call

I am using jinja2 to dynamically generate each function decorated with @ops, @jobs, @repository etc

• This moves the code generation stage outside of dagster's pipelines
• This ends up giving me 3 python files for each source (i.e. one for the job, one for the op, and one for the schedule)
• However, it makes it easy to reason about since I have separate operations in the dagster UI, not just one big dynamic crawl.

I found this easy to reason about, but am unclear if it is optimal

• There are lots of python files generated and code that is near duplicated in a scheduled step that is outside of dagster
• However, maybe this is ok since I want to be able to run the harvester for just one site at a time or all at once
  • This is the main reason I have kept it using templating. (I am still new to dagster and unclear how else to do this)
• I want my solution to be scalable if I have hundreds of URLs

Thank you! (And if anyone has dagster projects that dynamically generate their own runtime code I would be very interested!)

[C-Loftus]

And just as an fyi, I have seen https://docs.dagster.io/concepts/ops-jobs-graphs/dynamic-graphs but I am unclear if dynamic graphs generated at runtime are strictly better than generated code via templating

[cmpadden]

Hi Colton, nice to meet you! Fellow Colton here -

Your approach is interesting, and does solve the problem, but I think there are a few considerations in which dynamic graphs may have some benefit, and I'd encourage you to explore them.

Like you said, your current approach is a two-step process. First, you generate the files, and then you can orchestrate them using Dagster. I believe this introduces an unnecessary overhead and forces the user to require additional maintenance of these files, when they could likely be generated dynamically in Dagster itself. You would have the same parameters that you are using to generate the files, however, they will be used as input to your dynamic graph of operations. There are three components to consider:

• Scalability - as you scale up your number of sources, you would end up with a ton of files
• Maintainability - you now have to consider keeping your generated files up-to-date, whether that be the template changing, or a new source being added. You also want to consider where you store those files, and how it may impact
• Flexibility - with dynamic "parameterized" operations in Dagster, you can pass in parameters in the UI at runtime, for ad-hoc running of jobs. With the templated approach, this would be more difficult as you'd have to generate new files.

You may also be interested in the "factory-method" or domain specific language (DSL) approaches to building pipelines. With the former, you create a function that generates your DAG of operations. And with DSLs, you create an abstraction using something like YAML or JSON, that can then be read to create pipelines.

This factory-pattern guide is from our unreleased docs overhaul, but you may find it interesting!

• https://github.com/dagster-io/dagster/pull/23899/files

[C-Loftus]

That confirms, my understanding. Thank you for that info! Much appreciated, Colton 😃

[cmpadden]

You're welcome - best of luck!