awesome-dagster PyPI package

[sspaeti]

Motivation

As open-source is a must these days, I'd like to collaborate or share code as much as possible at one single place. We should not re-create the same code twice due to stability and also efficiency. As well it speeds up every deployment by x-times as you chose from awesome-dagster what you need, kind of like plugins. Solids or resources would make the most sense to me, but also types and others.

Proposal

I created an awesome-dagster PyPI package at work where we have all our generic types, solids and resources. Every workspace or user-code-repo can use these components by importing these with from awesome-dagster.solids import load_delta_table_to_data_frame for example. It works well, so I could imagine doing this in a dedicated dagster-repo where everyone is allowed to push their code with certain naming conventions.

I'm not sure if a PyPI package is a right way or what's possible farther, but this approach has versioning included and an easy way to test and import it into our pipelines. The other option would be an awesome-github-repo as other doing, e.g. awesome-pipeline where it's more like a reference, and everyone would copy the code over to their repos. But I'd definitely prefer something like a PyPI package to import it easily.

What do you think? And how should we start something like this?

[mrdavidlaing]

@sspaeti - I like your idea; but am struggling to find the awesome-dagster PyPI package.

Can you share a link / source repo?

[sspaeti]

I started a small initial list here: https://github.com/sspaeti-com/awesome-dagster. It is not much yet, but feel free to add. Whenever I have time, I will add more as well.

[mrdavidlaing]

AWESOME! Thanks