Workflows: Executing notebooks as a DAG?

[matthiasdv]

Note: This should be tagged as question / suggestion but I don't think I can do that myself.

I have a use case where I want to enable data scientists to execute the notebooks they create as a DAG. This would be part of their development workflow, in order to ensure a set of notebooks work as an integrated pipeline before it is scheduled in a production environment using something like Airflow.

My question; Is there currently a way to achieve this using papermill? The Readme mentions workflows, but papermill engines are the closest thing I can find to a workflow (interpreted as a pipeline of notebooks)

And if there's no such functionality, what would be the best way to integrate this with papermill, bookstore, scrapbook etc?

@MSeal , Am particularly interested in feedback if you have the time.

Some very high level details about the environment this is situated in;

• Data scientists author individual Jupyter notebooks
• When moved to production, these are scheduled as a DAG using Airflow
• Each Airflow task spawns it's own papermill kernell on a remote compute cluster; which is responsible for executing only the notebook described by the task instance.
• nteract-scrapbook collects logs and provides feedback to the task scheduler.

What I'm working on is a way the data scientists to formally define their set of notebooks as a DAG and be able to execute it during their development workflow, hopefully producing a better integrated pipeline of notebooks without requiring the data scientist to work with (or have knowledge of) the task scheduler (Airflow) and compute cluster internals.

This is similar to this component of Metaflow by Netflix;

[MSeal]

I'm not exactly sure what I can help you with here, but I can try to post some generic feedback on the ideas.

We do something very similar to this pattern at Netflix, albeit with Notebooks more in the background for most users and with the Metaflow tool still mostly executing pure python executions from the Data Scientists' pov (hoping that'll become more seamlessly integrated in the future). You've got the right jist of tools you can use for different tasks, though what you're probably missing is a control plane for the dependencies of the notebook executions. We control that by providing a base docker image that most everyone uses that has all the basics installed for the user ahead of time. That way their notebook server, dev machines, and scheduled nodes in a DAG all use the same execution context, which helps a lot with reducing support burdens even though it's not perfect.

nteract-scrapbook collects logs and provides feedback to the task scheduler.

I would be careful of collecting logs with scrapbook. It's saving the scraps within the notebook document so if you have a lot of logs you'll be saving 100's MB or even GBs of logs and will break tool that try to load the notebooks later on. If I, or someone else, ever finishes getting remote stored scraps this wouldn't be as tight of a constraint in the future.

A few other suggestions that have worked well for us -- have notebooks either be checked into source control and/or copy the notebook when a user updates their DAG. This avoids the "what did I change yesterday that broke our DAG pipeline?" as you can track each DAG version independently and lock in stable versions. How you will actually manage user notebooks will depend a lot on the tooling your users are familiar with / your infra has available.

Something else to look at is Dagster, which has built-in papermill support with it's dagstermill package, and might be a good DSL matchup for node management with data transfers. It's solids abstractions helps with linking data between nodes. I imagine with some custom code you can get to what you want with Metaflow or DAGster, which attempt to solve data processing problems from different scopes.

Hope that info helps :)

[swiftdiaries]

Thank you for the detailed reply.

[jmeisele]

Check out Apache Airflow documentation, you can most certainly use Papermill for DAGs. Here is the direct link

Papermill Operator