Move dlt educational courses into core repo

[sh-rp]

TLDR

We have two really cool educational courses in our dlthub-education repository: https://github.com/dlt-hub/dlthub-education:

• dlt fundamentals: https://github.com/dlt-hub/dlthub-education/tree/main/courses/dlt_fundamentals_dec_2024
• dlt advanced: https://github.com/dlt-hub/dlthub-education/tree/main/courses/dlt_advanced_2025

It would be cool to store them in our OSS repo, this way

• they should be easier to find for llm crawlers,
• we can lint them and keep them up to date
• we can create marimo versions
• we can launch colabs and possibly molabs (the marimo colab version) directly from the repo links.
• The users will also be able to run them locally if they want to.

Research phase

Let's do a test colab on a branch in our repo with the first one where check if the below steps work and discuss the outcome before we do all of them. For this,

• take the first colab and export it
• put it in our repo on a branch
• try to launch it directly in colab from the repo
• See what happens with the images and other content, we should only have code in the repo and might need to host images some place.

Implementation

Let's implement this in three phases, one PR each:

1. Copy colabs to oss repo

• Copy all colabs of the two courses (we can add more later) to the core repo as IPYNB files. I would propose to add them to docs/education/dlt-fundamentals-course and docs/education/dlt-advanced-course. Give them useful filenames derived from the topics.
• Create new docs pages where these are linked. Please talk to @AstrakhantsevaAA about where they go in the docs structure. Maybe getting started -> education? Those pages should look like the ones in the repo above with a list of links to directly open the colabs, but these links should open the colabs directly from the github version. (Example link: https://colab.research.google.com/github/dagworks-inc/hamilton/blob/main/examples/caching/tutorial.ipynb)
• Add instructions on how to run these colabs locally with jupyter
• You may need to exclude iypthon files from our max file size checker

2. Quality assurance and updated links

• Add linting to the colabs by creating a lint-notebooks command to the main makefile which lints and typechecks them. We need quite lenient linter rules, not much should have to be changed. We want to catch breaking changes that happen in the core library so that our notebooks stay up to date. @zilto knows how to lint ipython notebooks.
• Try to update all places where we link to our old colabs to use these new github links. This would be in the education repository and also in our learnworlds platform (talk to @AstrakhantsevaAA about this)

3. Marimo integration / support

Marimo has a converter that let's you convert IPYNB files into marimo notebooks. We can use this to generate marimo versions of each notebook. Test a few to see that they actually work before you fully convert all of them.

• Convert all notebooks that you copied into the repo in step 2 into marimo notebooks. Both versions should live side by side in the same folder.
• Add instructions on how to run those notebooks locally in edit mode to the docs pages you create in part 1
• Add linting and formatting to these marimo notebooks. They are regular python files, so it works the same way as for the rest of the codebase. Typechecking needs to be quite lenient again probably.
• Ideally we have a CI step that makes sure both IPYNB files and marimo notebooks stay in sync. This is optional, we can talk about it if we have time.
• Marimo will probably add a feature to molab (marimo colab) that would also allow us to launch there notebooks directly from github in molab, we'll have to see if this is available once we are at this step.

[adrianbr]

Comment on
"Add instructions on how to run these colabs locally with jupyter"
I think we should provide a colab link too, otherwise we force data engineer learners into a workflow and tooling they might not be happy to use. Running jupyter notebooks is a nice extra option but doesn't replace the ease of diving in of colabs

Comment on
"Marimo integration / support"

• we should ensure this puts users into our "workspace" workflows and look and feel, so I am not sure if this is default Marimo or Workspace app. We should not be forcing anyone to sign up for a 3rd party service to run code in a notebook.

[sh-rp]

@adrianbr

1. the local jupyter instructions are added additionally. We provide links firstly and foremost that open the notebook from the github source directly in colab.
2. I agree, but this would probably only work if we do not use molab but use local marimo execution, there we can simply add the same or a very similar css file as we have for the dashboard

[AstrakhantsevaAA]

Thanks Dave for such a detailed description! I have a few questions:

1. Not sure how LLM crawlers will parse the content, because I don't know how these crawlers work exactly. Do you think @sh-rp it's enough to store ipynbs in the repo? Or maybe it's better to convert them into html pages as @zilto mentioned? Or should we do both?
2. Do you think it's necessary to store ipynb in the repo? Repo will be heavy, some of these notebooks are 1.5+mb. Maybe we can store them in dlthub-education repo and just link these pages to dlthub.com/docs/education?
3. At step 2: will we have Colabs and ipynb both? How would look the updating process of these ipynb and colabs?
4. At step 3: do you want to host all the colabs so users could run them online (similar to colabs)?

[sh-rp]

@AstrakhantsevaAA

1. I think they should be able to understand notebook files. But if not, the marimo versions are plain python files that should work
2. I think so, we should just move out the images to some place, I'll check where they are actually stored if I download this file, I hadn't thought of that. If we keep them in the education repo, they will be in a repo with not enough stars for the crawlers to care. The last update i know of is that only repos with more than 1k stars are crawled.
3. If the ipynb files are in the repo, you can create a special link that will open this exact file directly in colab. @anuunchin will have to test this too, but I think the source of truth will be our repo and then you directly open them in colab. Let's see if this is good or not.
4. No, we do not host any notebooks, just the code.

[zilto]

@adrianbr

"Add instructions on how to run these colabs locally with jupyter"

Google Colab supports opening a notebook directly from GitHub. This is what happens in this notebook. There are two clickable badges: one to go to Colab, one to go to GitHub. The source code is stored on Github.

To make them usable, we also have to lint / verify that all required dependencies are installed by the notebook cell

[AstrakhantsevaAA]

@zilto interesting! they attach it as a link: https://github.com/apache/hamilton/blob/main/examples/caching/tutorial.ipynb?short_path=c4d4116#L18C2-L18C217

Hamilton caching tutorial [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dagworks-inc/hamilton/blob/main/examples/caching/tutorial.ipynb)

[AstrakhantsevaAA]

something like this: https://www.youtube.com/watch?v=M6zy5QS9Qao

upd: another interesting approach

Open the Github notebook that you want to work with in Google Colab. Change the domain from 'github.com' to 'githubtocolab.com'. The notebook will open in Colab.

[sh-rp]

@zilto good point about the dependencies, I did some quick tests and I am not quite sure yet how to solve this. They also don't get properly ported over to marimo with the marimo converter (although marimo will install a few dependencies automatically)

I also saw that when exporting those colabs they will inline the images, that is of no use to us because it bloats the repo. This we also need to solve.

[AstrakhantsevaAA]

images there are mostly memes. so we can remove them at all. but I'll be upset