SOW for content fetching from non-git sources for nbgitpuller

[yuvipanda]

nbgitpuller components

Note

Draft in progress here

It is useful to think about the various abstract functional components nbgitpuller has, so we can separate the various improvements and changes we need to make. It has five functional parts:

1. Fetch content (currently this is git only)
2. Resolve conflicts between changes in the upstream content and the user's changes (uses git, currently entangled with (1))
3. A web UI to show progress of this fetching and conflict resolution
4. A commandline UI to automatically do fetching and conflict resolution
5. A web UI to generate links (https://nbgitpuller.readthedocs.io/en/latest/link.html)

This is one of 4 separate but related SOWs we are producing, so they can be discussed, funded and worked on clearly.

The Problem

nbgitpuller's core purpose was to remove the accidental complexity of using git for consuming content from the user side. But it left the possible accidental complexity of having to use git for creating content from the producer side. As we expand to more and more users, this starts becoming a problem.

Currently, git is used for both fetching the content (1) as well as resolving conflicts (2). We want to support multiple different sources for (1) while still supporting using git for (2).

Definition of Done

The SOW should propose implementation plans for:

1. A mechanism to provide for alternate sources where the content can come from.
2. two additional implementations for a new content source to make sure we have added an extensible method for future content sources. Pick sources that do not require authentication.
3. Improvements to the url builder (5) to support additional content sources.
4. Improvements to the command line UX (4) to make sure that also supports additional content sources

Out of scope

1. The mechanism design should be left open to possibly support authenticated sources (as part of a different SOW, but the implementations only necessarily need to work with content sources that don't require authentication.
2. nbgitpuller is focused on distributing content (notebooks, code files, etc) rather than data. So let's focus on content providers that are predominantly used for sharing content rather than data.

Prior Art

Four years ago, I spent time trying to make this happen with jupyterhub/nbgitpuller#194. It failed for a variety of primarily social reasons that I'm happy to provide context for to whoever is working on this project.

[agoose77]

It's past close-of-business my-time, but I wanted to wrap up some of the work I've been doing on the SoWs.

The more time I spent on this SoW (looking at the existing landscape of solutions and researching approaches), I came to realise that it wasn't a sensible direction to propose change. I had a meeting with Yuvi to discuss this earlier today.

Instead, I think an alternative SoW makes sense that looks to pull in repoproviders to nbgitpuller. I've started work on that, but not finished the draft yet. As such, I'm closing this as not-planned, and will tee-up the replacement work tomorrow.

[yuvipanda]

I think using repoproviders is an implementation detail, and the work we do there should still meet the definitions of done defined here. If not, we should modify this until it does.

[agoose77]

I'm happy to rework this to fit. As far as implementation details go, my intuition says that it's a fairly important detail -- repoprovisers defines much of the interface (URLs) rather than nbgitpuller!

[agoose77]

I've moved this Up Next, and re-estimated it as a 3 (because I've done some of the work here already).

[agoose77]

@yuvipanda I'm handing this over to you for review — let me know if you'd prefer e.g. to ask @jnywong to do that task?

[yuvipanda]

i'll review this shortly

[jnywong]

I've put my eyes on this with particular attention to time estimates.

However the biggest comment I have is that a plan for

"two additional implementations for a new content source to make sure we have added an extensible method for future content sources. Pick sources that do not require authentication."

is missing from the Deliverables.

[agoose77]

Thanks Jenny!

RE the deliverables, I've now explicitly surfaced the fact that both GitHub and non-GitHub source can be used!

[yuvipanda]

Left a video review