How to set up a jupyverse for a team

[oscar6echo]

This dicussion is a follow up from issue #301

[oscar6echo]

Let me clarify the objective and the context:

Context:
AFAIK Jupyter comes in 3 flavors:

• Jupyter "Desktop" ie. the individual Jupyter Server - excellent but individual
• JupyterHub: ambitious initial idea but very complex to setup, demanding on the infrastructure, very expensive if many users
• JupyterLite: super scalable and convenient but limited scope (some packages do not work) for the foreseeable future

Now it seems there is a new kid on the block: Jupyverse. AFAIU it is a recent, modern version of Jupyter server and could maybe fill the gap I see in the Jupyter range to allow easy, convenient small scale teamwork.

Objective:
Allow a small (<10 people) team of engineers in a corp env (quite constrained), typically Python and Jupyter users but without other IT skills, to launch and use a common jupyverse from one of their Win10 machine or from a Linux VM, in order to work independently and collaboratively.

Constraints:

• On a Win10 machine only conda/(micro)mamba is installed with access to conda-forge and pypi through artifactory, no docker, internet access through proxy.
• On a Linux VM sudo is available and internet access through proxy.
• However it can be assumed that a fief server/worker and theirs satellites (PostgreSQL DB, Redis) are self hosted on a corp kube cluster by an IT team, and one team member would have access to a tenant to manage other team members access rights, and the OAuth provider would be the corporate one.

User access rights:
The jupyter permissions mentionned in the issue #301 are necessary to provide full individual server Jupyter Lab experience. But it would be useful, arguably necessary, to add file system access rights per user or group of users along the lines described in the following tentative yaml:

# file access-config.json
---
# groups just for convenience
groups:
- name: all
  users:
  - user-a
  - user-b
# file system
# only paths mentionned are visible
folders:
# this folder would have notebooks used in real time collab
- path: "./shared"
  rights:
    readonly: []
    readwrite:
    - all
# this folder would have immutable reference notebooks and data files
- path: "./ref"
  rights:
    readonly:
    - all
    readwrite: []
# this is user-a private folder
- path: "./folder-a"
  rights:
    readonly: []
    readwrite:
    - user-a
# this is user-b private folder
- path: "./folder-b"
  rights:
    readonly: []
    readwrite:
    - user-b

Assuming this can be envisaged, I would think these rights should be encoded as fief permissions ? Or should it stays with jupyverse as a config file read upon start like below ?

jupyverse \
    --set auth_fief.base_url=https://jupyverse-XXXXX.fief.dev \
    --set auth_fief.client_id=XXXXXXXXXXXXXXXXXXXXXXXX \
    --set auth_fief.client_secret=XXXXXXXXXXXXXXXXXXXXXX \
    --set filesystem.access_config=access-config.json

Naturally these are blunt proposals for illustration only.

I would also be useful, I think, to allow such filesystem.access_config without fief so that a team of engineers in constrained corp env with zero IT support could work on same server for a few hours or a day.

In general the small team use case has potential in the corp env I'm sure.
And this size limit may make some issues easy to solve e.g. who wants to collaborate with more than 5 people on a notebook, or maybe 10 on a text file ?

Food for thought.

[davidbrochart]

But it would be useful, arguably necessary, to add file system access rights per user or group of users along the lines described in the following tentative yaml

I see two solutions here:

• use WebDAV, since Jupyverse supports it and it's a technology that already solves this kind of problems. Their configuration file doesn't look too different from what you suggest.
• use a custom contents Jupyverse plugin, or extend fps-contents, to allow fine-grained file system access. Then we can use new permissions such as {"contents:folder-a": ["read", "write", "execute"]} for user-a, etc.

[oscar6echo]

use WebDAV

This does look interesting as it has all the features but:

• only for small data as access to large files (>100MB eg. large parquet files) will be slow at best, maybe impossible, through a web server, right ?
• It requires docker to run the webdav server, which is fine in general, but maybe beyond the limit in the context I describe

use a custom contents Jupyverse plugin, or extend fps-contents:

This requires to work out the fps-contents extension but when done, access rights are under full control and there is no extra complexity for the engineer user.
How would you go about customizing fps-contents to consume sample permission {"contents:folder-a": ["read", "write", "execute"]} granted to user-a ? I would be grateful for any indication as I do not even see where existing permissions contents:write and contents:read are checked.

[davidbrochart]

• only for small data

I don't know, why would it be different than the contents API? Anyway in collaborative mode the contents go through the Y-WebSocket, and WebDAV would not be used to actually load/save data.

• It requires docker to run the webdav server

No, fps-webdav is a Jupyverse plugin.

How would you go about customizing fps-contents to consume sample permission

I need to think of a way to do that. But since it would be on my free time, I cannot guarantee when/if it will happen.

[oscar6echo]

re webdav:

Then I misunderstand quite a few things about fps_webdav...
I thought that the file system was webdav-served for the jupyverse server to interact with, in which case the "distance" between kernel and file system would make file read/write slower than on local file system.
But I now gather (maybe) that plugin fps_webdav acts as a "replacement" for fps_content ?

Anyway I must read more and experiment.
Just one question: How to start jupyverse with webdav and a sample config file ?

I cannot guarantee when/if it will happen

No hurry/pressure !
At this stage I'm just trying to get hints/links about where decisions based on permissions are located in the code.
Your explanations are already super valuable.

[davidbrochart]

I thought that the file system was webdav-served for the jupyverse server to interact with

No, fps-webdav interacts with the file system directly, and the web client uses WebDAV.

But I now gather (maybe) that plugin fps_webdav acts as a "replacement" for fps_content ?

Yes it could, but I admit that we're not there yet. For instance, a JupyterLab extension could provide file system access through WebDAV instead of using the contents API.

Just one question: How to start jupyverse with webdav and a sample config file ?

Just pip install fps-webdav. There is some basic configuration.

At this stage I'm just trying to get hints/links about where decisions based on permissions are located in the code.

Actually this is not so easy, I will need to take a step back and think about it myself.

[mvelbaum]

@davidbrochart is it possible to make jupyverse listen on a unix socket? Jupyter Server supports this and allows to run multiple instances on a server shared by a small team without fiddling with ports. Users simply have a unix socket in their home dir and use an ssh tunnel to access it.

[davidbrochart]

Currently jupyverse uses the uvicorn server through asphalt-web. Uvicorn has a uds config to bind to a UNIX domain socket, but asphalt-web doesn't expose this option. So the first step would be to open a feature request in asphalt-web.

[mvelbaum]

Thanks, opened asphalt-framework/asphalt-web#67

[mvelbaum]

It looks like Asphalt v5 switched to hypercorn which allows you to bind to a unix domain socket by using the unix: prefix as part of the bind address: https://hypercorn.readthedocs.io/en/latest/how_to_guides/binds.html

Is there a plan to switch to this version?

[davidbrochart]

Yes, see #388.