Add microservices doc, move CLI to jupyverse_api

docs/usage/microservices.md

@@ -0,0 +1,100 @@
+Jupyverse's modularity allows to run services quite independently from each other, following a microservice architecture. This can be useful when resources are provided by separate systems, or just to have a clear separation of concerns. For instance, you may want to host the contents on AWS and the kernels on Google Cloud, or on different machines in your private network. One way to achieve this is to use a reverse proxy that will forward client requests to the corresponding Jupyverse server. We will show how to do that using [Nginx](https://en.wikipedia.org/wiki/Nginx) on a single computer with local servers. This can serve as a basis for more complicated architectures involving remote servers.
+
+## Nginx setup
+
+Nginx is a web server that can also be used as a reverse proxy. We will use it to forward client requests to separate Jupyverse servers, based on the request URL.
+
+First, install Nginx in its own environment:
+```bash
+micromamba create -n nginx
+micromamba activate nginx
+micromamba install -c conda-forge nginx
+```
+We will just edit the default configuration file at `~/micromamba/envs/nginx/etc/nginx/sites.d/default-site.conf` with the following:
+```
+server {
+    listen       8000;
+    server_name  localhost;
+
+    location / {
+        proxy_pass http://localhost:8001;
+    }
+
+    location /api/kernelspecs {
+        proxy_pass http://localhost:8002;
+    }
+    location /api/kernels {
+        proxy_pass http://localhost:8002;
+    }
+    location ~ \/api\/kernels\/.+\/channels {
+        proxy_pass http://localhost:8002;
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "Upgrade";
+    }
+    location /api/sessions {
+        proxy_pass http://localhost:8002;
+    }
+}
+```
+Our Nginx server will listen at `http://localhost:8000`. This is the URL the client will see, i.e. the one we enter in the browser.
+
+Our Jupyverse servers will run on:
+
+- `http://localhost:8001` for everything except the kernels API. This means that this server doesn't have the ability to run user code, and thus cannot be used to execute a notebook.
+- `http://localhost:8002` for the kernels API. This server only deals with executing code in a kernel. It cannot serve JupyterLab's UI, for instance.
+
+Together, these Jupyverse servers can serve a full JupyterLab API. But because they run on different machines (not exactly in this case, since ports `8001` and `8002` are on the same machine, but let's pretend), we need to make them appear as a unique server. That is the role of the reverse proxy server.
+
+!!! note
+    WebSocket forwarding requires extra-configuration. Here, we use a `regex` to redirect `/api/kernels/{kernel_id}/channels`, which is the WebSocket endpoint for the kernel protocol. We also set the `Upgrade` and `Connection` headers used to upgrade the connection from HTTP to WebSocket.
+
+We can now run our reverse proxy server. Just enter:
+```bash
+nginx
+```
+
+## Jupyverse setup
+
+### Server 1: everything but kernels
+
+Let's create a new environment in a new terminal:
+```bash
+micromamba create -n jupyverse1
+micromamba activate jupyverse1
+micromamba install -c conda-forge python
+```
+Since we don't want to install the fully-fledged Jupyverse server, we will install the required plugins individually:
+```bash
+pip install fps-auth
+pip install fps-contents
+pip install fps-frontend
+pip install fps-lab
+pip install fps-jupyterlab
+pip install fps-login
+```
+Now just launch Jupyverse at port `8001`:
+```bash
+jupyverse --port=8001
+```
+If you open your browser at `http://127.0.0.1:8000` (the URL of the Nginx reverse proxy), you should see the JupyterLab UI you're used to. But if you look closer, you can see that there is no icon for kernels in the launcher tab. And in the terminal where you launched Jupyverse, you will see a bunch of `404` for requests at e.g. `GET /api/kernels`. This is expected, because we didn't install the kernels plugin. You can still use JupyterLab if you don't want to execute a notebook, for instance. But let's close it for now, and install the kernels plugin in another Jupyverse instance.
+
+### Server 2: the kernels API
+
+Let's create a new environment in a new terminal:
+```bash
+micromamba create -n jupyverse2
+micromamba activate jupyverse2
+micromamba install -c conda-forge python
+```
+This time, we only want to install the kernels plugin. It also requires an authentication system (for the sake of simplicity we will install `fps-noauth` which gives unrestricted access) and the frontend plugin. Let's not forget to install a kernel, such as `ipykernel`:
+```bash
+pip install fps-kernels
+pip install fps-noauth
+pip install fps-frontend
+pip install ipykernel
+```
+Launch Juyverse at port `8002`:
+```bash
+jupyverse --port=8002
+```
+Now if you re-open a browser at `http://127.0.0.1:8000`, you should be able to create or open a notebook, and execute it.

jupyverse_api/jupyverse_api/kernels/__init__.py

@@ -14,3 +14,4 @@ async def watch_connection_files(self, path: Path) -> None:
 class KernelsConfig(Config):
     default_kernel: str = "python3"
     connection_path: Optional[str] = None
+    require_yjs: bool = False

jupyverse_api/pyproject.toml

@@ -26,6 +26,9 @@ classifiers = [
 dependencies = [
   "pydantic >=1.10.6,<2",
   "fastapi >=0.95.0,<1",
+  "rich-click >=1.6.1,<2",
+  "asphalt >=4.11.0,<5",
+  "asphalt-web[fastapi] >=1.1.0,<2",
 ]
 dynamic = ["version"]
 
@@ -35,6 +38,9 @@ text = "BSD 3-Clause License"
 [project.urls]
 Source = "https://github.com/jupyter-server/jupyverse/jupyverse_api"
 
+[project.scripts]
+jupyverse = "jupyverse_api.cli:main"
+
 [project.entry-points."asphalt.components"]
 app = "jupyverse_api.main:AppComponent"
 jupyverse = "jupyverse_api.main:JupyverseComponent"

mkdocs.yml

@@ -32,6 +32,7 @@ nav:
 - Usage:
   - usage/single_user.md
   - usage/multi_user.md
+  - usage/microservices.md
 - Plugins:
   - 'auth': plugins/auth.md
   - 'contents': plugins/contents.md

plugins/auth/fps_auth/config.py

@@ -16,3 +16,4 @@ class _AuthConfig(AuthConfig):
     clear_users: bool = False
     test: bool = False
     login_url: Optional[str] = None
+    directory: Optional[str] = None

plugins/auth/fps_auth/db.py

@@ -53,7 +53,11 @@ class Res:
 
 
 def get_db(auth_config: _AuthConfig) -> Res:
-    jupyter_dir = Path.home() / ".local" / "share" / "jupyter"
+    jupyter_dir = (
+        Path.home() / ".local" / "share" / "jupyter"
+        if auth_config.directory is None
+        else Path(auth_config.directory)
+    )
     jupyter_dir.mkdir(parents=True, exist_ok=True)
     name = "jupyverse"
     if auth_config.test:

plugins/kernels/fps_kernels/main.py

@@ -33,7 +33,7 @@ async def start(
         app = await ctx.request_resource(App)
         auth = await ctx.request_resource(Auth)  # type: ignore
         frontend_config = await ctx.request_resource(FrontendConfig)
-        yjs = await ctx.request_resource(Yjs)
+        yjs = await ctx.request_resource(Yjs) if self.kernels_config.require_yjs else None
 
         kernels = _Kernels(app, self.kernels_config, auth, frontend_config, yjs)
         ctx.add_resource(kernels, types=Kernels)

plugins/kernels/fps_kernels/routes.py

@@ -3,7 +3,7 @@
 import uuid
 from http import HTTPStatus
 from pathlib import Path
-from typing import Dict, List, Set, Tuple
+from typing import Dict, List, Optional, Set, Tuple
 
 from fastapi import APIRouter, Depends, HTTPException, Response
 from fastapi.responses import FileResponse
@@ -32,7 +32,7 @@ def __init__(
         kernels_config: KernelsConfig,
         auth: Auth,
         frontend_config: FrontendConfig,
-        yjs: Yjs,
+        yjs: Optional[Yjs],
     ) -> None:
         super().__init__(app)
 
@@ -228,6 +228,9 @@ async def execute_cell(
             kernel_id,
             user: User = Depends(auth.current_user(permissions={"kernels": ["write"]})),
         ):
+            if yjs is None:
+                raise RuntimeError("Cannot execute without a Yjs plugin.")
+
             r = await request.json()
             execution = Execution(**r)
             if kernel_id in kernels:

plugins/lab/fps_lab/routes.py

@@ -15,7 +15,6 @@
 from fastapi.staticfiles import StaticFiles
 from starlette.requests import Request
 
-import jupyverse
 from jupyverse_api.app import App
 from jupyverse_api.auth import Auth, User
 from jupyverse_api.frontend import FrontendConfig
@@ -85,7 +84,7 @@ async def get_root(
 
         @router.get("/favicon.ico")
         async def get_favicon():
-            return FileResponse(Path(jupyverse.__file__).parent / "static" / "favicon.ico")
+            return FileResponse(Path(__file__).parent / "static" / "favicon.ico")
 
         @router.get("/static/notebook/components/MathJax/{rest_of_path:path}")
         async def get_mathjax(rest_of_path):

pyproject.toml

@@ -9,10 +9,6 @@ keywords = ["jupyter", "server", "fastapi", "plugins"]
 dynamic = ["version"]
 requires-python = ">=3.8"
 dependencies = [
-    "rich-click >=1.6.1,<2",
-    "asphalt >=4.11.0,<5",
-    "asphalt-web[fastapi] >=1.1.0,<2",
-    "fastapi >=0.95.0,<1",
     "fps-contents >=0.1.2,<1",
     "fps-kernels >=0.1.2,<1",
     "fps-terminals >=0.1.2,<1",
@@ -37,9 +33,6 @@ text = "BSD 3-Clause License"
 [project.urls]
 Homepage = "https://jupyter.org"
 
-[project.scripts]
-jupyverse = "jupyverse.cli:main"
-
 [project.optional-dependencies]
 jupyterlab = ["fps-jupyterlab >=0.1.2,<1"]
 retrolab = ["fps-retrolab >=0.1.2,<1"]
@@ -109,7 +102,7 @@ lint = [
   "black --line-length 100 jupyverse ./plugins",
   "isort --profile=black jupyverse ./plugins",
 ]
-typecheck0 = """mypy \
+typecheck0 = """mypy --no-incremental \
 ./jupyverse_api \
 ./plugins/contents \
 ./plugins/frontend \

tests/conftest.py

@@ -24,6 +24,8 @@ def start_jupyverse(auth_mode, clear_users, cwd, unused_tcp_port):
         "--set",
         f"component.components.auth.clear_users={str(clear_users).lower()}",
         "--set",
+        "component.components.kernels.require_yjs=true",
+        "--set",
         f"component.port={unused_tcp_port}",
     ]
     p = subprocess.Popen(command_list)