System package information, spkg-configure for Jupyter "notebook" package, "rst2ipynb", and dependencies

[mkoeppe]

(From dimpase https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/4eKjmINXAQAJ)

We currently cannot use Python packages from the system (see #29023) - and spkg-configure.m4 is not the right tool for changing that.

However, the situation is different for the Jupyter notebook package (and its dependencies):

We do not really need to install it in our venv if a suitable version is available in the system. Rather, we should install our kernel into the system/user's jupyter installation (see https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion).

Likewise, rst2ipynb is a Python package but we only provide it to make a script available, not a Python module. So again, if the system provides a suitable version, we do not need to install it in our venv.

notebook's exclusive dependencies (see #26919, https://github.com/jupyter/notebook/blob/master/setup.py):

• terminado, prometheus_client, send2trash
• argon2-cffi (new dependency)

Shared between notebook and optional package rst2ipynb (via nbconvert):

• nbconvert, nbformat, jsonschema, mistune, entrypoints, bleach, pandocfilters, testpath, defusedxml
• pyrsistent (new dependency of jsonschema)
• pandoc, pandoc_attributes (?)

The following dependencies are shared between notebook and sagelib/ipython/ipykernel. So we need to install them even if they are present in the system python:

• pyzmq, dateutil, tornado, jupyter-client, jupyter-core, jinja2, pygments, ipython_genutils

Packages depending on notebook:

• pari_jupyter (no system packages: https://repology.org/projects/?search=pari_jupyter&maintainer=&category=&inrepo=&notinrepo=&repos=&families=&repos_newest=&families_newest=)
• r_jupyter (script package; https://repology.org/project/r:irkernel/versions)
• widgetsnbextension

See also:

• Sage's jupyter doesn't "talk" to a systemwide IRkernel R package #30120 Sage's jupyter doesn't "talk" to a systemwide IRkernel R package
• Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest #26919 Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest
• Add optional packages jupyterlab, jupyterlab_widgets to support interacts in JupyterLab #30246 Support interacts in JupyterLab

Depends on #28197
Depends on #30299

CC: @dimpase @nbruin @slel @paulmasson @kiwifb @EmmanuelCharpentier @jhpalmieri @thierry-FreeBSD @sheerluck @nthiery

Component: build: configure

Keywords: sd111

Branch/Commit: u/mkoeppe/system_information__spkg_configure_for_jupyter__notebook__package_and_dependencies @ b8164c1

Issue created by migration from https://trac.sagemath.org/ticket/30124

[mkoeppe]

comment:2

This code from src/sage/repl/ipython_kernel/install.py of course needs updating so that we can install the kernel outside of the Sage venv:

        if prefix is None:
            from sys import prefix
        jupyter_dir = os.path.join(prefix, "share", "jupyter")
        self.nbextensions_dir = os.path.join(jupyter_dir, "nbextensions")
        self.kernel_dir = os.path.join(jupyter_dir, "kernels", self.identifier())
        self._mkdirs()

[mkoeppe]

Description changed:

--- 
+++ 
@@ -4,4 +4,8 @@
 Sage to supply anything but the jupyter kernel...
 
 
+See also:
+- #30120 Sage's jupyter doesn't "talk" to a systemwide IRkernel R package
+- https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion
 
+

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,11 +1,13 @@
-From dimpase https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/4eKjmINXAQAJ:
+(From dimpase https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/4eKjmINXAQAJ)
 
-switch to external jupyter - there is  no reason for
-Sage to supply anything but the jupyter kernel...
+We currently cannot use Python packages from the system (see #29023) - and `spkg-configure.m4` is not the right tool for changing that.
+
+However, the situation is different for the Jupyter `notebook` package (and its dependencies): 
+
+We do not really need to install it in our venv if a suitable version is available in the system. Rather, we should install our kernel into the system/user's jupyter installation (see https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion).
+
 
 
 See also:
 - #30120 Sage's jupyter doesn't "talk" to a systemwide IRkernel R package
-- https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion
 
-

[kiwifb]

comment:6

That implies that we will have to ask a jupyter that is in the PATH to tell us about its installation PATH.

If we just call jupyter kernelspec install then it will install in a user location for a system wide jupyter (~/.local/share/jupyter/kernels on linux), for a sage jupyter it will probably go to the sage install ({sys.prefix}/share/jupyter/kernels on linux/OS X). But for distro it will become problematic straight away since you install in a stage before merging in the system tree. It is possible to pass a --prefix but this is not the same as DESTDIR which would be the useful thing for distros.

[mkoeppe]

comment:7

I think we will only need the following changes to sagelib:

• remove the call to sage.repl.ipython_kernel.install from the installation steps done by sagelib's setup.py -- it is not compatible with modern python packaging methods anyway
• add a console_script that installs a kernel spec (using sage.repl.ipython_kernel.install), call it sage-kernel (interface similar to: https://ipython.readthedocs.io/en/stable/install/kernel_install.html)
• Change sage-notebook (which implements sage -n) to a shell script that invokes the correct jupyter (from system if installed in system, from SAGE_LOCAL if installed there)

Calling sage-kernel would be done by sage-the-distribution (perhaps in build/pkgs/sagelib/spkg-install), and likewise by distribution packagers' scripts.

[mkoeppe]

comment:8

Replying to @mkoeppe:

I think we will only need the following changes to sagelib: ... Calling sage-kernel would be done by sage-the-distribution (perhaps in build/pkgs/sagelib/spkg-install)

I'll do the first two items in #30298

[mkoeppe]

comment:9

I have changed the plan a bit, see #30298 and #30299.

[mkoeppe]

Dependencies: #28197, #30299

[mkoeppe]

Branch: u/mkoeppe/system_information__spkg_configure_for_jupyter__notebook__package_and_dependencies

[mkoeppe]

Commit: b8164c1

[mkoeppe]

New commits:

b8164c1

build/pkgs/notebook/spkg-configure.m4: New

[mkoeppe]

Description changed:

--- 
+++ 
@@ -10,4 +10,4 @@
 
 See also:
 - #30120 Sage's jupyter doesn't "talk" to a systemwide IRkernel R package
-
+- Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest

[mkoeppe]

Description changed:

--- 
+++ 
@@ -6,8 +6,18 @@
 
 We do not really need to install it in our venv if a suitable version is available in the system. Rather, we should install our kernel into the system/user's jupyter installation (see https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion).
 
+`notebook`'s dependencies (see #26919):
+- `nbformat`, `nbconvert`, `terminado`, `prometheus_client`, `send2trash`
+
+The following dependencies are shared between `notebook` and `sagelib`/`ipython`/`ipykernel`. So we need to install them even if they are present in the system python:
+- `pyzmq`, `dateutil`, `tornado`, jupyter-client`, `jupyter-core`, `jinja2`
+
+Packages depending on `notebook`:
+- `pari_jupyter` (no system packages: https://repology.org/projects/?search=pari_jupyter&maintainer=&category=&inrepo=&notinrepo=&repos=&families=&repos_newest=&families_newest=)
+- `r_jupyter` (script package; https://repology.org/project/r:irkernel/versions)
+- `widgetsnbextension`
 
 
 See also:
 - #30120 Sage's jupyter doesn't "talk" to a systemwide IRkernel R package
-- Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest
+- #26919 Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest

[mkoeppe]

Description changed:

--- 
+++ 
@@ -6,11 +6,15 @@
 
 We do not really need to install it in our venv if a suitable version is available in the system. Rather, we should install our kernel into the system/user's jupyter installation (see https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion).
 
-`notebook`'s dependencies (see #26919):
-- `nbformat`, `nbconvert`, `terminado`, `prometheus_client`, `send2trash`
+`notebook`'s exclusive dependencies (see #26919):
+- `terminado`, `prometheus_client`, `send2trash`
+
+Shared between `notebook` and optional package `rst2ipynb` (via `nbconvert`):
+- `nbconvert`, `nbformat`, `jsonschema`, `mistune`, `entrypoints`, `bleach`, `pandocfilters`, `testpath`, `defusedxml`
+- `pandoc`, `pandoc_attributes` (?)
 
 The following dependencies are shared between `notebook` and `sagelib`/`ipython`/`ipykernel`. So we need to install them even if they are present in the system python:
-- `pyzmq`, `dateutil`, `tornado`, jupyter-client`, `jupyter-core`, `jinja2`
+- `pyzmq`, `dateutil`, `tornado`, `jupyter-client`, `jupyter-core`, `jinja2`, `pygments`
 
 Packages depending on `notebook`:
 - `pari_jupyter` (no system packages: https://repology.org/projects/?search=pari_jupyter&maintainer=&category=&inrepo=&notinrepo=&repos=&families=&repos_newest=&families_newest=)

[mkoeppe]

Description changed:

--- 
+++ 
@@ -6,15 +6,17 @@
 
 We do not really need to install it in our venv if a suitable version is available in the system. Rather, we should install our kernel into the system/user's jupyter installation (see https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion).
 
-`notebook`'s exclusive dependencies (see #26919):
+`notebook`'s exclusive dependencies (see #26919, https://github.com/jupyter/notebook/blob/master/setup.py):
 - `terminado`, `prometheus_client`, `send2trash`
+- `argon2-cffi` (new dependency)
 
 Shared between `notebook` and optional package `rst2ipynb` (via `nbconvert`):
 - `nbconvert`, `nbformat`, `jsonschema`, `mistune`, `entrypoints`, `bleach`, `pandocfilters`, `testpath`, `defusedxml`
+- `pyrsistent` (new dependency of `jsonschema`)
 - `pandoc`, `pandoc_attributes` (?)
 
 The following dependencies are shared between `notebook` and `sagelib`/`ipython`/`ipykernel`. So we need to install them even if they are present in the system python:
-- `pyzmq`, `dateutil`, `tornado`, `jupyter-client`, `jupyter-core`, `jinja2`, `pygments`
+- `pyzmq`, `dateutil`, `tornado`, `jupyter-client`, `jupyter-core`, `jinja2`, `pygments`, `ipython_genutils`
 
 Packages depending on `notebook`:
 - `pari_jupyter` (no system packages: https://repology.org/projects/?search=pari_jupyter&maintainer=&category=&inrepo=&notinrepo=&repos=&families=&repos_newest=&families_newest=)