support for environment variables in path

[jhgoebbert]

We provide a JupyterHub+JupyterLab solution to access our hpc systems at the Jülich Supercomputing Centre. Depending on the system and project a user requests at login, environment variables like $HOME, $PROJECT and $SCRATCH are set to different paths.

@rcthomas pointed me to the fantastic idea to use `jupyterlab-favorites' to pass these environment variables through to the UI and make them accessible with one click.

One could say, problem solved :) ... but it has one drawback:

There is no other way to add (the login-specific) $HOME, $PROJECT, $SCRATCH to the favorites than writing/updating the users owned settings file at $HOME/.jupyter/lab/user-settings/@jlab-enhanced/favorites/favorites.jupyterlab-settings every time a JupyterLab gets started for a user by JupyterHub.
Here is our update-script which ensures, that the users favorites are kept while $HOME, $PROJECT, $SCRATCH is updated.

Update script to add $HOME, $PROJECT, $SCRATCH as favorites to `favorites.jupyterlab-settings`

#!/usr/bin/env python3

import logging
import argparse
import os

import json5
import jsonschema

from jsonmerge import Merger

parser = argparse.ArgumentParser()
parser.add_argument('--debug', action='store_true')

logging.basicConfig()
if parser.parse_args().debug:
    logging.getLogger().setLevel(logging.DEBUG)
else:
    logging.getLogger().setLevel(logging.INFO)

merge_schema = {
    "jupyter.lab.setting-icon": "jupyterlab-favorites:filledStar",
    "jupyter.lab.setting-icon-label": "Favorites",
    "title": "Favorites",
    "description": "Favorites settings.",
    "type": "object",
    "additionalProperties": False,
    "properties": {
        "favorites": {
            "title": "Favorites",
            "description": "The list of favorites.",
            "items": {"$ref": "#/definitions/favorite"},
            "type": "array",
            "default": [],
            "mergeStrategy": "arrayMergeById",
            "mergeOptions": {"idRef": "name"},
        },
        "showWidget": {
            "title": "Show Widget",
            "description": "Toggles the favorites widget above the filebrowser breadcrumbs.",
            "type": "boolean",
            "default": True,
        },
    },
    "definitions": {
        "favorite": {
            "properties": {
                "root": {"type": "string"},
                "path": {"type": "string"},
                "contentType": {"type": "string"},
                "iconLabel": {"type": "string"},
                "name": {"type": "string"},
                "default": {"type": "boolean"},
                "hidden": {"type": "boolean"},
            },
            "required": ["root", "path"],
            "type": "object",
        }
    },
}
logging.debug(f"JSON schema for merging: {json5.dumps(merge_schema, indent=4)}")


def validate(instance, schema):
    try:
        jsonschema.validate(instance=instance, schema=schema)
    except jsonschema.ValidationError as e:
        logging.error("Exception occurred", exc_info=True)
        return e.schema["error_msg"]


def fav_jsonstr(envvar, jsonstr):
    path = os.getenv(envvar)
    hidden = "true" if not path else "false"
    jsonstr += f"""
        {{
            "root": "/",
            "contentType": "directory",
            "iconLabel": "ui-components:folder",
            "path": "{path}",
            "name": "${envvar}",
            "hidden": {hidden}
        }}"""
    return jsonstr


# create favorite-json for $HOME, $SCRATCH, $PROJECT
sys_fav_jsonstr = """{
    "favorites": ["""
sys_fav_jsonstr = fav_jsonstr("HOME", sys_fav_jsonstr) + ","
sys_fav_jsonstr = fav_jsonstr("PROJECT", sys_fav_jsonstr) + ","
sys_fav_jsonstr = fav_jsonstr("SCRATCH", sys_fav_jsonstr)
sys_fav_jsonstr += """
    ],
    "showWidget": true
}"""
logging.debug(f"JSON for additional favorites: {sys_fav_jsonstr}")
validate(instance=json5.loads(sys_fav_jsonstr), schema=merge_schema)

# get path to favorites.jupyterlab-settings
settings_dpath = os.getenv(
    "JUPYTERLAB_SETTINGS_DIR",
    os.path.join(os.environ["HOME"], ".jupyter/lab/user-settings"),
)
fav_settings_dpath = os.path.join(settings_dpath, "@jlab-enhanced/favorites/")
fav_settings_fpath = os.path.join(fav_settings_dpath, "favorites.jupyterlab-settings")
logging.debug(f"settings file path: {fav_settings_fpath}")

# if user settings file exists we need to merge
if os.path.exists(fav_settings_fpath):
    logging.debug(f"settings file exists: {fav_settings_fpath}")

    # read user-settings
    user_fav_json = {}
    with open(fav_settings_fpath, "r") as fav_file:
        try:
            user_fav_json = json5.load(fav_file)
        except ValueError:  # includes simplejson.decoder.JSONDecodeError
            logging.error("Decoding JSON has FAILED", exc_info=True)

    # merge JSONs
    fav_merger = Merger(merge_schema)
    try:
        merged_json = fav_merger.merge(user_fav_json, json5.loads(sys_fav_jsonstr))
    except:
        logging.error("Merging JSONs has FAILED", exc_info=True)

    # print result
    logging.debug(f"merged settings file: {json5.dumps(merged_json, indent=4)}")

    # validate result
    validate(instance=merged_json, schema=merge_schema)

    # write JSON to file
    jsonstr = json5.dumps(merged_json, indent=4)
    try:
        with open(fav_settings_fpath, "w") as fav_file:
            fav_file.write(jsonstr)
        logging.info(f"Writing merged settings file SUCCESSFULL")
    except:
        logging.error("Writing settings file FAILED", exc_info=True)

# if user settings file does NOT exist - we need to create it
else:
    logging.debug(
        f"settings file {fav_settings_fpath} does not exist - creating a new one"
    )

    # create file with content
    try:
        os.makedirs(fav_settings_dpath, exist_ok=True)
        with open(fav_settings_fpath, "w") as fav_file:
            fav_file.write(sys_fav_jsonstr)
        logging.info(f"Writing new settings file SUCCESSFULL")
    except:
        logging.error("Writing settings file FAILED", exc_info=True)

Describe the solution you'd like

It would be great, if jupyterlab-favorites would support environment variables to be added to its settings file in the favorite property path:

        {
            'root': '/',
            'contentType': 'directory',
            'iconLabel': 'ui-components:folder',
            'path': '$HOME',
            'name': '$HOME',
        }

That would allow us to do the settings once for all users at sys.prefix-level in <jlab-install-dir>/etc/jupyter/labconfig/default_setting_overrides.json.
(but to my understanding this might need to be supported by the JupyterLab's file bowser first)

Describe alternatives you've considered

• Setting it in <extension_name>/<plugin_name>.jupyterlab-settings is no option as the favorite-paths are user- and login-specific.
• Setting it in overrides.json in the application’s settings directory is not option as well for the same reason.
• overrides.json is not taken from a list of directories, so I cannot add an additional place to check
• Environment variables like $HOME are not resolved by JupyterLab’s file-browser

[fcollonval]

Thanks @jhgoebbert for the detailed description.

The short answer: this will not be implemented because it is too specific (relies on a content file manager exposing a file system) and need a backend side (the settings cannot be transformed and the os environment are only available in the server).

That said, a way around it and some additional information:

• It is now possible to define multiple overrides.json files: see add overrides.d for settings defaults jupyterlab/jupyterlab_server#224

The ordering of paths is:

• {app_settings_dir}/overrides.d/*.{json,json5} (many, namespaced by package)
• {app_settings_dir}/overrides.{json,json5} (singleton, owned by the user)

• Using a custom overrides.json for the generic list, you should be able to use a script like the one given above to generate the wanted settings.

Let me know if defining a dedicated overrides.json file is not working.

[jhgoebbert]

Hi @fcollonval,
thank you very much for your answer and explanation.
Yes, the wish for environment variables support would be a value for very specific systems (mainly in hpc) - and the required code change is a lot.

It is great to know, that it is possible to define multiple overrides.json. Anyway, I am not sure how that might help in my case, as the {app_settings_dir} is not a list of directries but a single one on sys-prefix level. But knowing this helps at other places.

[jhgoebbert]

I close this issue then.