Multiple instructors, multiple courses, and multiple solutions

[perllaghu]

This is a discussion-point, to try and bring together the various threads

We have a number of similar threads on multiple courses, multiple instructors, and the possibility of an external exchange service.

We all want to use NBGrader, as it's an excellent tool for creating tests & grading the responses.

It would be good to get a common solution, or at least several non-exclusive solutions, for these issues.

Please could people describe what they are trying to achieve, and the environment in wich they are running nbgrader?

[jhamrick]

Great idea, thanks for creating this thread. Maybe let's explicitly ping some of the other people who have been thinking about this lately or might be interested in this discussion: @FranLucchini @rkdarst @sgvasquez @fishfree @zonca @sigurdurb @yuvipanda @dsblank

• Related issues: Support for one instructor in charge of multiple courses #998 Document how to set up nbgrader for multiple graders when running without JupyterHub #961 Extend nbgrader.db to cope with multiple courses (and think beyond that) #960 Assignment List extension does not support multiple classes #544 Create a JupyterHub "exchange" service to replace the exchange directory #659
• Related PRs: [WIP] Support for multiple classes via Jupyterhub groups #893 Allow instructors to share files via shared group id #1000

(Please feel free to reference any additional issues/PRs if I missed anything)

[perllaghu]

Environment

• Running in a Kubenetes cluster, with everything as Docker Images
• Pros:
  • One central service for the whole organisation
  • Platform can be scaled by adding/removing worker nodes as needed (scale to suit load)
  • External service can deploy from a selection of Notebooks (different courses need different libraries)
  • Can use external data for users persistent storage (Network Storage services can be cheap)
  • Integrates with LTI to provide course_id, user_id, and role values (Devolves authentication & authorisation)
• Cons:
  • One central service for the whole organisation
  • All notebooks run as the same user (all external data is owned by the same user)
  • Persistent storage needs to be volume-mounted into the docker image (so it not dynamically changeable)
  • Selection of mount points becomes important

Aims

We want users identified as instructors to have access to formgrader whilst everyone else gets assignments [done]. We need to cope with multiple classes, and cannot have students able to "borrow" another students submission. [doing]

Current Plan

Our current plan is to dynamically create nbgrader_config .py at notebook startup, and set CourseDirectory.root to /courses/<course_id> - this will separate each course into it's own namespace

For the exchange stuff, the notebook Docker images will volume-mount:

For outbound:

/srv/nbgrader/exchange/outbound -> [external_stroage_point]/<course_id>/outbound

For inbound, it depends on whether you're an instructor or a student (this is what stops students seeing each others work):

/srv/nbgrader/exchange/inbound -> /disk/remote/courses/<course_id>/inbound   # Instructors
/srv/nbgrader/exchange/inbound -> /disk/remote/courses/<course_id>/inbound/<student_id>  # students

However, this does mean we need to modify the collect code to iterate over every student directory

Plan, stage 2

Future work is to either adopt hubshare [https://github.com/jupyterhub/hubshare], or write our own version.
We would also push for a central nbgrader.db, and add the functionality for multiple courses & the like to it... and an API to allow instructors/students to work any course they've previously been associated with, and for the inbound/outbound stuff to all of interacted with through an API (which then does authorization with this central database)

[perllaghu]

Also related: #1010

[sivasquez]

Enviroment

Currently using the basic system, and we intend to use deployment with either Docker or Kubernetes in a future.

Aims

• To allow users to be enrolled in more than one course at a given time, having an user-friendly UI that allows them to browse their courses and assignments in a simple way (current solution in docs allow students to have access to all courses at a given time, which in a real scenario would mean 100+ courses).

• To allow graders to be graders in more than one course at the same time (having full access to release, grade and feedback assignments in all their courses), without having to change any configuration files.

• To allow graders to be student in courses, different than the courses where they are graders, without losing functionality or having access to information they shouldn't have since they have the rank of grader in another course.

Current Plans

Based on the solution proposed in this issue, we intend to create an external service (we are considering using Flask for this) that allows users to query for the courses they are in a student, the courses they are in as graders, and the students they have in a given course if they are graders in said course. This separation will make it easier to work with another systems if we plan to integrate more systems to JHub/Nbgrader.

This would also include changes to the UI, but its implementation is dependent on whether we can do the backend changes - or not.

We'd like to ask for guidance on where to look at the files to start making changes. Personally, I've been toying around trying to change how Nbgrader manages the course_id, but with so many uses of the variables in so many different files is not easy to just go and edit the code. Any insight would be appreciated :)

[dsblank]

Rather than an external service, you can build on JupyterHub's services. See for example these two services:

https://github.com/BrynMawrCollege/jupyterhub

You can make sure that they are logged in, and then perform the various functions (retrieving class rosters for the teacher, etc.). Sounds like a great idea!

[sivasquez]

@dsblank Thanks for the feedback! We are planning to work on this ASAP, although I haven't worked with services before. Is there any guide on how to make good services? I'm thinking that for our solution (I'm working with @FranLucchini in this), we'll need to be able to pass parameters (current user id, course id, student ids in case of a grader) and receive responses to/from the service dinamically that will affect what the UI shows.

[dsblank]

The main docs are here: https://jupyterhub.readthedocs.io/en/stable/reference/services.html

It is just a regular web endpoint, so you should be able to do whatever you want. You will be able to check to see if the user is logged in, and get their user_id, for example.

[fishfree]

@perllaghu I think integration with LTI is very necessary, for there are already many ID providers, especially in Learning Management Systems.

[perllaghu]

@fishfree - remember that NBGrader itself has no need to know about LTI .... however it would be good if it coukd work in a JupyterHub environment driven by LTI connections.

[perllaghu]

I was thinking about this over the weekend.... and I think there are three distinct, but inter-related, parts that need to be resolved:

1. The formgrader and assignements functions need to be able to cope with multiple courses (and, consequently, multiple roles)
2. Abstract the exchange functionality and move towards an API-based system (such as hubshare.)
3. Acknowledge LTI as a potential access method for a Notebooks/Jupyterhub service

Taking the LTI issue first: this should not be something for NBGrader to concern itself with, but for some pre_start hook to manage. It's impact on NBGrader is that users, courses, and roles (students v's instructors) is not known at the time the jupyterhub service is created, so needs to be managed in the pre-start hook area.

The first issue, of multiple courses, seems to have a solution of extending the Jupyterhub User model so one can use that to get a list of courses for each user.

This leaves us with modifying the exchange, and the use of hubshare (https://github.com/jupyterhub/hubshare) - what is the state of this work?

What other issues are there that I've missed?

[moschlar]

Environment

• On-Premises Kubernetes Cluster deployed with Rancher 2.x
• Setup according to Zero to JupyterHub with Kubernetes
  • OpenID Connect authentication
  • NFS as storage provider using nfs-client-provisioner
• Custom singleuser image based on jupyter/scipy-notebook:
  • Install nbgitpuller and nbgrader
  • Add custom start.sh script for user handling, which basically does:

NB_HOME="/home/$NB_USER"
/usr/sbin/usermod -u $NB_UID -d "$NB_HOME" -l "$NB_USER" jovyan
sed -r "s#Defaults\s+secure_path=\"([^\"]+)\"#Defaults secure_path=\"\1:$CONDA_DIR/bin\"#" /etc/sudoers | grep secure_path > /etc/sudoers.d/path
mkdir -p "$NB_HOME"
chown -R "$NB_USER:" "$NB_HOME"
cd "$NB_HOME"
exec sudo -E -H -u "$NB_USER" PATH="$PATH" XDG_CACHE_HOME="/home/$NB_USER/.cache" PYTHONPATH="${PYTHONPATH:-}" $cmd

• Special Helm chart values to make it work:

...
  hub: 
    extraConfig: 
      environment: |-
        import string
        safe_chars = set(string.ascii_lowercase + string.digits)
        c.KubeSpawner.environment = {}
        c.KubeSpawner.environment["NB_UID"] = lambda self: str(1000 + int(self.user.id))
        c.KubeSpawner.environment["NB_USER"] = lambda self: ''.join([s if s in safe_chars else '-' for s in self.user.name.lower()])"
      priv: "c.KubeSpawner.privileged = True"
      uid: "c.KubeSpawner.uid = 0"
...
  singleuser: 
    cmd: "start-singleuser.sh"
    image: 
      name: ".../singleuser"
      tag: "..."
    storage: 
      dynamic: 
        storageClass: "nfs-client"
      extraVolumeMounts: 
        - 
          mountPath: "/srv/nbgrader/exchange"
          name: "exchange"
      extraVolumes: 
        - 
          name: "exchange"
          persistentVolumeClaim: 
            claimName: "exchange"
      homeMountPath: "/home"

Aims

• My primary goal would be to not need a custom singleuser image to decrease the number of moving parts I need to worry about

[perllaghu]

@moschlar - you have an environment very similar to ours... except we have multiple "single-user" notebooks, and an external service that identifies the user & which notebook they're wanting to use.

How are you managing persistent storage for your users?

[moschlar]

@perllaghu

How are you managing persistent storage for your users?

We have central NFS storage (NetApp appliance) which exports a volume (with sec=sys) for the Kubernetes cluster. And then we use nfs-client-provisioner to provision directories for each PersistentVolumeClaim.
Seems to work so far!

[rkdarst]

Aalto University environment

I'm late, but here's what we have...

• Jupyterhub + Kubernetes + docker
• All open but not documented enough to be reuseable: https://github.com/AaltoScienceIT/jupyterhub-aalto, the interesting stuff, mainly pre_spawn_hook and get_profile_list, is in jupyterhub_config.py

Multiple courses

• We use the kubespawner profiles to make profiles for different courses (and also generic instances usable outside of courses). These are dynamically generated by making KubeSpawner.profile_list a callable.
• Per-course configuration is stored in YAML files, processed in pre_spawn_hook to mount the right course data and select the right container image.

Multiple users

• use PAM authentication against university auth, so we have actual usernames.
• All data is actually stored according to the actual UID, not all as a single user. This is managed rather simply by docker uid/gid options and pre_spawn_hook
• Since data is stored by user, we can rely on university data storage. Data is also available to students in some classrooms and some shell servers.
• So far, user "home directories" are shared among all possible courses a user could spawn, and thus lives on even after a course is done.

Multiple instructors

• Multiple graders for the same course work. Graders spawn with their own UIDs, files shared via a GID. These are set in pre_spawn_hook.
• PR Allow instructors to share files via shared group id #1000 is needed in order to make jupyter/nbgrader write all files with group-shared permissions.
• Grader servers are the same as user servers, it just mounts the course directory in only instructor servers (via pre_spawn_hook).
• Instructors are not root on the system, so can't do things like copy feedback themselves.

All of this already works and many courses have used it. Some improvements could make things smoother, but really most work we do is local integration. If formgrader would support multiple courses, it could be designed so that you don't have to select the course at spawn-time... though this may still be needed to select the right container image.

Student and instructor UI (visible options depend on who is logged in):

[jhamrick]

Hi all, I wanted to give a quick update on this issue in light of what we've been working on at the Edinburgh hackathon! We made the decision to make nbgrader more compatible with multiple courses, with the following changes:

• Add a course_id table to the gradebook, so that multiple courses can use the same database. (Add course table, add course_id column to assignment #1099)
• Restrict the courses displayed in nbgrader list and the Assignment List based on which courses a student is actually a part of ([WIP] Support for multiple classes via Jupyterhub groups #893, Restrict access for students to different courses #1040). This is configurable and by default there are no access controls, but nbgrader includes access controls for JupyterHub and you can write your own custom plugin too if you want (Add authentication plugin support #1093).
• Update all of nbgrader's functionality to accept course_id as an argument (Integrate course_id into the api and apps #1118).
• Add a new "Course List" extension that will allow instructors to view which courses they have access to for grading. This is meant for use with the model of running multiple formgraders as separate JupyterHub services. The Course List extension then provides links to each of these separate services (Add a course list extension that shows all courses an instructor can manage #1113).
• Add more documentation for how all of this works (Documentation for multiple classes #1114).

Note that even with all these changes, we are still making the following assumptions:

• The formgrader service only knows about a single course, so you need to run multiple formgraders if you want to have multiple classes. This means you need to manually create a separate service for each class; the formgraders cannot currently be created on the fly.
• You need a separate course directory for each class, with separate nbgrader_config.py files.
• The exchange is still a regular filesystem exchange.

I plan to address these at some point which will make the setup for multiple classes even smoother, but probably not immediately. But I hope that the changes we've made so far will make having multiple classes a much nicer experience than it is currently!

Huge thanks to @sigurdurb @BertR and @damianavila who worked on this at the hackathon!! 🥇

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/jupyter-nbgrader-hackathon-topics/1040/1