Edinburgh hackathon

[jhamrick]

This is an issue to collect suggestions/ideas to work on during the nbgrader hackathon in Edinburgh from May 29-31.

[perllaghu]

Some thoughts from an Edinburgh coder (as these are things that affect the environment we have):

• Consider use when authentication comes from an LTI [https://www.imsglobal.org/basic-overview-how-lti-works] connection

  • Multiple Courses (ref: PR Restrict access for students to different courses #1040) v's Support for multiple classes via Jupyterhub groups (PR [WIP] Support for multiple classes via Jupyterhub groups #893)
  • Users/Courses not in the database at startup

• Multiple graders for one assignment (part of issue Multiple instructors, multiple courses, and multiple solutions #1030, which extends I Support for one instructor in charge of multiple courses #998)

• NBGrader has exchange as a configurable plugin (in a similar vein to Jupyterhubs authenticators & spawners)

  • API Tests have hard-coded file copy methods to pre-load the system to enable testing, and os.file_exists-type tests for release & submit tests

• Generation of feedback copies for students within the formgrader UI (to mirror the exisitng terminal command)

• Create some mechanism for getting feedback to students

[rkdarst]

I just saw this; I didn't know there was a nbgrader hackathon coming up. I couldn't have gone anyway, but is it still open for others to come? I'll have some people working with me this summer, who will probably devote some amount of time to nbgrader.

A professor here who has used nbgrader extensively made a wishlist, which I have added some of my items to. It could be relevant here: nbgrader wishlist. Some of these may be a bit local-deployment specific.

I could offer a presentation/notes about how we run nbgrader, since it seems like it's different than most, which leads to quite a few interesting issues. I'd be interested to see what others think.

[perllaghu]

I've pulled @rkdarst's list into here:

Assignment deadlines

• The students do not see the deadlines of the assignments. It would be nice to show the deadline (maybe in the Assignment tab).
• Now the deadline (in Formgrader tab in the instructor view) is set and displayed in UTC. This is very inconvenient. It would be nice to have the deadline in local time.

Where to solve assignments

• Is it possible to have exchange folder on a remote host? This way students could fetch/submit assignment from their computer.

Validation and submission

• If a student submits a notebook without meta-data or with duplicate cells, does validation report these problems? Perhaps for duplicate cells only. Can we automatically generate feedback that your submission cannot be autograded?
• Many students forget to click Submit button. It would be nice if students could see whether they submitted or not.
• The students do not see the version that they submitted. It would be nice to let the students open their submission. (and maybe see git diff to their current version).
• Can students request autograding (say min 3 times) to know how many points they will get? I am not sure I want to use this feature though. Students, of course, really want this feature.

Grading

• Currently: the latest submission is evaluated. We may want to evaluate the last submission before the deadline (if late submissions get zero points).
• There seems to be a bug in autograing. When the assignment cannot be autograded (e.g. due to duplicate cells), nbgrader sometimes exits with an error message instead of skipping the problematic submission. In this case, one has to remove the problematic submission and run autograding again. This is inconvenient.
  I have examples of such problematic cases:

[AutogradeApp | WARNING] Removing failed assignment: /course/autograded/saarinv11/4_conv
[AutogradeApp | ERROR] One or more notebooks in the assignment use an old version
	of the nbgrader metadata format. Please **back up your class files
	directory** and then update the metadata using:

	nbgrader update .

• How to run autograding on the background so that it does not stop when connection is lost? (nohup command &)?
• By default, in the Manual grading form students’ usernames are hidden. If the teacher wants to find a submission by username, s/he has to open usernames one by one. It would be nice to show all usernames with one click.

Analysis of results

• It would be nice to analyze points by autograding cells (to analyze what tasks were difficult).

Feedback

• Give teachers a way to copy the feedback (button: copy feedback or nbgrader command). If one publishes results through the exchange folder, then students have to know when the feedback needs an update

Other

• Port nbgrader Assignment List extension to JupyterLab
• Port nbgrader Formgrader extension to JupyterLab
• Script to automatically remove cached submitted assignments from .home/.local/share/jupyter/nbgrader_cache (perhaps this is mostly a local issue)
• nbgrader: warn if dotfiles in assignment directories, don't copy them to students, exchange, or wherever.

[perllaghu]

@rkdarst - two thoughts relating to Where to solve assignments

1. Have you looked at BinderHub (https://github.com/jupyterhub/binderhub)
2. We [Edina] are working on an exchange service (mostly because we're running in a multi-node Cluster)

The point in Feedback is also well made, and something we're also interested in seeing taken forward

[jamesaslack]

@rkdarst - We do still have some of the budget left if there are others that you may be able to recommend. Feel free to pass my contact email on to your colleagues : james.slack@ed.ac.uk

We are really looking for those who have a good functional understanding of Jupyter and nbgrader to be able to contribute to the hackathon

[rkdarst]

@perllaghu, I know of BinderHub (and colab, and our other own local services). In fact, one teacher here suggested that students download notebooks and take them to these services for training, before uploading them again.

For our exchange, we have a NFS share which holds all course and user data, so we can seamlessly scale to multiple nodes (we have six now).

[rkdarst]

I haven't yet found anyone who can make it, but would like to offer a presentation about our setup because it is quite different than the typical use case I see.

[jhamrick]

Here is also a link to a discussion the Jupyter Discourse Forum: https://discourse.jupyter.org/t/jupyter-nbgrader-hackathon-topics/1040/5

[jhamrick]

FYI, I've created the Edinburgh-Hackathon label and will be tagging issues/PRs with that to make it easier to track what people want to work on. If you'd like me to tag an issue with it please comment here or on the discussion thread with the issue number and I'll go ahead and tag it!

[psychemedia]

My use case is for feedback and grading system in a distance education setting where there are ~350 students and 15 markers, with equal numbers of scripts each.

Notebooks for marking are generally templated but with students free to split cells. Our requirements include:

• support for attaching manual marks;
• support for providing on script feedback that can be easily identified.

The workflow is essentially ad hoc; markers get a set of student files in a zip file, arranged hierarchically (subfolder per student). The markers work on their own machines (which could be anything) although we do provide a course VM and we could provide a package for markers to install in their VM to assist marking. A central service is not for the foreseeable future, unless we can make a compelling case. The Jupyter installation in the VM is typically15 months old when marking comes round. (Lead times that freeze VM are 3-6 months prior to course start, with course duration of 9 months.)

Tools to support individual markers in the first instance is probably our starting point. If enough markers independently use something, we can perhaps use that to advocate for a central service.

[sdyang1729]

Hello, everybody,

Well, this is my first time ever to leave any sort of public message in github, so please bear with me if my writing is kind of out of nowhere. I do not even know how my identity would appear if I press the green Comment button. But I am so pleased to learn that people with the ability to make nbgrader as an extension to jupyterlab are gathering together! Being able to run nbgrader in Jupyterlab has been in my wish list for quite a while.

I am a professor of mathematics for more than 20 years, and have been trying to adapt Jupyter notebook and nbgrader in my teaching for about 2~3 years. So far, it is not that successful as I hoped in the beginning, due to many factors. Hopefully it would get better.

As for the hackathon, I truly wish I could be there, but I cannot. I found about it too late to make a travel plan. Anyways, I am not a fluent coder so I don't think I can contribute to this wonderful project by providing codes. So I will wait until I hear the outcomes of the hackathon.

My wishlist about nbgrader is practically a sublist of @rkdarst's list mentioned in the fourth comment of this issue. I copy and paste some of them here in the order of my preference.

• Port nbgrader Assignment List extension to JupyterLab
• Port nbgrader Formgrader extension to JupyterLab
• The students do not see the deadlines of the assignments. It would be nice to show the deadline.
• The deadline (in Formgrader tab in the instructor view) is set and displayed in UTC. This is very inconvenient. It would be nice to have the deadline in local time. (I work in Seoul, Korea)

Finally, I truly express my deep gratitude to those who invented these wonderful tools (jupyter notebook, jupyterhub, jupyterlab, nbgrader, etc) and make it alive.

Best,

[jhamrick]

@psychemedia Hmm, could you clarify exactly how your adhoc use case would work after the graders are finished? nbgrader currently has a way of exporting grades from the database to a CSV file, so depending on what you're looking for one thing you could do would be to have each grader export their nbgrader grades and then you can concatenate them into one CSV. If you want to get all the grades back into a centralized version of nbgrader, that's a bit trickier.

@sdyang1729 thanks for commenting and for the kind words! Getting nbgrader to work with JupyterLab is definitely something I'd like to see happen (#1006) but isn't something I've had time to work on personally. I also have not yet done any development with JupyterLab and don't really use it so it would be much better for someone who is more familiar with it to take on that refactoring. But perhaps there will be someone at the hackathon who can work on this 🙂

[psychemedia]

@jhamrick After the marking, markers record marks on an individual basis in another system. In the current assessment model, we are looking mainly at how to mark cells manually and provide to students and provide easy to find summaries of marks on an individual basis (total mark; subtotal mark in subgroups of questions) for markers to then transfer to another system. nbgrader may or may not be the most appropriate tool for this but I am try to see what parts of nbgrader process we may be able to co-opt to help out markers out.

At the moment, assignment return by markers is an automated process: one folder per student, commented student script and a standard feedback form in each folder, then the whole lot zipped and returned. This process is a fixed legacy one, unchanged for years, that serves hundreds of courses across the institution. There is one course that uses notebooks for assessment... When it comes to returning marks, that's out of scope for use atm.

[jhamrick]

Since the hackathon is over, I am going to go ahead and close this. Unfortunately we didn't accomplish everything that was requested, but I think we did make a ton of progress nonetheless! (I suppose there's lots that can be improved about nbgrader 🙂). To see everything that was accomplished during the hackathon, see the project: https://github.com/jupyter/nbgrader/projects/1

Huge thanks to everyone who participated: @sigurdurb @damianavila @BertR @perllaghu @dsblank @danielmaitre @billywardrop29 @takluyver @jamesaslack @psychemedia 🎉💯

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/jupyter-nbgrader-hackathon-topics/1040/1