Backfilling Data

You might need to start using the system partway through a rotation. There are some things to consider if you do.

Before you start:

• Disable email sending in the config file – it's probably easiest to copy the Python debug SMTP server config from the development docker-compose configuration (that is, copy the mailserver stanza from development/docker-compose.yaml into testing/docker-compose.yaml), and update the backend config file to refer to the hostname mailserver instead of the real mail server.
• Enable development mode – change DEVELOPER=false to DEVELOPER=true in the frontend section of docker-compose.yaml.
• Recreate the database and uploads folders.

The actual process to follow varies depending on what state the rotation is in. In the best case, students haven't yet been sent the list of available projects; in the worst case, students have already uploaded their reports. If the rotation is not too close to done, you should be able to use a single Grad Office account to fill in most of the details.

• Add all users to the system via the Edit Users page.
• Create the rotation, making sure to set the year and part correctly.
  • If the deadline for students to make their project choices has not yet passed, use the correct dates; otherwise, set all dates to some arbitrary point in the future, and fix them later (however, if any deadlines have not yet passed, you will make your life significantly easier by entering them correctly – in particular student_complete).
• Create all projects submitted so far, making sure to assign them to the correct supervisor.
  • If students have already been assigned to their projects, set up these assignments when you create the projects. If you make a mistake and assign the wrong student to a project, it will probably be easier to swap the project details than to move the student to another project, though you can also reassign the student via the database (make sure to remove them from the old project!) or just go through the normal finalisation process, although in that case you will need to reset all students' priorities afterwards, since you have not entered their votes.
• If the rotation is at the stage where students vote for projects, it will be easiest if you can get the students to use their own accounts to submit their votes into the system. If this isn't possible, you must set the votes by hand, directly in the database; whilst you can do this by writing SQL, it will probably be easier to use a Python REPL (see the instructions in the backend README), since you can more easily search for projects with a particular name, and assign projects directly to e.g. first_option instead of assigning IDs to e.g. first_option_id.
  • This step is essentially optional! The Graduate Office were obviously planning to allocate projects without use of the electronic Student Portal, so you can just let them do that as normal, and ask them to let you know when they've assigned everyone a project, then go from there.
• If CoGS markers have already been assigned to projects (in which case students must have already been assigned projects), assign them using the UI.
• If students have already started to submit their projects, you will need to set the rotation to student_uploadable (use the developer menu), then you can use the UI to upload projects for each student.
  • Think carefully about how each project's grace_deadline job will be scheduled when you do this – it's likely that they will be scheduled for the future, since student_complete will be some arbitrary date in the future, and they won't be rescheduled when you correct the deadlines later. If this job runs at the wrong time, you will confuse the markers – they might get emails asking them to mark a project that they've already marked or that they're not allowed to mark yet, or you might crash the application if the student hasn't yet uploaded their report.
• If markers have begun to give feedback on projects, you will need to effectively run the grace period job by hand – run something like UPDATE projects SET grace_passed=true WHERE uploaded=true;. You can then use the UI to upload feedback for each supervisor and CoGS member.

Afterwards, there is some cleanup required:

• If students have already chosen their projects and were assigned to them directly from the "Create Project" screen, you must unset their votes before the next rotation: you will need to run something like UPDATE users SET first_option_id=null;.
• If the deadline for students to make their project choices had passed, you will need to correct the rotation dates – you should do this one date at a time, starting from the earliest, and you must set each deadline to yesterday before you move it to the correct date, to allow the scheduled jobs to run (every time you change a deadline, press save). Check afterwards with the developer menu that the rotation is in the correct state.
• Make sure that all scheduled jobs are as you expect them to be. This is especially important if students have started to submit their reports, as the grace_deadline jobs aren't rescheduled when you change the student_complete deadline.
  • If there are jobs scheduled that should not run, you can just delete them from the database, and they will not be executed.
  • If there are jobs scheduled that should run at a different time (e.g. grace_deadline jobs), you cannot just update the next_run_time column. The job which is serialised into the job_state column also contains details of when the job should be executed, and it's not clear what would happen if they got out of sync. It's probably easiest to shut down the backend (but keep Postgres running), connect APScheduler to Postgres in a Python REPL, and change the trigger for the job via APScheduler. (If you are very brave, you can modify the serialised job directly; use pickletools.dis to disassemble the pickle, find the DateTriggers (note that the reminder jobs use CronTriggers, which look much uglier when pickled, and you will not want to do this with those), and use datetime.datetime.__reduce_ex__() to obtain a new bytestring to overwrite the pickled datetime.datetime with. Make sure to also update the next_run_time if you do this!)
  • If there are jobs that aren't scheduled that should be, you should schedule them (via APScheduler in a Python REPL, unless you want to construct a valid serialised job pickle yourself!), then restart the backend. If you don't restart the backend after adding a new job, it will likely be run very late (if at all!), because APScheduler sleeps for long periods between jobs, and unless it knows about the new job, it won't wake up to run it on time.
• Disable development mode – change DEVELOPER=true to DEVELOPER=false (or just git checkout testing/docker-compose.yaml).
• Re-enable emails – switch the backend config file back to the real mail server instead of mailserver, and remove the mailserver stanza from testing/docker-compose.yaml (or just git checkout testing/docker-compose.yaml).

If you want to send out emails, it's likely to be easier to do this by using a mail client, but it is possible to use a Python REPL to emulate the emails sent out by the system – see the code used to send emails, in the scheduled jobs and elsewhere.