Needs Consolidation is buggy #65
Description
When graders disagree by more than a threshold, another student is supposed to be assigned as an additional grader. This is determined in an automatic (i.e., done by the system, not a user) Needs Consolidation task following an Assessment task with siblings. If the assessments differ by more than the threshold amount then the Consolidation task is triggered for a user, otherwise the Consolidation task is bypassed. (The Consolidation task is followed by a Dispute task, if disputes are allowed. If an Assessment task has only one TI instance, then this is unnecessary and there should be no Needs Consolidation task or Consolidation task generated, but instead the Dispute task would be next.)
(Problem 1) What's the problem?: Needs consolidation is unreliable. The first image below will show you two needs consolidations TIs in the same WA. One triggered the consolidation task, and the other bypassed the consolidation task, even though the assessment scores for the latter had a larger difference. This was an example with numeric grades.
The following image shows a threshold of 10%, the prior tasks from TI 6416, which shows the two Grade Analysis TIs 6410 and 6411, which clearly are more than 10% different, but that the Consolidate Grades task was bypassed.
Approach: TaskTrigger.js > findGrades appears to be where the threshhold is checked (called by TaskTrigger > needsConsolidate). (Triggering the task takes place in Tasktrigger > trigger.)
As a side note: It is not clear whether the threshold should apply to each grading field in the assessment task, or to the sum of all grading field scores. If this needs to be part of a fix, we can implement either one. For simplicity we can choose the latter for now.
(Problem 2) Needs Consolidation and Consolidation tasks should not be included in the workflow (or at least the Consolidation task should always be bypassed) if there was only a single assessment task (i.e., no siblings). The following screenshot shows that this is not happening.
Approach: This is probably generated when the assignment editor creates the assignment, which appears to take place in TaskFactory. I'm not sure if we handle it here, or where the assignment specs are generated (which may be in this file - need to ask Jimmy or search further).
(Problem 3) Another problem is with non-numerical grades such as labels. Here, Jimmy tells me that needs consolidation definitely does not function correctly. (I don't have any images of this.) Labels are converted to a numeric value by giving them even values along a scale. For example if there are 3 labels, the first would be worth 33.3%, the second 66.6%, the third 100%.
Approach: Individual fields appear to be converted to numeric equivalents towards the top of TaskTrigger.js > findGrades. (TaskTrigger.js > findGrades appears to be where the threshhold is checked (called by TaskTrigger > needsConsolidate).)
(Problem 4) Also, we should ensure that if the threshold is 0% that consolidation will always trigger, and if the threshold is 100% that consolidation never triggers.
Approach: This can probably be inserted into TaskTrigger.js > findGrades where max, min and average are handled. (findGrades appears to be where the threshhold is checked (called by TaskTrigger > needsConsolidate).)