Document that `@EXPECTED_RESULTS@: ` tag is ignored for standard submission directories

[mpsijm]

Description of the problem

The @EXPECTED_RESULTS@: tag is ignored when an imported submission is one of the standard submission directories (e.g., accepted/, wrong_answer/) since 3b9ec83. This is not documented anywhere and led to confusion for me during the last few contests.

Your environment

DOMjudge, since 7.0.0

Current behaviour

Some examples, when importing a problem zip:

• For submissions/accepted/solution.py, the expected result is ["CORRECT"].
• For submissions/accepted/solution.py with any @EXPECTED_RESULTS@: (...) tag, the expected result is ["CORRECT"].
• For submissions/mixed/solution.py with @EXPECTED_RESULTS@: ACCEPTED,TIME_LIMIT_EXCEEDED tag, the expected result is ["CORRECT","TIMELIMIT"].
  • Also, after importing, the info alert at the top of the page says "Annotated result 'CORRECT, TIMELIMIT' does not match directory for submissions/mixed/solution.py".
• For submissions/mixed/solution.py without any @EXPECTED_RESULTS@: tag, the expected result is ["MIXED"].
  • The info alert at the top of the page shows no warning for this, but perhaps that would be a good idea to add 😛

Any other information that you want to share?

I wouldn't mind if the commit linked above would be reverted (i.e., that the @EXPECTED_RESULTS@: tag would take precedence over the submission's directory name again). But you probably have a good reason to keep the current behaviour 😇

[vmcj]

I think the reason for this ordering is that stuff which is in the spec takes precedence over things which are not in it. Otherwise you would have different results between DOMjudge and another system following the CLICS spec when uploading the problemset.

[eldering]

Indeed, there are two different methods to classify the expected outcome (directory and @EXPECTED_RESULTS@ tag), and we somehow have to make a decision which one to prioritize. I think the current one is relatively simple, but might indeed not be expected, so would be good to document this.
FYI, there's an unresolved issue on how to add the tags to the problem package specification, see Kattis/problem-package-format#4

[mpsijm]

Thanks for your explanations and for the PR to the documentation! I've updated BAPCtools to match DOMjudge's behaviour regarding @EXPECTED_RESULTS@: at RagnarGrootKoerkamp/BAPCtools#242, that should also help lessen confusion in the future ☺️