Mention EXPECTED_RESULTS in the docs #1862
Conversation
doc/manual/problem-format.rst
Outdated
| @@ -12,6 +12,9 @@ On top, DOMjudge defines a few extensions: | |||
| distributed to participants. The file extension determines any of | |||
| three supported formats. If multiple files matching this pattern are | |||
| available, any one of those will be used. | |||
| * Annotation with ``EXPECTED_RESULTS`` for jury solutions. The ICPC | |||
| problem package specification always has precedence. Use this to annotate | |||
| possible outcomes a submission can have to use together with the Judging verifier. | |||
There was a problem hiding this comment.
It would be good to clarify how these tags should be used exactly, and maybe also refer to a sample submission.
BTW, if you're importing submissions from a problem package, then they have to be located inside a submissions/* directory, so will always be overridden, I guess.
There was a problem hiding this comment.
Agreed that a clarification on the usage would be nice. Perhaps a link to Kattis/problem-package-format#4 suffices, but I would at least write the tag in the exact required format (i.e., two @s and a space after the :).
Indeed, the @EXPECTED_RESULTS@: tag is overridden when the submission is in one of the standard submission directories (which are submissions/{accepted,wrong_answer,time_limit_exceeeded,run_time_error}/. Perhaps we can explicitly state that this tag therefore only works in non-standard submission directories (like mixed/ or rejected/).
There was a problem hiding this comment.
maybe also refer to a sample submission
I also agree with this one, but the current example submissions in https://github.com/DOMjudge/domjudge/tree/main/example_problems do not exactly match the spec, so linking to them in their current state may be confusing. For many of these submissions, the @EXPECTED_RESULTS@: tag is ignored because it lives in a "standard" submission directory. For BAPCtools, I've had to re-arrange many of these submissions to keep the tests passing (see RagnarGrootKoerkamp/BAPCtools@8b12d05) 😛
doc/manual/problem-format.rst
Outdated
| @@ -12,6 +12,9 @@ On top, DOMjudge defines a few extensions: | |||
| distributed to participants. The file extension determines any of | |||
| three supported formats. If multiple files matching this pattern are | |||
| available, any one of those will be used. | |||
| * Annotation with ``EXPECTED_RESULTS`` for jury solutions. The ICPC | |||
| problem package specification always has precedence. Use this to annotate | |||
| possible outcomes a submission can have to use together with the Judging verifier. | |||
There was a problem hiding this comment.
maybe also refer to a sample submission
I also agree with this one, but the current example submissions in https://github.com/DOMjudge/domjudge/tree/main/example_problems do not exactly match the spec, so linking to them in their current state may be confusing. For many of these submissions, the @EXPECTED_RESULTS@: tag is ignored because it lives in a "standard" submission directory. For BAPCtools, I've had to re-arrange many of these submissions to keep the tests passing (see RagnarGrootKoerkamp/BAPCtools@8b12d05) 😛
|
After #2006 is merged there will be a file we can link to as an example. |
Co-authored-by: Maarten Sijm <9739541+mpsijm@users.noreply.github.com>
See: https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#inline-literals, a solution might be: https://stackoverflow.com/a/31332035 but this feels more complicated.
|
|
Closes: #1861
Currently the EXPECTED_RESULTS are not part of the ICPC problem package and I'm not sure if they will be soon. We mention the EXPECTED_RESULTS in the README of the example problems but not in the doc. This PR tries to fix this.
I'm not sure how much we want to mention here as it will become part of the spec?
Still need to link to the Judging verifier also but that is hard to do in a PR via GitHub itself. If we agree on the wording I'll also make that link so the functionality is bundled together with its primary usage in the docs.We already link from the Judging Verifier to this page which is most likely the better route.