Skip to content

Add @EXPECTED_RESULTS@ tag. #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Add @EXPECTED_RESULTS@ tag. #4

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fb087d0
Add @EXPECTED_RESULTS@ tag.
RagnarGrootKoerkamp Nov 25, 2020
841b369
Change to verdict abbreviations; rename tag to EXPECTED_VERDICTS
RagnarGrootKoerkamp Dec 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 31 additions & 4 deletions spec/problem_format.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -396,12 +396,16 @@ subdirectories of `submissions/`. The possible subdirectories are:

| Value | Requirement | Comment |
| - | - | - |
| accepted | Accepted as a correct solution for all test files | At least one is required. |
| `accepted` | Accepted as a correct solution for all test files | At least one is required. |
| <s class="problemarchive"> partially\_accepted </s> | <s class="problemarchive"> Overall verdict must be Accepted. Overall score must not be max of range if objective is max and min of range if objective is min. </s> | <s class="problemarchive"> Must not be used for pass-fail problems. </s> |
| wrong\_answer | Wrong answer for some test file, but is not too slow and does not crash for any test file | |
| time\_limit\_exceeded | Too slow for some test file. May also give wrong answer but not crash for any test file. | |
| run\_time\_error | Crashes for some test file | |
| `wrong_answer` | Wrong answer for some test file, but is not too slow and does not crash for any test file | |
| `time_limit_exceeded` | Too slow for some test file. May also give wrong answer but not crash for any test file. | |
| `run_time_error` | Crashes for some test file | |
| `mixed_result` | The submission contains `@EXPECTED_VERDICTS@:` with expected verdicts, as explained below. | Use when the result is not deterministic. |

```todo
`mixed` vs `multiple` and `result` vs `verdict`.
```

<div class="problemarchive">

Expand All @@ -423,6 +427,29 @@ filenames. It is mandatory to provide at least one accepted solution.
Submissions must read input data from standard input, and write output
to standard output.

### EXPECTED_VERDICTS

Submissions in the `submissions/mixed_result` directory must contain an
`@EXPECTED_VERDICTS@: ` tag (including the final space) exactly once in one
of their source files. This must be followed (on the same line) by a non-empty
comma separated subset of:

- `AC`
- `WA`
- `TLE`
- `RTE`

Additional white space is ignored and the tags are not case sensitive.

This tag implies that the submission may get any of the listed verdicts as
final verdict.

If `@EXPECTED_RESULTS@: ` is found in a submission in any of the other
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't match the @EXPECTED_VERDICTS@ above.

directories, a warning is raised when the directory verdict is not listed. Note
that this does not have any other effect, and in particular only the
verdict matching the directory is allowed.


## Input Validators

Input Validators, for verifying the correctness of the input files, are
Expand Down