-
Notifications
You must be signed in to change notification settings - Fork 78
Open
Description
We currently run the validator on every ans file provided, and emit a warning (or error, if in sample/) if the test case is not accepted. The logic is a bit flawed, but we'd need to restructure things a bit to get a clean way to express checks where the logic depends on the result across all test cases (we currently build up a class hierarchy very closely resembling the directory tree).
My impression is that it's typically the case that the answer file is either:
- The same format as what submissions should output.
- A completely different format with some auxiliary data needed by the output validator - here, every ans file is expected to get WA, so the current warning gets super spammy.
There's no way in the problem format package to indicate which of the two worlds we're in, but better logic for this check would be:
- If the answer file gets a WA for all test cases, just add a notice.
- If the answer file gets a WA for any test case in sample, that's an error (this part of the current check is good).
- If the answer file gets a JE for any test case, that's an error.
- If the answer file gets a WA for some test cases, but AC for others, that's a very strong warning (I'd love to make this an error, but this is technically fine, and could be valid if the validator only sometimes needs auxiliary data and the problem creator took care to avoid it for e.g. samples)
Metadata
Metadata
Assignees
Labels
No labels