Skip to content

Improve warning for when the ans file of a test case is not accepted by the validator #344

@gkreitz

Description

@gkreitz

We currently run the validator on every ans file provided, and emit a warning (or error, if in sample/) if the test case is not accepted. The logic is a bit flawed, but we'd need to restructure things a bit to get a clean way to express checks where the logic depends on the result across all test cases (we currently build up a class hierarchy very closely resembling the directory tree).

My impression is that it's typically the case that the answer file is either:

  1. The same format as what submissions should output.
  2. A completely different format with some auxiliary data needed by the output validator - here, every ans file is expected to get WA, so the current warning gets super spammy.

There's no way in the problem format package to indicate which of the two worlds we're in, but better logic for this check would be:

  • If the answer file gets a WA for all test cases, just add a notice.
  • If the answer file gets a WA for any test case in sample, that's an error (this part of the current check is good).
  • If the answer file gets a JE for any test case, that's an error.
  • If the answer file gets a WA for some test cases, but AC for others, that's a very strong warning (I'd love to make this an error, but this is technically fine, and could be valid if the validator only sometimes needs auxiliary data and the problem creator took care to avoid it for e.g. samples)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions