Skip to content
This repository was archived by the owner on Sep 21, 2026. It is now read-only.

Test debt from the step 7 and 8 verification: weak assertions and untested edges #63

Open

Description

@craig-dt

Collected from the /project:verify reviews of PRs #54 and #55. None are defects in shipped behaviour; all are places a plausible-but-wrong implementation would pass.

Tests that restate the code rather than constrain it

  • test_notice_terms_come_from_the_snapshot_the_labels_cite (test_labels.py) sets the manifest licence to MIT and the label licence to MIT, then asserts "MIT" in text. An implementation reading the licence from config.load_sources() — the exact defect the test is named for — passes. So does one that hardcodes "MIT". The disagreement test beside it is the real protection.
  • test_tools_records_the_zeek_flags_so_a_lost_D_is_visible (test_provenance.py) asserts the fixture's own ("-C","-D") survived a dict copy. It would not notice a lost -D; step 5 owns that gate. The docstring overclaims.

Untested edges in correlation

  • Widening-induced ambiguity. A detection (B, 0, A, 0) against flows (A,8,B,0) and (B,0,A,8) matches both only because of the ICMP counterpart rule; the clock then decides. This is the one place the widening changes candidate cardinality, and it is untested.
  • More than two candidates where exactly one contains the timestamp. Only n=2 is covered.
  • A malformed ICMP alert with no icmp_type yields (0, 0) from suricata.py, which then matches an ICMPv4 echo flow in reverse via counterpart[0] == 8 — a silent attribution from a record that carried no type at all. Wants a guard, or at minimum a test recording the behaviour.
  • SnapshotError on a detection that matched no flow. The identify-class equivalent is tested; this sibling path is not.
  • Zero-duration flows as multiple candidates (ts_first == ts_last), discussed in zeek.py's comments but never exercised in correlation.

Smaller notes worth carrying

  • _collect_failures dedupes by value, so two genuinely separate failures of the same tool with identical argv, exit code and message collapse to one. Probably desirable — the count is "distinct failures", not "failure events". Worth stating in the docstring.
  • Ties in sources and unmatched_detections fall back to input order (Python's sort is stable). Spec §10 dictates those exact sort keys so this is conformant, but it is a theoretical Goal 2 hole for entries identical in the key fields and differing elsewhere.
  • tests/test_labels.py imports test_provenance at test time to borrow MANDATORY_FIELDS. Right instinct — don't copy the list — but it couples two test modules through sys.path and breaks under pytest's --import-mode=importlib. Move the constant to tests/conftest.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions