This repository was archived by the owner on Sep 21, 2026. It is now read-only.
Test debt from the step 7 and 8 verification: weak assertions and untested edges #63
Copy link
Copy link
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Collected from the /project:verify reviews of PRs #54 and #55. None are defects in shipped behaviour; all are places a plausible-but-wrong implementation would pass.
Tests that restate the code rather than constrain it
test_notice_terms_come_from_the_snapshot_the_labels_cite(test_labels.py) sets the manifest licence to MIT and the label licence to MIT, then asserts"MIT" in text. An implementation reading the licence fromconfig.load_sources()— the exact defect the test is named for — passes. So does one that hardcodes"MIT". The disagreement test beside it is the real protection.test_tools_records_the_zeek_flags_so_a_lost_D_is_visible(test_provenance.py) asserts the fixture's own("-C","-D")survived a dict copy. It would not notice a lost-D; step 5 owns that gate. The docstring overclaims.
Untested edges in correlation
- Widening-induced ambiguity. A detection
(B, 0, A, 0)against flows(A,8,B,0)and(B,0,A,8)matches both only because of the ICMP counterpart rule; the clock then decides. This is the one place the widening changes candidate cardinality, and it is untested. - More than two candidates where exactly one contains the timestamp. Only n=2 is covered.
- A malformed ICMP alert with no
icmp_typeyields(0, 0)fromsuricata.py, which then matches an ICMPv4 echo flow in reverse viacounterpart[0] == 8— a silent attribution from a record that carried no type at all. Wants a guard, or at minimum a test recording the behaviour. SnapshotErroron a detection that matched no flow. The identify-class equivalent is tested; this sibling path is not.- Zero-duration flows as multiple candidates (
ts_first == ts_last), discussed inzeek.py's comments but never exercised in correlation.
Smaller notes worth carrying
_collect_failuresdedupes by value, so two genuinely separate failures of the same tool with identical argv, exit code and message collapse to one. Probably desirable — the count is "distinct failures", not "failure events". Worth stating in the docstring.- Ties in
sourcesandunmatched_detectionsfall back to input order (Python's sort is stable). Spec §10 dictates those exact sort keys so this is conformant, but it is a theoretical Goal 2 hole for entries identical in the key fields and differing elsewhere. tests/test_labels.pyimportstest_provenanceat test time to borrowMANDATORY_FIELDS. Right instinct — don't copy the list — but it couples two test modules throughsys.pathand breaks under pytest's--import-mode=importlib. Move the constant totests/conftest.py.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
test: five gate weaknesses, each verified by breaking it (#92, #95) (#…