The malicious canary is unsourced: Goal 5's sensitivity half is unmet #103

Filed at Phase 1 sign-off, because the gap had no open tracker. docs/status.yaml's phase_1_signoff.deviations cites #24 as where this lives, and #24 — "Step 10: Canaries and reproducibility gates" — was correctly closed when step 10 merged. So the one PRD success criterion Phase 1 ships without was being tracked by a closed build issue and a skipif.

The gap

benign.pcap proves specificity: zero labels on traffic that is benign by construction. Its pair proves sensitivity, and without it a pipeline that silently stopped labelling anything would pass every gate we have. That is the whole reason the pair exists.

tests/integration/test_canaries.py::test_the_malicious_canary_produces_at_least_one_label skips while tests/fixtures/malicious.pcap is absent and activates itself the moment it lands, so the gap stays visible in test output rather than disappearing from it. That skip is the current marker and should stay until this closes.

Accepted, not overlooked

Craig accepted this for Phase 1 sign-off on 2026-08-13 after the most promising candidate was ruled out by measurement, not by assumption:

benign-corpus/smb-eicar-file-segmentation-random.pcap clears licence, personal-data and size outright. It fails on detectability. ET Open is the only feed of the nine carrying any EICAR rule, and none of its three enabled ones detects EICAR as such — 2022932 and 2022933 require a Symantec MIME name-overflow shape, 2039680 requires an x-powered-by: Kaspersky Labs response header, and all three are alert http while the fixture is SMB. The one rule matching the raw EICAR string, 2012591, is # ET DELETED upstream and also demands an MZ header. The metadata filter was never the blocker.

What a candidate must clear

Recorded in tests/fixtures/README.md, in order of how likely each is to block:

1. A licence permitting redistribution in a public repo. Not merely "publicly downloadable".
2. No real personal or customer data. The criterion most published malware captures fail.
3. Small — single-digit MB; it is fetched on every scheduled run.
4. It must fire a rule in an admitted source, after admission. A capture tripping only a rule the metadata filter excludes proves nothing about this pipeline.
5. Documented: origin URL, licence, publication date, and which sid it is expected to trigger. "Something, probably" cannot detect a sensitivity regression — only a total outage.

Candidate sources whose terms are worth reading first: Netresec's published captures, the Stratosphere IPS datasets, malware-traffic-analysis.net.

The alternative, if sourcing stays blocked

A synthesized positive control targeting one stable admitted sid would catch a total labelling outage against the live ruleset, which is currently caught only by the PR suite against a small synthetic snapshot. It would not prove sensitivity on realistic traffic. tests/fixtures/README.md argues against synthesizing this canary, and that argument stands — but a narrow smoke control is a different thing from the canary, and worth considering separately if this issue stays open.