Verify hosts are alive before emitting wayback URLs

[liquidsec]

Problem

With urls=True, wayback emits a URL_UNVERIFIED for every URL that survives filtering, with no check that the host still exists. A CDX response for a large domain routinely contains hundreds of hosts that are long dead, and every one of their URLs gets emitted for the rest of the scan to chase: http requests each one, excavate parses the results, the spider follows them. That cost is large and entirely wasted.

Nothing in the emission path checked liveness. _pre_process_urls applies an in-scope test, but only for the archive/parameter/interesting-file metadata; the URL_UNVERIFIED list bypassed it. abort_if runs after DNS resolution but only requires the in-scope tag, which an unresolvable host still carries.

Change

Probe each (scheme, netloc) once with HEAD / before emitting any of its URLs, and drop the URLs of hosts that don't answer.

• Batched through request_batch_stream at 25 concurrency, follow_redirects=False, so it's one request per host regardless of URL count.
• Any response counts as alive, including 4xx/5xx. Only connect failures and timeouts mark a host dead.
• Explicit retries=1 rather than inheriting web.http_retries, so a single dropped packet can't condemn a live host even if a user sets that to 0.
• Verdicts cached in an LRUCache(maxsize=40000), measured at 9.44 MB when full with realistic hostnames (248 bytes/entry).

Only reachable from the urls=True branch, so the default config sends no probes.

The archive path is deliberately unaffected

_archive_cache is populated before the liveness gate. Dead hosts are exactly what the archive feature exists for, so finish() still fetches every snapshot it would have before. Eviction behavior is unchanged too: it only fires on a live 2xx URL event, which a dead host never produced anyway. The six existing archive tests all run against http://127.0.0.1:1/..., i.e. dead hosts, and pass unchanged.

_parameter_cache is now only populated for live hosts. Those entries could only ever be read when a live 2xx URL event arrived, so nothing is lost.

Tests

New TestWaybackDeadHostSkip: two hosts with identical DNS, only one answering HTTP. Asserts only the live host's URLs are emitted, that the host is probed exactly once regardless of URL count, and the resulting cache contents.

Seven existing tests needed a liveness mock, since their hosts are mock-intercepted and an unmatched URL now reads as dead. TestWaybackParameters got an explicit / handler rather than relying on the httpserver's 500-fallback.

20 passed, 0 failed.

Notes

• Verdicts are cached for the whole scan, so a host that's briefly unreachable at probe time loses its URLs for the rest of the run. The retry is the mitigation.
• http-only wayback records for a host that has since moved to HTTPS-only now get dropped rather than emitted. They were already dead ends, since http would have failed the same connection.
• This makes the module send target traffic when urls=True while it's still flagged passive. That's a pre-existing inconsistency (_is_http_wildcard_host already probes targets in every config) and is tracked separately in Module flags are static, but some modules are active or passive depending on config #3358 rather than fixed here.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs wayback-host-liveness (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	4.25ms	4.32ms	+1.8% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	17.78ms	17.64ms	-0.8% ⚪	✅
Large Closest Match Lookup	341.13ms	327.47ms	-4.0% ⚪	✅
Realistic Closest Match Workload	179.37ms	177.47ms	-1.1% ⚪	✅
Event Memory Medium Scan	1404 B/event	1402 B/event	-0.2% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	401.70ms	402.54ms	+0.2% ⚪	✅
Event Validation Full Scan Startup Large Batch	515.43ms	510.86ms	-0.9% ⚪	✅
Make Event Autodetection Small	20.30ms	20.29ms	-0.0% ⚪	✅
Make Event Autodetection Large	206.14ms	206.13ms	-0.0% ⚪	✅
Make Event Explicit Types	8.48ms	8.46ms	-0.3% ⚪	✅
Excavate Single Thread Small	3.941s	3.758s	-4.6% ⚪	✅
Excavate Single Thread Large	9.503s	9.448s	-0.6% ⚪	✅
Excavate Parallel Tasks Small	3.970s	3.928s	-1.1% ⚪	✅
Excavate Parallel Tasks Large	6.007s	5.957s	-0.8% ⚪	✅
Intercept Throughput Small	871.33ms	876.15ms	+0.6% ⚪	✅
Intercept Throughput Medium	870.90ms	864.66ms	-0.7% ⚪	✅
Dns Throughput Quiet	2.549s	2.547s	-0.1% ⚪	✅
Dns Throughput Loaded	1.735s	1.754s	+1.1% ⚪	✅
Dns Throughput Inherited	2.225s	2.217s	-0.4% ⚪	✅
Is Ip Performance	2.28ms	2.33ms	+2.3% ⚪	✅
Make Ip Type Performance	211.07µs	209.05µs	-1.0% ⚪	✅
Mixed Ip Operations	2.40ms	2.41ms	+0.6% ⚪	✅
Memory Use Web Crawl	1.0 MB	1.0 MB	+0.0% ⚪	✅
Memory Use Subdomain Enum	28.3 MB	28.3 MB	+0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	12.0 MB	13.8 MB	+15.5% 🟡🟡	⚠️
Scan Throughput 100	2.575s	2.568s	-0.2% ⚪	✅
Scan Throughput 1000	18.144s	18.107s	-0.2% ⚪	✅
Typical Queue Shuffle	5.69µs	5.66µs	-0.5% ⚪	✅
Priority Queue Shuffle	26.90µs	27.20µs	+1.1% ⚪	✅

🎯 Performance Summary

! 1 regression ⚠️
  30 unchanged ✅

🔍 Significant Changes (>10%)

• Memory Use Parallel Chains: 15.5% 🐌 more memory

---

🐍 Python Version 3.11.15

[codecov]

Codecov Report

❌ Patch coverage is 95.16129% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (3be13de) to head (7c52a63).

Files with missing lines	Patch %	Lines
bbot/modules/wayback.py	92%	3 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3359   +/-   ##
=====================================
- Coverage     90%     90%   -0%     
=====================================
  Files        450     450           
  Lines      46327   46387   +60     
=====================================
+ Hits       41588   41632   +44     
- Misses      4739    4755   +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.