Scripts

discover.py — scan a range into the lukidis DB

Discovers live hosts with zmap, finds open ports with nmap, then performs protocol-specific grabs (HTTP/HTTPS, RTSP, SSH, generic TCP) so stored banners match the Shodan-shaped host/search API (minus vendor metadata like _shodan).

Org / ISP / coords prefer ip-api.com geo; ASN falls back to whois.

Dependencies

Tool	Role	Notes
zmap	Fast TCP SYN discovery	Usually needs root or CAP_NET_RAW
nmap	Open ports + light -sV	Full banners come from grabs, not NSE
whois	ASN / org fallback	Optional with --skip-whois
Python 3.10+	Orchestration + SQLite	stdlib only

# Debian/Ubuntu
sudo apt install zmap nmap whois

# Optional: run zmap without full root
sudo setcap cap_net_raw+ep "$(which zmap)"

./scripts/discover.py --check-deps

Database

Source	Default
--db	—
DATABASE_URL	from env (same as backend)
fallback	backend/lukidis.db

If the DB file is missing, migrations under backend/migrations/ are applied. banners.extras_json is ensured for protocol-specific root fields (merged by the API).

Usage

# Plan only
./scripts/discover.py --dry-run 203.0.113.0/24

# Full pipeline (default ports = crawl catalog, includes 80/443/554/…)
sudo ./scripts/discover.py --db sqlite:lukidis.db 203.0.113.0/24

# Single host, common web/camera ports
./scripts/discover.py --skip-zmap --max-hosts 1 --ports 80,443,554,8080 \
  --db sqlite:lukidis-user.db 154.70.175.216/32

# Skip enrichment / grabs
./scripts/discover.py --skip-whois --skip-geo --skip-grab --skip-zmap --max-hosts 1 8.8.8.8/32

What gets stored (Shodan-shaped)

Field	Source
All open ports in --ports	nmap
data	Raw HTTP/RTSP/SSH banner (+ parsed product summary when available)
http	status, headers, html, title, hashes, favicon, robots, security.txt…
product / version	Grab parsers + nmap fallback
Root extras (e.g. device modules)	banners.extras_json → merged into API match
org / isp / lat-lon	geo (preferred)
asn	geo, else whois

Re-scanning the same IP replaces non-historical banners for that IP/port/transport.

Limitations

• IPv4 only (zmap path).
• zmap only finds hosts that answer on --zmap-port (default 80).
• Full default port list is large; narrow with --ports for faster runs.
• Geo uses a public rate-limited API; prefer --skip-geo for large runs.
• Only scan networks you are authorized to probe.