Excavate / HTTPX / DNSresolve Analyzing Out of Scope Links

[Sh4d0wHunt3rX]

So, scan takes so much longer then usual:

debug.log

Thanks 🙏

[TheTechromancer]

Looked a bit deeper into this and found lots of events like this:

{
  "type": "RAW_DNS_RECORD",
  "id": "RAW_DNS_RECORD:24591d5877d17b2482761275ace1b72d4faafaec", 
  "scope_description": "distance-3",
  "data": {
    "host": "burkhan.ru",
    "type": "MX",
    "answer": "10 mx1.beget.com."
  },
  "host": "burkhan.ru",
  "resolved_hosts": [
    "45.130.41.33"
  ],
  "dns_children": {},
  "web_spider_distance": 0,
  "scope_distance": 3,
  "scan": "SCAN:b9eaa43d357e7129e6320df579071b7203cb0052",
  "timestamp": "2024-08-27T20:47:48.014105+00:00",
  "parent": "DNS_NAME:c039ff1555eb06218fee0c14d3fee9257d709595",   
  "tags": [
    "mx-record",
    "distance-3"
  ],
  "module": "dnsresolve",
  "module_sequence": "dnsresolve",
  "discovery_context": "MX lookup on burkhan.ru produced RAW_DNS_RECORD",
  "discovery_path": [
    [
      "DNS_NAME:fc89b3e576d30033e7ef2f144196c2c156866ade",
      "Scan foreboding_ciri seeded with DNS_NAME: superama.com"    
    ],
    [
      "CODE_REPOSITORY:fd8d50bc6134f04f05c8ad0abf8a7036d69f6d06",  
      "github_codesearch discovered CODE_REPOSITORY: {'url': 'https://github.com/tech234a/annotation-urls'}"
    ],
    [
      "URL_UNVERIFIED:5d405974c7e64e936cd522815573f4d392e3618f",   
      "file matching query \"superama.com\" is at URL_UNVERIFIED: https://raw.githubusercontent.com/tech234a/annotation-urls/5c0300471b87c6a5f3dd8883484f8155b5ff33d7/kX_urls.txt"
    ],
    [
      "URL:5d405974c7e64e936cd522815573f4d392e3618f",
      "httpx visited https://raw.githubusercontent.com/tech234a/annotation-urls/5c0300471b87c6a5f3dd8883484f8155b5ff33d7/kX_urls.txt and got status code 200"
    ],
    [
      "HTTP_RESPONSE:5b1f906c19991c1de452a3abc3b232cb2d732469",    
      "HTTP_RESPONSE was 233.98KB with text/plain content type"
    ],
    [
      "URL_UNVERIFIED:d8754931ba878130a401b17e3da052cb6b123be6",   
      "Excavate's URLExtractor emitted URL_UNVERIFIED http://burkhan.ru/vopros, because HTTP response (body) contains full URL"
    ],
    [
      "DNS_NAME:c039ff1555eb06218fee0c14d3fee9257d709595",
      "URL_UNVERIFIED has host DNS_NAME: burkhan.ru"
    ],
    [
      "RAW_DNS_RECORD:24591d5877d17b2482761275ace1b72d4faafaec",   
      "MX lookup on burkhan.ru produced RAW_DNS_RECORD"
    ]
  ]
}

Basically, github_codesearch is searching for the target domain and finds a garbage repo with a metric ton of URLs, which all get extracted by excavate, speculated to DNS_NAMEs, and resolved by dnsresolve.

The question is, where should we cut this off?

[TheTechromancer]

There are two places this should be stopped, and they're both in dnsresolve:

1. dnsresolve should not extract hosts from events that are outside of dns.search_distance.
2. dnsresolve should not resolve events that are outside of dns.search_distance.

[liquidsec]

There are two places this should be stopped, and they're both in dnsresolve:

1. dnsresolve should not extract hosts from events that are outside of dns.search_distance.
2. dnsresolve should not resolve events that are outside of dns.search_distance.

I think i'd slightly prefer option 2, just because it feels closer to what I would expect dns.search_distance to mean given its name

[TheTechromancer]

Fixed in #1723.