[FEATURE REQUEST] Content-based auto filtering

[pich4ya]

feroxbuster performs auto-filtering based on HTTP response. By default, it will return result for the HTTP status code 200 204 301 302 307 308 401 403 405.

  -s, --status-codes <STATUS_CODE>...         Status Codes to include (allow list) (default: 200 204 301 302 307 308 401 403 405)

During a HackTheBox machine hacking, a machine contains the path /api/, in which, it returns HTTP status code 404, as same as other non-existing paths, but it contains unique/suspicious content length.

For example,

GET /api/ HTTP/1.1
Host: target.ltd

HTTP/1.1 404 Not Found
Server: nginx/1.14.0 (Ubuntu)
Date: Sun, 12 Feb 2023 07:59:50 GMT
Content-Type: application/json
Connection: keep-alive
Content-Length: 50

{"status":"404","status_text":"route not defined"}

and non-existing pages.

GET /non-existing/ HTTP/1.1
Host: target.ltd

HTTP/1.1 404 Not Found
Server: nginx/1.14.0 (Ubuntu)
Date: Sun, 12 Feb 2023 08:24:36 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Content-Length: 0

With ffuf (https://github.com/ffuf/ffuf), we can use the -mc all -ac options to handle this specific case automatically.

  -mc                 Match HTTP status codes, or "all" for everything. (default: 200,204,301,302,307,401,403,405,500)
 -ac                 Automatically calibrate filtering options (default: false)

However, feroxbuster does not contain such automatic mechanism to auto-filter HTTP Status Code 404 with Content-Length: 50 and Content-Length: 0. I know we can do manual /api/ and set --filter-size, but we cannot know beforehand if the web server will return which Content-Length for such existing paths like /api/ with the HTTP Status Code 404.

I do like --smart option on feroxbuster which does not exist in ffuf, however, feroxbuster does not have -mc all -ac.
Please consider adding them.

[epi052]

interesting.

I took a brief look at the auto calibration code in ffuf. my understanding of what's going on:

• make a request for the base directory; let the response be response-0
• make a few requests that shouldn't exist; let the responses be response-1..n
• compare size/word/line counts of response-1..n to response-0
  • when the size/word/line doesn't match response-0, add a filter for that specific size/word/line count

Does that sound correct?

[pich4ya]

interesting.

I took a brief look at the auto calibration code in ffuf. my understanding of what's going on:

• make a request for the base directory; let the response be response-0

• make a few requests that shouldn't exist; let the responses be response-1..n

• compare size/word/line counts of response-1..n to response-0

  • when the size/word/line doesn't match response-0, add a filter for that specific size/word/line count

Does that sound correct?

Likely yes!

I think feroxbuster should provide a way to not only make decision on the HTTP status code for its auto-filtering mechanism. It should take the content-length into consideration as well. So, that the 98% of HTTP status 404 with responses Content-Length: 0 should be hidden, and the 2% of responses of HTTP status 404 with the Content-Length: 50 should be displayed to the user.

[epi052]

when you use the -mc all -ac options in ffuf, i assume you're using -u http://.../api/ and you know that /api/ is a valid endpoint; is that correct?

i'm guessing that the assumption for this to work is the user pointed the tool at a valid endpoint to begin with

[epi052]

notes for whoever implements this:

• my initial guess at the base url being used in this comparison is wrong. the responses[0] is actually one of the random responses mentioned below.

ffuf performs the following actions:

• generates 6 random paths, 3 of length 8 and 3 of length 16. 2 are fully random, 2 start with admin, and 2 start with .htaccess. src
  • /TNwNeZTN
  • /CQiVsqyKXhDeqFCt
  • /.htaccessaDNFyTrS
  • /.htaccessqvdNYLWaPjZnQESQ
  • /adminJEcAbqyO
  • /adminSkHCYWgnbhYLlCUY
• optionally generates 4 more paths the same way it generated the fully random and admin strings, however, these 4 end with a / character src
• sends those 6 requests and performs the following analysis on the responses
  • content length check
    • lookup resonse[0]'s content length src
    • set a sentry value to true
    • if all responses content length match each other
      • check to see if the content length is already filtered, if not, add a filter for it src
      • return from function, nothing else to do
  • number of words check
    • same exact steps as the content length check, just less precise
    • if a filter is found, check it, add it, return from function
  • number of lines check
    • same as the other two, just the least precise

based on the actions the code takes, the example provided in the ticket would have requested a bunch of non-existent pages, each (presumably) having a content length of 0. Then when /api/ was requested, it had a content length of 50, allowing it to slide through the filter(s).

this logic falls squarely in feroxbuster's heuristics module. implementation should roughly follow the steps outlined above.

[epi052]

somewhat related: #635

[epi052]

happened to find a machine that replied in this exact way 😉

i reworked how heuristics picks up 404-like pages, to include the traditional wildcards. I tested the new logic against that machine and it works as desired.

I'm considering just making auto-detection of 404s and allowing all status codes by default.

[epi052]

@all-contributors add @pich4ya for ideas

[allcontributors]

@epi052

I've put up a pull request to add @pich4ya! 🎉