Create a plugin for clustering a huge amount of scan results

[artem-smotrakov]

Most of existing plugins are based on static checks like searching for pre-defined patterns (for example, they can look for typical error messages from database servers). This approach allows to catch only issues which the plugins are aware of. But a scan may cause an application to behave in an unusual and unexpected way which actually may be a vulnerability. This may be a logical bug or just very application-specific. If such a behavior doesn't match to the patterns which used by plugins, then most probably no one notice the problem. In particular, the more extensive a scan is, and the less likely a tester notices a problem because it may require a (semi)manual analysis of a huge amount or scan logs.

To help a tester with analysis of scan results, we can add a plugin which applies some machine learning techniques to the scan results. For example, the plugin can apply a clustering algorithm to HTTP requests and responses which put similar ones to the same bucket (cluster). Then, the tester can review a couple of random samples from each cluster to make sure that the application behaved correctly.

This plugin doesn't guarantee that all issues are found during testing. In fact, it should be even fine if the plugin doesn't report issues at all. This plugin should be considered as a tool which helps with analysis of huge amount of scan results.

Hope I can find some time to prototype such a plugin. Meanwhile, I'd appreciate any feedback.

[artem-smotrakov]

See also #17171

[andresriancho]

Not sure how related it is, but when I read this issue I thought about Burp's Backslash Powered Scanner.

[artem-smotrakov]

I read Backslash Powered Scanning: hunting unknown vulnerability classes, it looks a bit related but Backslash Powered Scanner tries to discover new interesting payloads, and then detect anomalies. The plugin I described above is a completely passive tool - it's supposed to analyze results produced by other plugins.

[floyd-fuh]

I brought up the idea of response clustering in 2010 and submitted a grep plugin. But it seems it never made it into w3af. Don't ask me why, I don't remember. It was called clusterResponseBodies. Here's the discussion:
https://sourceforge.net/p/w3af/mailman/message/26445306/
The further links seem broken. For some of the performance optimizations for similarity matching in w3af we've done my main motivation was that such a response clustering would work okish performance-wise.

By now I implemented it for Burp, see https://portswigger.net/bappstore/e63f09f290ad4d9ea20031e84767b303 and I use successfully to find various interesting things. It is very helpful.

Please don't call this machine learning. The absolute bare minimum to qualify as machine learning would be a neural network, which this isn't.

[andresriancho]

Not sure why this wasn't merged back in 2010!

For reference, the code that @floyd-fuh wrote and is in burp can be found at https://github.com/PortSwigger/response-clusterer/blob/master/ResponseClusterer.py

At this point I believe that w3af has taken the route of "point and shoot" scanner, more than a tool for manual analysis. Given that these response clustering methods tend to require a lot of manual analysis and are false-positive prone, I'm not sure how useful they would be for the regular user.