This is an offical CRS plugin that logs information about requests that is not normally logged.
We are planning to do performance testing. But that depends on real world sample traffic. Now the problem is there is relatively little literature on representative HTTP traffic.
How many HTTP headers are typically submitted in a request these days? How many parameters do you have in a query string on average? What size does a normal request body have?
If we have responses on these question, we can then create performance testing traffic that is actually resembling the real world traffic. So future performance optimization will actually result in improvements on normal traffic.
More information about this idea and existing scholarly literature on the subject: https://github.com/coreruleset/coreruleset/wiki/Dev-Retreat-2025-Sample-Traffic
Unfortunately, the literature is not comprehensive at all. We need to find out ourselves.
So this plugin extracts this information and writes it into the error.log. And it also brings an analyzer script that extracts the information and writes a summary report that is free from sensitive information and thus sharable.
The plugin writes a single log message (non-blocking alert) per request.
Put the contents of the plugin folder into your installation's CRS4 plugin folder. The plugin is active by default.
The plugin writes it's messages as JSON. Well almost as JSON. ModSec does not allow double quotes in the log messages, so we write single quotes and we need to translate that into double quotes before we can work on the data.
The alias meldata is part of a larger collection of useful ModSecurity aliases hosted together with a lot of tutorials at netnea.com. This is also where you can find the basicstats.awk script.
$ alias meldata='grep -o "\[data [^]]*" | cut -d\" -f2'
$ grep 9526200 error.log | tail -1
[2025-11-23 23:38:44.665603] [security2:error] 217.113.196.248:30561 aSOM9KWBJ_OKwMQcydUBEAAAAAo ModSecurity: Warning. Unconditional match in SecAction. [file "/etc/apache2/crs/plugins/traffic-observation-after.conf"] [line "100"] [id "9526200"] [msg "Logging traffic-observation-plugin data"] [data "{ 'Protocol' : 'HTTP/1.1', 'Method' : 'GET', 'LenFilename' : '10', 'LenQueryString' : '', 'NumQueryStringArgs' : '0', 'NumReqHeaders' : '6', 'LenCookies' : '', 'NumCookies' : '0', 'ReqContentType' : '', 'ReqContentLength' : '', 'StatusCode' : '200', 'NumRespHeaders' : '15', 'RespContentType' : 'application/rss+xml', 'RespContentLength' : '1479'}"] [ver "traffic-observation-plugin/1.0.0"] [tag "traffic-observation"] [hostname "www.netnea.com"] [uri "/index.php"] [unique_id "aSOM9KWBJ_OKwMQcydUBEAAAAAo"]
$ grep 9526200 error.log | meldata | tr "'" "\"" | tail -1 | jq
{
"Protocol": "HTTP/1.1",
"Method": "GET",
"LenFilename": "54",
"LenQueryString": "",
"NumQueryStringArgs": "0",
"NumReqHeaders": "3",
"LenCookies": "",
"NumCookies": "0",
"ReqContentType": "",
"ReqContentLength": "",
"StatusCode": "200",
"NumRespHeaders": "9",
"RespContentType": "text/html",
"RespContentLength": "3887"
}
$ grep 9526200 error.log | meldata | tr "'" "\"" | jq -r .NumReqHeaders | ~/bin/basicstats.awk
Num of values: 301.00
Mean: 7.88
Median: 6.00
Min: 2.00
Max: 16.00
Range: 14.00
Std deviation: 3.64There is a script bin/create-traffic-report.sh that processes the logs generated by
the plugin and writes a summary report in JSON format.
This report does not contain any sensitive information. It can therefore be submitted to the project. Providing a description of the service producing the logs and contact informatioin is optional.
The only mandatory command line option is the traffic type. It is one of web, api or mixed.
$ cat error.log | ./bin/create-traffic-report.sh -traffic-type mixed --contact "dune73" --description "1 day of production server netnea.com"Here is an example report: AGT-2025-11-24.json
The report covers the following data points:
| Field Name | Description |
|---|---|
| Contact | optional contact information |
| Date | date of submission of data |
| Description | optional description of service producing the logs |
| Protocol | HTTP protocol version |
| Method | HTTP method |
| LenFilename | Length of filename part of the URI |
| LenQueryString | Length of query string in bytes |
| NumQueryStringArgs | Number of query string arguments |
| NumReqHeaders | Number of request headers |
| LenCookies | Length of cookie header |
| NumCookies | Number of cookies |
| ReqContentType | Request content type distribution |
| ReqContentLength | Length of request body; actually this is value of the Content-Length request header |
| StatusCode | HTTP status code |
| NumRespHeaders | Number of response headers |
| RespContentType | Response content type distribution |
| RespContentLength | Length of response body; actually this is value of the Content-Length response header |
It is easy to look at the report by hand or with the help of jq.
$ cat report.json | jq -r .NumReqHeaders
{
"NumValues": 10246,
"Minimum": 0,
"Maximum": 20,
"Range": 20,
"Mean": 7.09936,
"Median": 7,
"StdandardDeviation": 4.05312,
"Variance": 16.4278,
"CoefficientOfVariation": 0.570913
}In order to optimize CRS for better performance we need to have an understanding of typical HTTP traffic.
And we need people to share information about their traffic with us, ideally in the format of the report described above.
Please submit your report via mail to FIXME or as a pull request into the traffic-reports folder of this repo.
No matter how you submit, please create a random file name as follows:
$ echo $(pwgen 3 1 | tr "a-z" "A-Z")-$(date +%Y-%m-%d).json
7PT-2025-11-24.jsonThere are services where a log message for every request is far too much.
The sampling mode of this plugin allows you to log only on a small percentage of requests.
The variable supports numbers from 0.0 to 100. So a single digit after the dot is supported, effectively bringing you permille (i.e. 1 permille means request in 1,000).
The sampling mode is off by default. That means the plugin writes a log message for
every request. If you want to use sampling, uncomment rule 9500020
in traffic-observation-config.conf and set the variable tx.traffic-observation-sampling_percentage
as you see fit.
Copyright (c) 2021-2025 OWASP CRS project. All rights reserved.
The OWASP CRS and its official plugins are distributed under Apache Software License (ASL) version 2. Please see the enclosed LICENSE file for full details.