-
Notifications
You must be signed in to change notification settings - Fork 26
HTTP
Tom Sellers edited this page Sep 3, 2021
·
10 revisions
The data referenced in this document can be found in the HTTP GET Responses and HTTPS GET Responses sections of the Rapid7 Open Data website.
Project Sonar produces a HTTP/HTTPS data on a regular basis. This data is gathered by first performing a TCP SYN scan across the entire IPv4 address space for certain ports known to host HTTP services. A collection script is then run against every system that returned a positive response. The collection script sends a HTTP GET / HTTP/1.1 request with the Host header set to the IP address of the system.
The data format is gzip-compressed JSON, with one record per line, and the complete HTTP response, including the HTTP headers stored as a base64-encoded string in the "data" key. The example below grabs the first 10 results from the 2014-12-09 scan.
Note that many servers require a known hostname to be sent in the Host header and return a 400 response if only the IP address is sent, as this scan does. We have experimented with a VHOST-enabled scanner (leveraging the FDNS dataset), but this results in over 1.5Tb of compressed data per scan. The current GET scan results in 64Gb or more data per run.
The current collection script will save up to 150K of data per host.
curl -s https://scans.io/data/rapid7/sonar.http/20141209-http.gz | \
zcat | head -n 10
{"vhost": "23.41.188.80", "host": "23.41.188.80", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTQNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2O2JkMDFmZWE1JiM0NjsxNDE4MjA3MzAwJiM0NjsxNzViODJiNAo8L0JPRFk+PC9IVE1MPgo=", "port": 80, "ip": "23.41.188.80"}
{"vhost": "141.101.123.250", "host": "141.101.123.250", "data": "SFRUUC8xLjEgNDAzIEZvcmJpZGRlbg0KRGF0ZTogV2VkLCAxMCBEZWMgMjAxNCAxMDoyODoyMCBHTVQNCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PVVURi04DQpUcmFuc2Zlci1FbmNvZGluZzogY2h1bmtlZA0KQ29ubmVjdGlvbjogY2xvc2UNClNldC1Db29raWU6IF9fY2ZkdWlkPWQzMzYzZDlhZjc4ZjkxNjBjNDg4NWVlZjgzYmQyNTg3ZTE0MTgyMDczMDA7IGV4cGlyZXM9VGh1LCAxMC1EZWMtMTUgMTA6Mjg6MjAgR01UOyBwYXRoPS87IEh0dHBPbmx5DQpDYWNoZS1Db250cm9sOiBtYXgtYWdlPTE1DQpFeHBpcmVzOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjM1IEdNVA0KU2VydmVyOiBjbG91ZGZsYXJlLW5naW54DQpDRi1SQVk6IDE5NjhjMTRlYTgwOTBjM2ItU0VBDQoNCmUwNQ0KPCFET0NUWVBFIGh0bWw+CjwhLS1baWYgbHQgSUUgN10+IDxodG1sIGNsYXNzPSJuby1qcyBpZTYgb2xkaWUiIGxhbmc9ImVuLVVTIj4gPCFbZW5kaWZdLS0+CjwhLS1baWYgSUUgN10+ICAgIDxodG1sIGNsYXNzPSJuby1qcyBpZTcgb2xkaWUiIGxhbmc9ImVuLVVTIj4gPCFbZW5kaWZdLS0+CjwhLS1baWYgSUUgOF0+ICAgIDxodG1sIGNsYXNzPSJuby1qcyBpZTggb2xkaWUiIGxhbmc9ImVuLVVTIj4gPCFbZW5kaWZdLS0+CjwhLS1baWYgZ3QgSUUgOF0+PCEtLT4gPGh0bWwgY2xhc3M9Im5vLWpzIiBsYW5nPSJlbi1VUyI+IDwhLS08IVtlbmRpZl0tLT4KPGhlYWQ+Cjx0aXRsZT5EaXJlY3QgSVAgYWNjZXNzIG5vdCBhbGxvd2VkIHwgMTQxLjEwMS4xMjMuMjUwIHwgQ2xvdWRGbGFyZTwvdGl0bGU+PC90aXRsZT4KPG1ldGEgY2hhcnNldD0iVVRGLTgiIC8+CjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PVVURi04IiAvPgo8bWV0YSBodHRwLWVxdWl2PSJYLVVBLUNvbXBhdGlibGUiIGNvbnRlbnQ9IklFPUVkZ2UsY2hyb21lPTEiIC8+CjxtZXRhIG5hbWU9InJvYm90cyIgY29udGVudD0ibm9pbmRleCwgbm9mb2xsb3ciIC8+CjxtZXRhIG5hbWU9InZpZXdwb3J0IiBjb250ZW50PSJ3aWR0aD1kZXZpY2Utd2lkdGgsaW5pdGlhbC1zY2FsZT0xLG1heGltdW0tc2NhbGU9MSIgLz4KPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBpZD0iY2Zfc3R5bGVzLWNzcyIgaHJlZj0iL2Nkbi1jZ2kvc3R5bGVzL2NmLmVycm9ycy5jc3MiIHR5cGU9InRleHQvY3NzIiBtZWRpYT0ic2NyZWVuLHByb2plY3Rpb24iIC8+CjwhLS1baWYgbHQgSUUgOV0+PGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBpZD0nY2Zfc3R5bGVzLWllLWNzcycgaHJlZj0iL2Nkbi1jZ2kvc3R5bGVzL2NmLmVycm9ycy5pZS5jc3MiIHR5cGU9InRleHQvY3NzIiBtZWRpYT0ic2NyZWVuLHByb2plY3Rpb24iIC8+PCFbZW5kaWZdLS0+CjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+Ym9keXttYXJnaW46MDtwYWRkaW5nOjB9PC9zdHlsZT4KPCEtLVtpZiBsdGUgSUUgOV0+PHNjcmlwdCB0eXBlPSJ0ZXh0L2phdmFzY3JpcHQiIHNyYz0iL2Nkbi1jZ2kvc2NyaXB0cy9qcXVlcnkubWluLmpzIj48L3NjcmlwdD48IVtlbmRpZl0tLT4KPCEtLVtpZiBndGUgSUUgMTBdPjwhLS0+PHNjcmlwdCB0eXBlPSJ0ZXh0L2phdmFzY3JpcHQiIHNyYz0iL2Nkbi1jZ2kvc2NyaXB0cy96ZXB0by5taW4uanMiPjwvc2NyaXB0PjwhLS08IVtlbmRpZl0tLT4KPHNjcmlwdCB0eXBlPSJ0ZXh0L2phdmFzY3JpcHQiIHNyYz0iL2Nkbi1jZ2kvc2NyaXB0cy9jZi5jb21tb24uanMiPjwvc2NyaXB0PgoKPC9oZWFkPgo8Ym9keT4KICA8ZGl2IGlkPSJjZi13cmFwcGVyIj4KICAgIDxkaXYgY2xhc3M9ImNmLWFsZXJ0IGNmLWFsZXJ0LWVycm9yIGNmLWNvb2tpZS1lcnJvciIgaWQ9ImNvb2tpZS1hbGVydCIgZGF0YS10cmFuc2xhdGU9ImVuYWJsZV9jb29raWVzIj5QbGVhc2UgZW5hYmxlIGNvb2tpZXMuPC9kaXY+CiAgICA8ZGl2IGlkPSJjZi1lcnJvci1kZXRhaWxzIiBjbGFzcz0iY2YtZXJyb3ItZGV0YWlscy13cmFwcGVyIj4KICAgICAgPGRpdiBjbGFzcz0iY2Ytd3JhcHBlciBjZi1oZWFkZXIgY2YtZXJyb3Itb3ZlcnZpZXciPgogICAgICAgIDxoMT4KICAgICAgICAgIDxzcGFuIGNsYXNzPSJjZi1lcnJvci10eXBlIiBkYXRhLXRyYW5zbGF0ZT0iZXJyb3IiPkVycm9yPC9zcGFuPgogICAgICAgICAgPHNwYW4gY2xhc3M9ImNmLWVycm9yLWNvZGUiPjEwMDM8L3NwYW4+CiAgICAgICAgICA8c21hbGwgY2xhc3M9ImhlYWRpbmctcmF5LWlkIj5SYXkgSUQ6IDE5NjhjMTRlYTgwOTBjM2I8L3NtYWxsPgogICAgICAgIDwvaDE+CiAgICAgICAgPGgyIGNsYXNzPSJjZi1zdWJoZWFkbGluZSIgZGF0YS10cmFuc2xhdGU9ImVycm9yX2Rlc2MiPkRpcmVjdCBJUCBhY2Nlc3Mgbm90IGFsbG93ZWQ8L2gyPgogICAgICA8L2Rpdj48IS0tIC8uaGVhZGVyIC0tPgoKICAgICAgPHNlY3Rpb24+PC9zZWN0aW9uPjwhLS0gc3BhY2VyIC0tPgoKICAgICAgPGRpdiBjbGFzcz0iY2Ytc2VjdGlvbiBjZi13cmFwcGVyIj4KICAgICAgICA8ZGl2IGNsYXNzPSJjZi1jb2x1bW5zIHR3byI+CiAgICAgICAgICA8ZGl2IGNsYXNzPSJjZi1jb2x1bW4iPgogICAgICAgICAgICA8aDIgZGF0YS10cmFuc2xhdGU9IndoYXRfaGFwcGVuZWQiPldoYXQgaGFwcGVuZWQ/PC9oMj4KICAgICAgICAgICAgPHA+WW91J3ZlIHJlcXVlc3RlZCBhbiBJUCBhZGRyZXNzIHRoYXQgaXMgcGFydCBvZiB0aGUgPGEgZGF0YS1vcmlnLXByb3RvPSJodHRwcyIgZGF0YS1vcmlnLXJlZj0id3d3LmNsb3VkZmxhcmUuY29tIiB0YXJnZXQ9Il9ibGFuayI+Q2xvdWRGbGFyZTwvYT4gbmV0d29yay48L3A+CiAgICAgICAgICA8L2Rpdj4KCiAgICAgICAgICAKICAgICAgICAgIDxkaXYgY2xhc3M9ImNmLWNvbHVtbiI+CiAgICAgICAgICAgIDxoMiBkYXRhLXRyYW5zbGF0ZT0id2hhdF9jYW5faV9kbyI+V2hhdCBjYW4gSSBkbz88L2gyPgogICAgICAgICAgICA8cD5JZiB5b3UgYXJlIGludGVyZXN0ZWQgaW4gbGVhcm5pbmcgbW9yZSBhYm91dCBDbG91ZEZsYXJlLCBwbGVhc2UgPGEgZGF0YS1vcmlnLXByb3RvPSJodHRwcyIgZGF0YS1vcmlnLXJlZj0id3d3LmNsb3VkZmxhcmUuY29tIiB0YXJnZXQ9Il9ibGFuayI+dmlzaXQgb3VyIHdlYnNpdGU8L2E+LjwvcD4KICAgICAgICAgIDwvZGl2PgogICAgICAgICAgCiAgICAgICAgPC9kaXY+CiAgICAgIDwvZGl2PjwhLS0gLy5zZWN0aW9uIC0tPgoKICAgICAgPGRpdiBjbGFzcz0iY2YtZXJyb3ItZm9vdGVyIGNmLXdyYXBwZXIiPgogIDxwPgogICAgPHNwYW4gY2xhc3M9ImNmLWZvb3Rlci1pdGVtIj5DbG91ZEZsYXJlIFJheSBJRDogPHN0cm9uZz4xOTY4YzE0ZWE4MDkwYzNiPC9zdHJvbmc+PC9zcGFuPgogICAgPHNwYW4gY2xhc3M9ImNmLWZvb3Rlci1zZXBhcmF0b3IiPiZidWxsOzwvc3Bhbj4KICAgIDxzcGFuIGNsYXNzPSJjZi1mb290ZXItaXRlbSI+PHNwYW4gZGF0YS10cmFuc2xhdGU9InlvdXJfaXAiPllvdXIgSVA8L3NwYW4+OiA1NC42OC45MS43NDwvc3Bhbj4KICAgIDxzcGFuIGNsYXNzPSJjZi1mb290ZXItc2VwYXJhdG9yIj4mYnVsbDs8L3NwYW4+CiAgICA8c3BhbiBjbGFzcz0iY2YtZm9vdGVyLWl0ZW0iPjxzcGFuIGRhdGEtdHJhbnNsYXRlPSJwZXJmb3JtYW5jZV9zZWN1cml0eV9ieSI+UGVyZm9ybWFuY2UgJmFtcDsgc2VjdXJpdHkgYnk8L3NwYW4+IDxhIGRhdGEtb3JpZy1wcm90bz0iaHR0cHMiIGRhdGEtb3JpZy1yZWY9Ind3dy5jbG91ZGZsYXJlLmNvbS81eHgtZXJyb3ItbGFuZGluZyIgaWQ9ImJyYW5kX2xpbmsiIHRhcmdldD0iX2JsYW5rIj5DbG91ZEZsYXJlPC9hPjwvc3Bhbj4KICAgIAogIDwvcD4KPC9kaXY+PCEtLSAvLmVycm9yLWZvb3RlciAtLT4KCgogICAgPC9kaXY+PCEtLSAvI2NmLWVycm9yLWRldGFpbHMgLS0+CiAgPC9kaXY+PCEtLSAvI2NmLXdyYXBwZXIgLS0+CgogIDxzY3JpcHQgdHlwZT0idGV4dC9qYXZhc2NyaXB0Ij4KICB3aW5kb3cuX2NmX3RyYW5zbGF0aW9uID0ge307CiAgCiAgCjwvc2NyaXB0PgoKPC9ib2R5Pgo8L2h0bWw+Cg0KMA0KDQo=", "port": 80, "ip": "141.101.123.250"}
{"vhost": "23.193.132.247", "host": "23.193.132.247", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTQNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2O2MwMmMwZTZiJiM0NjsxNDE4MjA3MzAwJiM0NjsxY2JmMjMwZQo8L0JPRFk+PC9IVE1MPgo=", "port": 80, "ip": "23.193.132.247"}
{"vhost": "205.164.8.208", "host": "205.164.8.208", "data": "SFRUUC8xLjEgNDA0IE5vdCBGb3VuZA0KQ29udGVudC10eXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAwDQpDb25uZWN0aW9uOiBjbG9zZQ0KRGF0ZTogV2VkLCAxMCBEZWMgMjAxNCAxMDoyODoyMCBHTVQNClNlcnZlcjogVU5PDQoNCg==", "port": 80, "ip": "205.164.8.208"}
{"vhost": "23.5.208.123", "host": "23.5.208.123", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTMNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2OzIyZGNmMTgwJiM0NjsxNDE4MjA3MzAwJiM0NjszNGM0YzYwCjwvQk9EWT48L0hUTUw+Cg==", "port": 80, "ip": "23.5.208.123"}
{"vhost": "172.252.77.210", "host": "172.252.77.210", "data": "SFRUUC8xLjEgNDAzIEZvcmJpZGRlbg0KU2VydmVyOiBuZ2lueA0KRGF0ZTogV2VkLCAxMCBEZWMgMjAxNCAxMDoyNjowMCBHTVQNCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sDQpDb250ZW50LUxlbmd0aDogNTY0DQpDb25uZWN0aW9uOiBjbG9zZQ0KDQo8aHRtbD4NCjxoZWFkPjx0aXRsZT40MDMgRm9yYmlkZGVuPC90aXRsZT48L2hlYWQ+DQo8Ym9keSBiZ2NvbG9yPSJ3aGl0ZSI+DQo8Y2VudGVyPjxoMT40MDMgRm9yYmlkZGVuPC9oMT48L2NlbnRlcj4NCjxocj48Y2VudGVyPm5naW54PC9jZW50ZXI+DQo8L2JvZHk+DQo8L2h0bWw+DQo8IS0tIGEgcGFkZGluZyB0byBkaXNhYmxlIE1TSUUgYW5kIENocm9tZSBmcmllbmRseSBlcnJvciBwYWdlIC0tPg0KPCEtLSBhIHBhZGRpbmcgdG8gZGlzYWJsZSBNU0lFIGFuZCBDaHJvbWUgZnJpZW5kbHkgZXJyb3IgcGFnZSAtLT4NCjwhLS0gYSBwYWRkaW5nIHRvIGRpc2FibGUgTVNJRSBhbmQgQ2hyb21lIGZyaWVuZGx5IGVycm9yIHBhZ2UgLS0+DQo8IS0tIGEgcGFkZGluZyB0byBkaXNhYmxlIE1TSUUgYW5kIENocm9tZSBmcmllbmRseSBlcnJvciBwYWdlIC0tPg0KPCEtLSBhIHBhZGRpbmcgdG8gZGlzYWJsZSBNU0lFIGFuZCBDaHJvbWUgZnJpZW5kbHkgZXJyb3IgcGFnZSAtLT4NCjwhLS0gYSBwYWRkaW5nIHRvIGRpc2FibGUgTVNJRSBhbmQgQ2hyb21lIGZyaWVuZGx5IGVycm9yIHBhZ2UgLS0+DQo=", "port": 80, "ip": "172.252.77.210"}
{"vhost": "198.15.186.4", "host": "198.15.186.4", "data": "SFRUUC8xLjEgNDA0IE5vdCBGb3VuZA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9dXMtYXNjaWkNClNlcnZlcjogTWljcm9zb2Z0LUhUVFBBUEkvMi4wDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDAyOjI0OjI0IEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCkNvbnRlbnQtTGVuZ3RoOiAzMTUNCg0KPCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMDEvL0VOIiJodHRwOi8vd3d3LnczLm9yZy9UUi9odG1sNC9zdHJpY3QuZHRkIj4NCjxIVE1MPjxIRUFEPjxUSVRMRT5Ob3QgRm91bmQ8L1RJVExFPg0KPE1FVEEgSFRUUC1FUVVJVj0iQ29udGVudC1UeXBlIiBDb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXMtYXNjaWkiPjwvSEVBRD4NCjxCT0RZPjxoMj5Ob3QgRm91bmQ8L2gyPg0KPGhyPjxwPkhUVFAgRXJyb3IgNDA0LiBUaGUgcmVxdWVzdGVkIHJlc291cmNlIGlzIG5vdCBmb3VuZC48L3A+DQo8L0JPRFk+PC9IVE1MPg0K", "port": 80, "ip": "198.15.186.4"}
{"vhost": "104.148.103.201", "host": "104.148.103.201", "data": "SFRUUC8xLjEgNDAwIEJhZCBSZXF1ZXN0DQpDb250ZW50LVR5cGU6IHRleHQvaHRtbA0KRGF0ZTogV2VkLCAxMCBEZWMgMjAxNCAxMDoyODoyMSBHTVQNCkNvbm5lY3Rpb246IGNsb3NlDQpDb250ZW50LUxlbmd0aDogMzkNCg0KPGgxPkJhZCBSZXF1ZXN0IChJbnZhbGlkIEhvc3RuYW1lKTwvaDE+", "port": 80, "ip": "104.148.103.201"}
{"vhost": "23.73.210.133", "host": "23.73.210.133", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTMNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2OzMxNWRmNmQ4JiM0NjsxNDE4MjA3MzAwJiM0Njs2YWY5NzE5CjwvQk9EWT48L0hUTUw+Cg==", "port": 80, "ip": "23.73.210.133"}
{"vhost": "23.6.1.2", "host": "23.6.1.2", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTMNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2O2M1ODgwMmNjJiM0NjsxNDE4MjA3MzAwJiM0NjsxMmM5ZGEwCjwvQk9EWT48L0hUTUw+Cg==", "port": 80, "ip": "23.6.1.2"}
A sample request is shown below. Each line is terminated by a CRLF and an extra CRLF is appended to the request.
GET / HTTP/1.1
Host: ${IP_ADDRESS}
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2)
Accept-Encoding: none
Connection: close
The DAP utility can be used to process the JSON data as a stream:
$ curl -s https://scans.io/data/rapid7/sonar.http/20141209-http.gz | \
zcat | head -n 1 | \
bin/dap json + transform data=base64decode + decode_http_reply data + json
{
"data.http_title": "Invalid URL",
"data.http_body": "<HTML><HEAD>\n<TITLE>Invalid URL</TITLE>\n</HEAD><BODY>\n<H1>Invalid URL</H1>\nThe requested URL \"/\", is invalid.<p>\nReference #9.bd01fea5.1418207300.175b82b4\n</BODY></HTML>\n",
"data.http_date": "20141210T04:28:20",
"data.http_server": "AkamaiGHost",
"vhost": "23.41.188.80",
"host": "23.41.188.80",
"data": "HTTP/1.0 400 Bad Request\r\nServer: AkamaiGHost\r\nMime-Version: 1.0\r\nContent-Type: text/html\r\nContent-Length: 194\r\nExpires: Wed, 10 Dec 2014 10:28:20 GMT\r\nDate: Wed, 10 Dec 2014 10:28:20 GMT\r\nConnection: close\r\n\r\n<HTML><HEAD>\n<TITLE>Invalid URL</TITLE>\n</HEAD><BODY>\n<H1>Invalid URL</H1>\nThe requested URL \"/\", is invalid.<p>\nReference #9.bd01fea5.1418207300.175b82b4\n</BODY></HTML>\n",
"port": 80,
"ip": "23.41.188.80",
"data.http_code": 400,
"data.http_message": "Bad Request",
"data.http_raw_headers": {
"connection": "close",
"expires": "Wed, 10 Dec 2014 10:28:20 GMT",
"content-type": "text/html",
"mime-version": "1.0"
}
}