suricata-sample-data

This is a project and repository of different outputs of Suricata run against different pcap data sets. You can download the Suricata data sets from the releases page of this repository.

eve.json files

Select samples of alerts from the zip

Samples README and starting folder of the files generated from the eve files
samples

The mission of the Collegiate Cyber Defense Competition (CCDC) system 2018

samples/wrccdc-2018 are generated from http://www.wrccdc.org/ mirrored here

A sampling of 1 of each alert from its eve.json
samples/wrccdc-2018/alerts-only.json

A list of id's, signature names, and hyperlinks from the rules references section
samples/wrccdc-2018/references.md

A list of id's, sampling of 1 of each alert from its eve.json
samples/wrccdc-2018/alerts-only.json

A unique list of sids (Suricata Id's)
samples/wrccdc-2018/ids-list.txt

A list of signatures that map 1-1 with the ids-list
samples/wrccdc-2018/signature-list.txt

The mission of the Collegiate Cyber Defense Competition (CCDC) system 2017

samples/wrccdc-2017 are generated from http://www.wrccdc.org/ mirrored here

A sampling of 1 of each alert from its eve.json
samples/wrccdc-2017/alerts-only.json

A list of id's, signature names, and hyperlinks from the rules references section
samples/wrccdc-2017/references.md

A unique list of sids (Suricata Id's)
samples/wrccdc-2017/ids-list.txt

A list of signatures that map 1-1 with the ids-list
samples/wrccdc-2017/signature-list.txt

Hands-on Network Forensics - Training PCAP dataset from FIRST 2015

samples/first-org-conf-2015 are generated from the pcaps mirrored here from the first.org conference

A sampling of 1 of each alert from its eve.json
samples/first-org-conf-2015/alerts-only.json

A list of id's, signature names, and hyperlinks from the rules references section
samples/first-org-conf-2015/references.md

A unique list of sids (Suricata Id's)
samples/first-org-conf-2015/ids-list.txt

A list of signatures that map 1-1 with the ids-list
samples/first-org-conf-2015/signature-list.txt

Honey Pot with Modern Honeypot Network

samples/honeypot-2018 was generated from using modern honey pot on digital ocean
https://threatstream.github.io/mhn/

A sampling of 1 of each alert from its eve.json
samples/honeypot-2018/alerts-only.json

A list of id's, signature names, and hyperlinks from the rules references section
samples/honeypot-2018/references.md

A unique list of sids (Suricata Id's)
samples/honeypot-2018/ids-list.txt

A list of signatures that map 1-1 with the ids-list
samples/honeypot-2018/signature-list.txt

Optionally you can read below on how to (re)create your own data sets from your own pcap files and Suricata rules.

Background

Suricata is an engine that is capable of real time intrusion detection IDS, inline intrusion prevention IPS, network security monitoring (NSM) and offline pcap processing.

This repository contains sample eve.json log files created by Suricata from existing pcap files as well as instructions on how to create them yourself. This is useful if you want to see what alerts only data sets look like or play with the rules yourself and re-create your own eve files for learning purposes to write your own eve.json real time simulator.

The pcaps I found interesting for rich data was the The Western Regional Cyber Defense 2018 (wrccdc). wrccdc has a rich set of different pcap files in their archives from various competitions. I used their 2018 pcap data set to create my even.json files for personal use.

wrccdc 2018's topology from their competition is a close to real world scenario.

I also used the Hands-on Network Forensics - Training PCAP dataset 2015 from this mirror and followed along with their PDF guide

How to download all the PCAPS from the 2018 competition

Use wget

wget -r -np -k https://archive.wrccdc.org/pcaps/2018/

This will download over a long period of time all the files to the sub-folder

archive.wrccdc.org/pcaps/2018

unzip those using gunzip

cd archive.wrccdc.org/pcaps/2018
gunzip *.gz

How to make an alerts only configuration

Open your suricata.yaml

vim /usr/local/etc/suricata/suricata.yaml

And remove the sections of http, dns, tls, files, ssh, stats, and flow events. Also set your stats to enabled: false. See conf/suricata.yaml for my example.

How to write a script to parse each file

See scripts/ingest_pcap.sh for a simple for loop which will run suricata and append to your eve.json file for each pcap file in a particular folder.

SID allocations

For the signature lists see this page for the allocation of signature ids https://doc.emergingthreats.net/bin/view/Main/SidAllocation

1000000-1999999 Reserved for Local Use -- Put your custom rules in this range to avoid conflicts

The following are the reservations for SIDs in the 2000000 space allocated to emerging threats:

2000000-2099999 Emerging Threats Open Rulesets

2100000-2103999 Forked ET Versions of the Original Snort GPL Signatures Originally sids 3464 and prior, forked to be maintained and converted to Suricata

2200000-2200999 Suricata Decoder Events

2210000-2210999 Suricata Stream Events

2220000-2299999 Suricata Reserved

2800000-2899999 Emerging Threats Pro Full Coverage Ruleset -- ETProRules

Dynamicly Updated Rules

2400000-2400999 SpamHaus DROP List — Updated Daily -- SpamHausDROPList

2402000-2402299 Dshield Top Attackers Rules — Updated Daily -- DshieldTopAttackers

2403300-2403499 CIArmy.com Top Attackers Rules — Updated Daily - See http://www.ciarmy.com#list -- CiArmy

2404000-2405999 Shadowserver.org Bot C&C List — Updated Daily -- BotCC

2404000-2405999 Shadowserver.org Bot C&C List Grouped by Port — Updated Daily -- BotCC

2406000-2406999 Russian Business Network Known Nets --- OBSOLETED -- RussianBusinessNetwork

2408000-2408499 Russian Business Network Known Malvertisers --- OBSOLETED -- RussianBusinessNetwork

2520000-2521999 Tor Exit Nodes List Updated Daily -- TorRules

2522000-2525999 Tor Relay Nodes List (NOT Exit nodes) Updated Daily -- TorRules

Command line jq tips and tricks with a eve.json

Install jq and go to a working directory that contains an eve file.

cd /usr/local/var/log/suricata

# or from the releases zip you can use any of the eve.json
cd ./release/wrcddc-2018
cd ./release/first-org-conf-2015

To get all signatures from a eve.json

jq '.alert.signature' eve.json

This will return a list

"ET WEB_SERVER allow_url_include PHP config option in uri"
"ET WEB_SERVER safe_mode PHP config option in uri"
"ET WEB_SERVER suhosin.simulation PHP config option in uri"
"ET WEB_SERVER disable_functions PHP config option in uri"
"ET WEB_SERVER open_basedir PHP config option in uri"

To get a single sample signature id from a large eve.json using a sid (e.x. 2012647)

jq 'select(.alert.signature_id==2012647)' eve.json | jq -s '.[0]'

To get a list of all uniq and sorted signature id's

jq 'select(.alert.signature_id)|.alert.signature_id' eve.json | sort | uniq

It will return sids sorted asc by number:

2001219
2001595
2001743
2002157

You can add that to an array like in a script

EVE_FILE=eve.json
SIGNATURES=(`jq 'select(.alert.signature_id)|.alert.signature_id' ${EVE_FILE} | sort | uniq`)

You can loop over that array to print a sample of each signature

EVE_FILE=eve.json
file_list=()
for SIGNATURE_ID in "${SIGNATURES[@]}"
do
  sample=`jq "select(.alert.signature_id==$SIGNATURE_ID)" ${EVE_FILE} | jq -s '.[0]'`
  file_list=("${file_list[@]}" "$sample")
done

echo "${file_list[@]}" | jq -s '.'

To sort a eve.json object of alerts by timestamp in ascending order:

jq -s 'sort_by(.timestamp)' eve.json

To get an ad-hoc timeline of signature strings from an eve.json file of all alerts:

jq -s 'sort_by(.timestamp)|.[].alert.signature' eve.json

To get an array of alerts in one of the sample sub-folders sorted by timestamp in ascending order:

cd ${ROOT_OF_THIS_PROJECT}
jq 'sort_by(.timestamp)' samples/first-org-conf-2015/alerts-only.json

To get an ad-hoc timeline of signature strings from one of the samples by timestamp:

cd ${ROOT_OF_THIS_PROJECT}
jq 'sort_by(.timestamp)|.[].alert.signature' samples/first-org-conf-2015/alerts-only.json