Skip to content

Commit 0b61842

Browse files
author
Sebastian Wagner
committed
Merge branch 'maintenance' into develop
2 parents e6d4ea3 + c2d19d3 commit 0b61842

File tree

11 files changed

+804
-19
lines changed

11 files changed

+804
-19
lines changed

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,7 @@ Update allowed classification fields to 2020-01-28 version (#1409, #1476). Old n
137137
- Add support for new field `SourceIpInfo.SourceIpv4Int` (PR#1940 by Sebastian Wagner).
138138
- Fix mapping of "ConnectionType" fields, this is not `protocol.application`. Now mapped to `extra.*.connection_type` (PR#1940 by Sebastian Wagner).
139139
- `intelmq.bots.parsers.shadowserver._config`:
140-
- Add support for the new feeds *Honeypot-Amplification-DDoS-Events* and *Honeypot-Brute-Force-Events* (PR#1950 and PR#1952 by Birger Schacht and Sebastian Wagner).
140+
- Add support for the new feeds *Honeypot-Amplification-DDoS-Events*, *Honeypot-Brute-Force-Events*, *Honeypot-Darknet*, *IP-Spoofer-Events*, *Sinkhole-Events*, *Sinkhole-HTTP-Events* (PR#1950, PR#1952, PR#1953 and PR#1954 by Birger Schacht and Sebastian Wagner).
141141

142142
#### Experts
143143

‎docs/user/bots.rst‎

Lines changed: 16 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1603,17 +1603,14 @@ These are the supported feed name and their corresponding file name for automati
16031603
Accessible-VNC `scan_vnc`
16041604
Blacklisted-IP (deprecated) `blacklist`
16051605
Blocklist `blocklist`
1606-
CAIDA-IP-Spoofer `caida_ip_spoofer`
16071606
Compromised-Website `compromised_website`
1608-
Darknet `darknet`
16091607
DNS-Open-Resolvers `scan_dns`
1610-
Drone `botnet_drone`
16111608
Honeypot-Amplification-DDoS-Events `event4_honeypot_ddos_amp`
16121609
Honeypot-Brute-Force-Events `event4_honeypot_brute_force`
1610+
Honeypot-Darknet `event4_honeypot_darknet`
16131611
HTTP-Scanners `hp_http_scan`
16141612
ICS-Scanners `hp_ics_scan`
1615-
IPv6-Sinkhole-HTTP-Drone `sinkhole6_http`
1616-
Microsoft-Sinkhole `microsoft_sinkhole`
1613+
IP-Spoofer-Events `event4_ip_spoofer`
16171614
NTP-Monitor `scan_ntpmonitor`
16181615
NTP-Version `scan_ntp`
16191616
Open-Chargen `scan_chargen`
@@ -1642,7 +1639,8 @@ These are the supported feed name and their corresponding file name for automati
16421639
Outdated-DNSSEC-Key-IPv6 `outdated_dnssec_key_v6`
16431640
Sandbox-URL `cwsandbox_url`
16441641
Sinkhole-DNS `sinkhole_dns`
1645-
Sinkhole-HTTP-Drone `sinkhole_http_drone`
1642+
Sinkhole-Events `event4_sinkhole`
1643+
Sinkhole-HTTP-Events `event46_sinkhole_http`
16461644
Spam-URL `spam_url`
16471645
SSL-FREAK-Vulnerable-Servers `scan_ssl_freak`
16481646
SSL-POODLE-Vulnerable-Servers `scan_ssl_poodle`
@@ -1652,12 +1650,18 @@ These are the supported feed name and their corresponding file name for automati
16521650

16531651
In addition, the following legacy reports are supported:
16541652

1655-
=========================== ==================================== ========================
1656-
feed name successor feed name file name
1657-
=========================== ==================================== ========================
1658-
Amplification-DDoS-Victim Honeypot-Amplification-DDoS-Events ``ddos_amplification``
1659-
Drone-Brute-Force Honeypot-Brute-Force-Events ``drone_brute_force``
1660-
=========================== ==================================== ========================
1653+
=========================== =================================================== ========================
1654+
feed name successor feed name file name
1655+
=========================== =================================================== ========================
1656+
Amplification-DDoS-Victim Honeypot-Amplification-DDoS-Events ``ddos_amplification``
1657+
CAIDA-IP-Spoofer IP-Spoofer-Events ``caida_ip_spoofer``
1658+
Darknet Honeypot-Darknet ``darknet``
1659+
Drone Sinkhole-Events ``botnet_drone``
1660+
Drone-Brute-Force Honeypot-Brute-Force-Events, Sinkhole-HTTP-Events ``drone_brute_force``
1661+
Microsoft-Sinkhole Sinkhole-HTTP-Events ``microsoft_sinkhole``
1662+
Sinkhole-HTTP-Drone Sinkhole-HTTP-Events ``sinkhole_http_drone``
1663+
IPv6-Sinkhole-HTTP-Drone Sinkhole-HTTP-Events ``sinkhole6_http``
1664+
=========================== =================================================== ========================
16611665

16621666
More information on these legacy reports can be found in `Changes in Sinkhole and Honeypot Report Types and Formats <https://www.shadowserver.org/news/changes-in-sinkhole-and-honeypot-report-types-and-formats/>`_.
16631667

‎intelmq/bots/parsers/shadowserver/_config.py‎

Lines changed: 187 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -391,6 +391,7 @@ def convert_date_utc(value):
391391
}
392392

393393
# https://www.shadowserver.org/wiki/pmwiki.php/Services/Sinkhole-HTTP-Drone
394+
# legacy (replaced by event46_sinkhole_http)
394395
sinkhole_http_drone = {
395396
'required_fields': [
396397
('time.source', 'timestamp', add_UTC_to_timestamp),
@@ -437,6 +438,7 @@ def convert_date_utc(value):
437438
}
438439

439440
# https://www.shadowserver.org/wiki/pmwiki.php/Services/Sinkhole6-HTTP-Drone
441+
# legacy (replaced by event46_sinkhole_http)
440442
ipv6_sinkhole_http_drone = {
441443
'required_fields': [
442444
('time.source', 'timestamp', add_UTC_to_timestamp),
@@ -478,6 +480,7 @@ def convert_date_utc(value):
478480
}
479481

480482
# https://www.shadowserver.org/wiki/pmwiki.php/Services/Microsoft-Sinkhole
483+
# legacy (replaced by event46_sinkhole_http)
481484
microsoft_sinkhole = {
482485
'required_fields': [
483486
('time.source', 'timestamp', add_UTC_to_timestamp),
@@ -1131,6 +1134,7 @@ def convert_date_utc(value):
11311134
}
11321135

11331136
# https://www.shadowserver.org/what-we-do/network-reporting/drone-botnet-drone-report/
1137+
# legacy (replaced by event4_sinkhole, event4_honeypot_darknet and event46_sinkhole_http)
11341138
drone = {
11351139
'required_fields': [
11361140
('time.source', 'timestamp', add_UTC_to_timestamp),
@@ -1955,6 +1959,7 @@ def convert_date_utc(value):
19551959
}
19561960

19571961
# https://www.shadowserver.org/wiki/pmwiki.php/Services/Darknet
1962+
# legacy (replaced by event4_honeypot_darknet)
19581963
darknet = {
19591964
'required_fields': [
19601965
('time.source', 'timestamp', add_UTC_to_timestamp),
@@ -2357,6 +2362,7 @@ def convert_date_utc(value):
23572362

23582363
# https://www.shadowserver.org/what-we-do/network-reporting/caida-ip-spoofer-report/
23592364
# NOTE: The "type" field is included twice with the same values
2365+
# legacy (replaced by event4_ip_spoofer)
23602366
caida = {
23612367
'required_fields': [
23622368
('time.source', 'timestamp', add_UTC_to_timestamp),
@@ -2542,6 +2548,176 @@ def convert_date_utc(value):
25422548
}
25432549
}
25442550

2551+
# https://www.shadowserver.org/what-we-do/network-reporting/ip-spoofer-events-report/
2552+
event4_ip_spoofer = {
2553+
'required_fields': [
2554+
('time.source', 'timestamp', add_UTC_to_timestamp),
2555+
('source.ip', 'src_ip'),
2556+
('source.port', 'src_port'),
2557+
],
2558+
'optional_fields': [
2559+
('protocol.transport', 'protocol'),
2560+
('source.asn', 'src_asn', invalidate_zero),
2561+
('source.geolocation.cc', 'src_geo'),
2562+
('source.geolocation.region', 'src_region'),
2563+
('source.geolocation.city', 'src_city'),
2564+
('source.reverse_dns', 'src_hostname'),
2565+
('extra.source.naics', 'src_naics', convert_int),
2566+
('extra.source.sector', 'src_sector', validate_to_none),
2567+
('extra.', 'device_vendor', validate_to_none),
2568+
('extra.', 'device_type', validate_to_none),
2569+
('extra.', 'device_model', validate_to_none),
2570+
('extra.', 'public_source', validate_to_none),
2571+
('classification.identifier', 'infection'),
2572+
('extra.', 'family', validate_to_none),
2573+
('extra.', 'tag', validate_to_none),
2574+
('extra.', 'application', validate_to_none),
2575+
('extra.', 'version', validate_to_none),
2576+
('extra.', 'event_id', validate_to_none),
2577+
('source.network', 'network', validate_to_none),
2578+
('extra.', 'routedspoof', validate_to_none),
2579+
('extra.', 'session', convert_int),
2580+
('extra.', 'nat', convert_bool),
2581+
],
2582+
'constant_fields': {
2583+
# FIXME Check if the classification is correct
2584+
'classification.identifier': 'ip-spoofer',
2585+
'classification.taxonomy': 'fraud',
2586+
'classification.type': 'masquerade',
2587+
}
2588+
}
2589+
2590+
# https://www.shadowserver.org/what-we-do/network-reporting/honeypot-darknet-events-report/
2591+
event4_honeypot_darknet = {
2592+
'required_fields': [
2593+
('time.source', 'timestamp', add_UTC_to_timestamp),
2594+
('source.ip', 'src_ip'),
2595+
],
2596+
'optional_fields': [
2597+
('source.port', 'src_port'),
2598+
('source.asn', 'src_asn', invalidate_zero),
2599+
('source.geolocation.cc', 'src_geo'),
2600+
('source.geolocation.region', 'src_region'),
2601+
('source.geolocation.city', 'src_city'),
2602+
('source.reverse_dns', 'src_hostname'),
2603+
('extra.source.naics', 'src_naics', convert_int),
2604+
('extra.source.sector', 'src_sector', validate_to_none),
2605+
('extra.', 'device_vendor', validate_to_none),
2606+
('extra.', 'device_type', validate_to_none),
2607+
('extra.', 'device_model', validate_to_none),
2608+
('destination.ip', 'dst_ip', validate_ip),
2609+
('destination.port', 'dst_port', convert_int),
2610+
('destination.asn', 'dst_asn', invalidate_zero),
2611+
('destination.geolocation.cc', 'dst_geo'),
2612+
('destination.geolocation.region', 'dst_region'),
2613+
('destination.geolocation.city', 'dst_city'),
2614+
('destination.reverse_dns', 'dst_hostname'),
2615+
('extra.destination.naics', 'dst_naics', invalidate_zero),
2616+
('extra.destination.sector', 'dst_sector', validate_to_none),
2617+
('extra.', 'public_source', validate_to_none),
2618+
('malware.name', 'infection'),
2619+
('extra.', 'family', validate_to_none),
2620+
('classification.identifier', 'tag'), # different values possible in this report
2621+
('extra.', 'application', validate_to_none),
2622+
('extra.', 'version', validate_to_none),
2623+
('extra.', 'event_id', validate_to_none),
2624+
('extra.', 'count', convert_int),
2625+
],
2626+
'constant_fields': {
2627+
'classification.taxonomy': 'other',
2628+
'classification.type': 'other',
2629+
},
2630+
}
2631+
2632+
event4_sinkhole = {
2633+
'required_fields': [
2634+
('time.source', 'timestamp', add_UTC_to_timestamp),
2635+
('source.ip', 'src_ip'),
2636+
('source.port', 'src_port'),
2637+
],
2638+
'optional_fields': [
2639+
('protocol.transport', 'protocol'),
2640+
('source.asn', 'src_asn', invalidate_zero),
2641+
('source.geolocation.cc', 'src_geo'),
2642+
('source.geolocation.region', 'src_region'),
2643+
('source.geolocation.city', 'src_city'),
2644+
('source.reverse_dns', 'src_hostname'),
2645+
('extra.source.naics', 'src_naics', convert_int),
2646+
('extra.source.sector', 'src_sector', validate_to_none),
2647+
('extra.', 'device_vendor', validate_to_none),
2648+
('extra.', 'device_type', validate_to_none),
2649+
('extra.', 'device_model', validate_to_none),
2650+
('destination.ip', 'dst_ip', validate_ip),
2651+
('destination.port', 'dst_port'),
2652+
('destination.asn', 'dst_asn', invalidate_zero),
2653+
('destination.geolocation.cc', 'dst_geo'),
2654+
('destination.geolocation.region', 'dst_region'),
2655+
('destination.geolocation.city', 'dst_city'),
2656+
('destination.reverse_dns', 'dst_hostname'),
2657+
('extra.destination.naics', 'dst_naics', invalidate_zero),
2658+
('extra.destination.sector', 'dst_sector', validate_to_none),
2659+
('extra.', 'public_source', validate_to_none),
2660+
('malware.name', 'infection'),
2661+
('extra.', 'family', validate_to_none),
2662+
('classification.identifier', 'tag'),
2663+
('extra.', 'application', validate_to_none),
2664+
('extra.', 'version', validate_to_none),
2665+
('extra.', 'event_id', validate_to_none),
2666+
],
2667+
'constant_fields': {
2668+
'classification.taxonomy': 'malicious-code',
2669+
'classification.type': 'infected-system',
2670+
},
2671+
}
2672+
2673+
event46_sinkhole_http = {
2674+
'required_fields': [
2675+
('time.source', 'timestamp', add_UTC_to_timestamp),
2676+
('source.ip', 'src_ip'),
2677+
('source.port', 'src_port'),
2678+
],
2679+
'optional_fields': [
2680+
('protocol.transport', 'protocol'),
2681+
('source.asn', 'src_asn', invalidate_zero),
2682+
('source.geolocation.cc', 'src_geo'),
2683+
('source.geolocation.region', 'src_region'),
2684+
('source.geolocation.city', 'src_city'),
2685+
('source.reverse_dns', 'src_hostname'),
2686+
('extra.source.naics', 'src_naics', convert_int),
2687+
('extra.source.sector', 'src_sector', validate_to_none),
2688+
('extra.', 'device_vendor', validate_to_none),
2689+
('extra.', 'device_type', validate_to_none),
2690+
('extra.', 'device_model', validate_to_none),
2691+
('destination.ip', 'dst_ip', validate_ip),
2692+
('destination.port', 'dst_port'),
2693+
('destination.asn', 'dst_asn', invalidate_zero),
2694+
('destination.geolocation.cc', 'dst_geo'),
2695+
('destination.geolocation.region', 'dst_region'),
2696+
('destination.geolocation.city', 'dst_city'),
2697+
('destination.reverse_dns', 'dst_hostname'),
2698+
('extra.destination.naics', 'dst_naics', invalidate_zero),
2699+
('extra.destination.sector', 'dst_sector', validate_to_none),
2700+
('extra.', 'public_source', validate_to_none),
2701+
('malware.name', 'infection'),
2702+
('extra.', 'family', validate_to_none),
2703+
('classification.identifier', 'tag'),
2704+
('extra.', 'application', validate_to_none),
2705+
('extra.', 'version', validate_to_none),
2706+
('extra.', 'event_id', validate_to_none),
2707+
('destination.url', 'http_url', convert_http_host_and_url, True),
2708+
('destination.fqdn', 'http_host', validate_fqdn),
2709+
('extra.', 'http_agent', validate_to_none),
2710+
('extra.', 'forwarded_by', validate_to_none),
2711+
('extra.', 'ssl_cipher', validate_to_none),
2712+
('extra.', 'http_referer', validate_to_none),
2713+
],
2714+
'constant_fields': {
2715+
'classification.taxonomy': 'malicious-code',
2716+
'classification.type': 'infected-system',
2717+
'protocol.application': 'http',
2718+
},
2719+
}
2720+
25452721
mapping = (
25462722
# feed name, file name, function
25472723
('Accessible-ADB', 'scan_adb', accessible_adb),
@@ -2564,18 +2740,20 @@ def convert_date_utc(value):
25642740
('Amplification-DDoS-Victim', 'ddos_amplification', amplification_ddos_victim), # legacy (replaced by honeypot-ddos-amp)
25652741
('Blacklisted-IP', 'blacklist', blocklist),
25662742
('Blocklist', 'blocklist', blocklist),
2567-
('CAIDA-IP-Spoofer', 'caida_ip_spoofer', caida),
2743+
('CAIDA-IP-Spoofer', 'caida_ip_spoofer', caida), # legacy (replaced by event4_ip_spoofer)
25682744
('Compromised-Website', 'compromised_website', compromised_website),
25692745
('DNS-Open-Resolvers', 'scan_dns', dns_open_resolvers),
2570-
('Darknet', 'darknet', darknet),
2571-
('Drone', 'botnet_drone', drone),
2746+
('Darknet', 'darknet', darknet), # legacy (replaced by event4_honeypot_darknet)
2747+
('Drone', 'botnet_drone', drone), # legacy (replaced by event4_sinkhole, event4_honeypot_darknet and event46_sinkhole_http)
25722748
('Drone-Brute-Force', 'drone_brute_force', drone_brute_force), # legacy (replaced by honeypot_brute_force)
25732749
('HTTP-Scanners', 'hp_http_scan', http_scanners),
25742750
('Honeypot-Amplification-DDoS-Events', 'event4_honeypot_ddos_amp', honeypot_ddos_amp),
25752751
('Honeypot-Brute-Force-Events', 'event4_honeypot_brute_force', honeypot_brute_force),
2752+
('Honeypot-Darknet', 'event4_honeypot_darknet', event4_honeypot_darknet),
25762753
('ICS-Scanners', 'hp_ics_scan', ics_scanners),
2577-
('IPv6-Sinkhole-HTTP-Drone', 'sinkhole6_http', ipv6_sinkhole_http_drone),
2578-
('Microsoft-Sinkhole', 'microsoft_sinkhole', microsoft_sinkhole),
2754+
('IPv6-Sinkhole-HTTP-Drone', 'sinkhole6_http', ipv6_sinkhole_http_drone), # legacy (replaced by event46_sinkhole_http)
2755+
('IP-Spoofer-Events', 'event4_ip_spoofer', event4_ip_spoofer),
2756+
('Microsoft-Sinkhole', 'microsoft_sinkhole', microsoft_sinkhole), # legacy (replaced by event46_sinkhole_http)
25792757
('NTP-Monitor', 'scan_ntpmonitor', ntp_monitor),
25802758
('NTP-Version', 'scan_ntp', ntp_version),
25812759
('Open-Chargen', 'scan_chargen', open_chargen),
@@ -2606,7 +2784,10 @@ def convert_date_utc(value):
26062784
('SSL-POODLE-Vulnerable-Servers', 'scan_ssl_poodle', ssl_poodle_vulnerable_servers),
26072785
('Sandbox-URL', 'cwsandbox_url', sandbox_url),
26082786
('Sinkhole-DNS', 'sinkhole_dns', sinkhole_dns),
2609-
('Sinkhole-HTTP-Drone', 'sinkhole_http_drone', sinkhole_http_drone),
2787+
('Sinkhole-Events', 'event4_sinkhole', event4_sinkhole),
2788+
('Sinkhole-Events-HTTP IPv4', 'event4_sinkhole_http', event46_sinkhole_http),
2789+
('Sinkhole-Events-HTTP IPv6', 'event6_sinkhole_http', event46_sinkhole_http),
2790+
('Sinkhole-HTTP-Drone', 'sinkhole_http_drone', sinkhole_http_drone), # legacy (replaced by event46_sinkhole_http)
26102791
('Spam-URL', 'spam_url', spam_url),
26112792
('Vulnerable-ISAKMP', 'scan_isakmp', vulnerable_isakmp),
26122793
('Vulnerable-HTTP', 'scan_http', accessible_vulnerable_http),

0 commit comments

Comments
 (0)