ParserBot: Fix line recovery and message dumping #2192
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ParserBot.process first iterates over all message in a parsed report and catches the errors, saves the erroneous message and at the end dumps all of them in one row. The recover_line methods were not consistent in their use of the line given as parameter and the use of the _current_line class member. This lead to parsers always dumping the last message (_current_line which was active at the time of the dumping) leading to data loss for dumped messages for `recover_line_csv_dict` and `recover_line_json_stream`.
mainly for ParserBot's recover_line methods and related
Use RewindableFileHandle to use the orignal current line for line recovery
use the unparsed current line as string instead of the parsed line
reports must not be dropped silently if their format is unknown (can also happen because of misconfiguration)
sebix
added
bug
wagner-intevation
force-pushed
the
fix-bot-dump
branch
2 times, most recently
from
aeba819 to
62ce61c
Compare
|
fyi @elsif2 there's also a small fix in the shadowserver parser |
|
Fix to intelmq/bots/parsers/shadowserver/parser.py added to #2143. |
waldbauer-certat
approved these changes
Jul 12, 2022
elsif2
added a commit
that referenced
this pull request
Aug 1, 2022
Update to existing test cases to match current report types. New tests for added report types. pycodestyle fixes add testdata licenses pycodestyle fix Added reports parameter Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Update scan_snmp.csv Realign columns Remove duplicates Changed malware.name to extra.infection Updated SPDX-FileCopyrightText shadowserver api: document and warn on old parameter document the old parameter `country` and its status warn if used adapt the test DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright and raw field updates Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks. Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test updated schema Updated scan_smb tests Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Updated report schema for June 2022 Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Revert: Fix for recover_line method as commited in #2192 Added scan_couchdb Test case for scan_couchdb Added scan6_rpd Added/updated README with maintainer details Restored feed names and classification.identifiers to minimize upgrade impact. Merge repair pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Added additional IPv6 aliases Fix for recover_line method as commited in #2192
elsif2
added a commit
that referenced
this pull request
Aug 1, 2022
author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392927 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392915 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392880 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392820 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392766 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392468 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392457 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392039 +0000 Update parser to support all available reports. Update to existing test cases to match current report types. New tests for added report types. pycodestyle fixes add testdata licenses pycodestyle fix Added reports parameter Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Update scan_snmp.csv Realign columns Remove duplicates Changed malware.name to extra.infection Updated SPDX-FileCopyrightText shadowserver api: document and warn on old parameter document the old parameter `country` and its status warn if used adapt the test DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright and raw field updates Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks. Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test updated schema Updated scan_smb tests Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Updated report schema for June 2022 Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Revert: Fix for recover_line method as commited in #2192 Added scan_couchdb Test case for scan_couchdb Added scan6_rpd Added/updated README with maintainer details Restored feed names and classification.identifiers to minimize upgrade impact. Merge repair pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Added additional IPv6 aliases Fix for recover_line method as commited in #2192 Replaced misleading _country_ parameter with _reports_. Update parser to support all available reports. Update to existing test cases to match current report types. pycodestyle fixes add testdata licenses pycodestyle fix Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Realign columns Remove duplicates Changed malware.name to extra.infection DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Fix for recover_line method as commited in #2192 Revert: Fix for recover_line method as commited in #2192 Test case for scan_couchdb Restored feed names and classification.identifiers to minimize upgrade impact. Added function to notify if any unsupported feeds are configured. pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Update intelmq/bots/collectors/shadowserver/collector_reports_api.py Co-authored-by: Sebastian <sebix@sebix.at> Update intelmq/bots/collectors/shadowserver/README.md Co-authored-by: Sebastian <sebix@sebix.at> Update intelmq/tests/bots/parsers/shadowserver/README.md Co-authored-by: Sebastian <sebix@sebix.at> resync to develop
elsif2
added a commit
that referenced
this pull request
Aug 1, 2022
author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395284 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395281 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395278 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395264 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395260 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395256 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395141 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395131 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395127 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395122 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395058 +0000 Update parser to support all available reports. Update to existing test cases to match current report types. New tests for added report types. pycodestyle fixes add testdata licenses pycodestyle fix Added reports parameter Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Update scan_snmp.csv Realign columns Remove duplicates Changed malware.name to extra.infection Updated SPDX-FileCopyrightText shadowserver api: document and warn on old parameter document the old parameter `country` and its status warn if used adapt the test DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright and raw field updates Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks. Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test updated schema Updated scan_smb tests Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Updated report schema for June 2022 Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Revert: Fix for recover_line method as commited in #2192 Added scan_couchdb Test case for scan_couchdb Added scan6_rpd Added/updated README with maintainer details Restored feed names and classification.identifiers to minimize upgrade impact. Merge repair pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Added additional IPv6 aliases Fix for recover_line method as commited in #2192
elsif2
added a commit
that referenced
this pull request
Aug 2, 2022
author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659452139 +0000 parent ba7d24b author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659452135 +0000 parent ba7d24b author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659452132 +0000 parent ba7d24b author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659452116 +0000 parent ba7d24b author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659452111 +0000 parent ba7d24b author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659452106 +0000 parent ba7d24b author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659451910 +0000 parent ba7d24b author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659451905 +0000 parent ba7d24b author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659451900 +0000 parent ba7d24b author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659451775 +0000 Update parser to support all available reports. Update to existing test cases to match current report types. New tests for added report types. pycodestyle fixes add testdata licenses pycodestyle fix Added reports parameter Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Update scan_snmp.csv Realign columns Remove duplicates Changed malware.name to extra.infection Updated SPDX-FileCopyrightText shadowserver api: document and warn on old parameter document the old parameter `country` and its status warn if used adapt the test DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright and raw field updates Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks. Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test updated schema Updated scan_smb tests Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Updated report schema for June 2022 Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Revert: Fix for recover_line method as commited in #2192 Added scan_couchdb Test case for scan_couchdb Added scan6_rpd Added/updated README with maintainer details Restored feed names and classification.identifiers to minimize upgrade impact. Merge repair pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Added additional IPv6 aliases Fix for recover_line method as commited in #2192 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392927 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392915 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392880 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392820 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392766 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392468 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392457 +0000 parent 6194014 author elsif2 <elsif@shadowserver.org> 1659389509 +0000 committer elsif2 <elsif@shadowserver.org> 1659392039 +0000 Update parser to support all available reports. Update to existing test cases to match current report types. New tests for added report types. pycodestyle fixes add testdata licenses pycodestyle fix Added reports parameter Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Update scan_snmp.csv Realign columns Remove duplicates Changed malware.name to extra.infection Updated SPDX-FileCopyrightText shadowserver api: document and warn on old parameter document the old parameter `country` and its status warn if used adapt the test DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright and raw field updates Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks. Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test updated schema Updated scan_smb tests Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Updated report schema for June 2022 Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Revert: Fix for recover_line method as commited in #2192 Added scan_couchdb Test case for scan_couchdb Added scan6_rpd Added/updated README with maintainer details Restored feed names and classification.identifiers to minimize upgrade impact. Merge repair pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Added additional IPv6 aliases Fix for recover_line method as commited in #2192 Replaced misleading _country_ parameter with _reports_. Update parser to support all available reports. Update to existing test cases to match current report types. pycodestyle fixes add testdata licenses pycodestyle fix Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Realign columns Remove duplicates Changed malware.name to extra.infection DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Fix for recover_line method as commited in #2192 Revert: Fix for recover_line method as commited in #2192 Test case for scan_couchdb Restored feed names and classification.identifiers to minimize upgrade impact. Added function to notify if any unsupported feeds are configured. pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Update intelmq/bots/collectors/shadowserver/collector_reports_api.py Co-authored-by: Sebastian <sebix@sebix.at> Update intelmq/bots/collectors/shadowserver/README.md Co-authored-by: Sebastian <sebix@sebix.at> Update intelmq/tests/bots/parsers/shadowserver/README.md Co-authored-by: Sebastian <sebix@sebix.at> resync to develop parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395284 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395281 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395278 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395264 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395260 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395256 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395141 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395131 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395127 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395122 +0000 parent 7dc5b74 author elsif2 <elsif@shadowserver.org> 1643216571 +0000 committer elsif2 <elsif@shadowserver.org> 1659395058 +0000 Update parser to support all available reports. Update to existing test cases to match current report types. New tests for added report types. pycodestyle fixes add testdata licenses pycodestyle fix Added reports parameter Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Update scan_snmp.csv Realign columns Remove duplicates Changed malware.name to extra.infection Updated SPDX-FileCopyrightText shadowserver api: document and warn on old parameter document the old parameter `country` and its status warn if used adapt the test DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright and raw field updates Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks. Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test updated schema Updated scan_smb tests Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Updated report schema for June 2022 Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Revert: Fix for recover_line method as commited in #2192 Added scan_couchdb Test case for scan_couchdb Added scan6_rpd Added/updated README with maintainer details Restored feed names and classification.identifiers to minimize upgrade impact. Merge repair pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Added additional IPv6 aliases Fix for recover_line method as commited in #2192 resolve conflict Replaced misleading _country_ parameter with _reports_. Update parser to support all available reports. Update to existing test cases to match current report types. New tests for added report types. pycodestyle fixes add testdata licenses pycodestyle fix Added reports parameter Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Update scan_snmp.csv Realign columns Remove duplicates Changed malware.name to extra.infection Updated SPDX-FileCopyrightText shadowserver api: document and warn on old parameter document the old parameter `country` and its status warn if used adapt the test DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright and raw field updates Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks. Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test updated schema Updated scan_smb tests Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Updated report schema for June 2022 Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Revert: Fix for recover_line method as commited in #2192 Added scan_couchdb Test case for scan_couchdb Added scan6_rpd Added/updated README with maintainer details Restored feed names and classification.identifiers to minimize upgrade impact. Merge repair pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Added additional IPv6 aliases Fix for recover_line method as commited in #2192 Update to existing test cases to match current report types. add testdata licenses pycodestyle fix Proposed details for the release Test script updates for suggested changes Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Realign columns Remove duplicates DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file cleanup renamed license files updated scan_mdns test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test New scan_docker test New scan_kubernetes test New scan_mysql test Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Fix for recover_line method as commited in #2192 Revert: Fix for recover_line method as commited in #2192 Test case for scan_couchdb Restored feed names and classification.identifiers to minimize upgrade impact. Added function to notify if any unsupported feeds are configured. pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Update intelmq/bots/collectors/shadowserver/collector_reports_api.py Co-authored-by: Sebastian <sebix@sebix.at> Update intelmq/bots/collectors/shadowserver/README.md Co-authored-by: Sebastian <sebix@sebix.at> Update intelmq/tests/bots/parsers/shadowserver/README.md Co-authored-by: Sebastian <sebix@sebix.at> resync to develop
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ParserBot.process first iterates over all message in a parsed report and
catches the errors, saves the erroneous message and at the end dumps all
of them in one row.
The recover_line methods were not consistent in their use of the line
given as parameter and the use of the _current_line class member.
This lead to parsers always dumping the last message (_current_line
which was active at the time of the dumping) leading to data loss for
dumped messages for
recover_line_csv_dictandrecover_line_json_stream.mainly for ParserBot's recover_line methods and related
Use RewindableFileHandle to use the orignal current line for line recovery
use the unparsed current line as string instead of the parsed line
reports must not be dropped silently if their format is unknown (can
also happen because of misconfiguration)