Merge branch 'maintenance' into develop

Author: Sebastian Wagner

CHANGELOG.md

@@ -21,14 +21,18 @@ CHANGELOG
 #### Collectors
 - `intelmq.bots.collectors.shodan.collector_stream`: Fix access to parameters, the bot wrongly used `self.parameters` (PR#2020 by Mikk Margus Möll).
 - `intelmq.bots.collectors.mail.collector_mail_attach`: Add attachment file name as `extra.file_name` also if the attachment is not compressed (PR#2021 by Alex Kaplan).
+- `intelmq.bots.collectors.http.collector_http_stream`: Fix access to parameters, the bot wrongly used `self.parameters` (by Sebastian Wagner).
 
 #### Parsers
 
 #### Experts
 
 #### Outputs
+- `intelmq.bots.outputs.mcafee.output_esm_ip`: Fix access to parameters, the bot wrongly used `self.parameters` (by Sebastian Wagner).
+- `intelmq.bots.outputs.misp.output_api`: Fix access to parameters, the bot wrongly used `self.parameters` (by Sebastian Wagner).
 
 ### Documentation
+- Various formatting fixes (by Sebastian Wagner).
 
 ### Packaging
 - intelmq-update-database crontab: Add missing `recordedfuture_iprisk` update call (by Sebastian Wagner).

docs/dev/feeds-wishlist.rst

@@ -14,6 +14,7 @@ See :ref:`feeds documentation` for more information on this.
 This list evolved from the issue :issue:`Contribute: Feeds List (#384) <384>`.
 
 - Lists of feeds:
+
   - `threatfeeds.io <https://threatfeeds.io>`_
   - `TheCyberThreat <http://thecyberthreat.com/cyber-threat-intelligence-feeds/>`_
   - `sbilly: Awesome Security <https://github.com/sbilly/awesome-security#threat-intelligence>`_
@@ -27,16 +28,16 @@ This list evolved from the issue :issue:`Contribute: Feeds List (#384) <384>`.
 - List of potentially interesting data sources:
 
   - `Abuse.ch SSL Blacklists <https://sslbl.abuse.ch/blacklist/>`_
-  - `Adblock Plus Malwaredomains <https://easylist-msie.adblockplus.org/malwaredomains_full.tpl>`_
+  - `Adblock Plus <https://adblockplus.org/en/subscriptions>`_
   - `apivoid IP Reputation API <https://www.apivoid.com/api/ip-reputation/>`_
   - `Anomali Limo Free Intel Feed <https://www.anomali.com/resources/limo>`_
   - `APWG's ecrimex <https://www.ecrimex.net>`_
-  - `Bad IPs <https://www.badips.com>`_
   - `Berkeley <https://security.berkeley.edu/aggressive_ips/ips>`_
   - `Binary Defense <https://www.binarydefense.com/>`_
   - `Bot Invaders Realtime tracker <http://www.marc-blanchard.com/BotInvaders/index.php>`_
   - `Botherder Targetedthreats <https://github.com/botherder/targetedthreats/>`_
   - `Botscout Last Caught <http://botscout.com/last_caught_cache.htm>`_
+  - `botvrij <https://www.botvrij.eu/>`_
   - `Carbon Black Feeds <https://github.com/carbonblack/cbfeeds>`_
   - `CERT.pl Phishing Warning List <http://hole.cert.pl/domains/>`_
   - `Chaos Reigns <http://www.chaosreigns.com/spam/>`_
@@ -45,7 +46,6 @@ This list evolved from the issue :issue:`Contribute: Feeds List (#384) <384>`.
   - `Cyber Crime Tracker <http://cybercrime-tracker.net/all.php>`_
   - `drb-ra C2IntelFeeds <https://github.com/drb-ra/C2IntelFeeds>`_
   - `DNS DB API <https://api.dnsdb.info>`_
-  - `Dyn DNS <http://security-research.dyndns.org/pub/>`_
   - `ESET Malware Indicators of Compromise <https://github.com/eset/malware-ioc>`_
   - `Facebook Threat Exchange <https://developers.facebook.com/docs/threat-exchange>`_
   - `FilterLists <https://filterlists.com>`_
@@ -57,20 +57,18 @@ This list evolved from the issue :issue:`Contribute: Feeds List (#384) <384>`.
   - `HP Feeds <https://github.com/rep/hpfeeds>`_
   - `IBM X-Force Exchange <https://exchange.xforce.ibmcloud.com/>`_
   - `ImproWare AntiSpam <https://antispam.imp.ch/>`_
-  - `ISC SANS <https://isc.sans.edu/ipsascii.html>`_
   - `ISightPartners <http://www.isightpartners.com/>`_
   - `James Brine <https://jamesbrine.com.au/>`_
   - `Joewein <http://www.joewein.net>`_
   - Maltrail:
+
     - `Malware <https://github.com/stamparm/maltrail/tree/master/trails/static/malware>`_
     - `Suspicious <https://github.com/stamparm/maltrail/tree/master/trails/static/suspicious>`_
     - `Mass Scanners <https://github.com/stamparm/maltrail/blob/master/trails/static/mass_scanner.txt>`_ (for whitelisting)
   - `Malshare <https://malshare.com/>`_
   - `MalSilo Malware URLs <https://malsilo.gitlab.io/feeds/dumps/url_list.txt>`_
   - `Malware Config <http://malwareconfig.com>`_
   - `Malware DB (cert.pl) <https://mwdb.cert.pl/>`_
-  - `MalwareDomainList <http://www.malwaredomainlist.com/zeuscsv.php>`_
-  - `MalwareDomains <http://www.malwaredomainlist.com/hostslist/yesterday_urls.php>`_
   - `MalwareInt <http://malwareint.com>`_
   - `Malware Must Die <https://malwared.malwaremustdie.org/rss.php>`_
   - `Manity Spam IP addresses <http://www.dnsbl.manitu.net/download/nixspam-ip.dump.gz>`_
@@ -79,20 +77,18 @@ This list evolved from the issue :issue:`Contribute: Feeds List (#384) <384>`.
   - `mIRC Servers <http://www.mirc.com/servers.ini>`_
   - `Monzymerza <https://github.com/monzymerza/parthenon>`_
   - `Multiproxy <http://multiproxy.org/txt_all/proxy.txt>`_
-  - `MVPS <http://mvps.org>`_
   - `Neo23x0 signature-base <https://github.com/Neo23x0/signature-base/tree/master/iocs>`_
-  - `Null Secure <http://nullsecure.org>`_
   - `OpenBugBounty <https://www.openbugbounty.org/>`_
-  - `Payload Security <http://payload-security.com>`_
+  - `Phishing Army <https://phishing.army/>`_
   - `Project Honeypot (#284) <http://www.projecthoneypot.org/list_of_ips.php?rss=1>`_
   - `RST Threat Feed <https://rstcloud.net/>`_ (offers a free and a commercial feed)
+  - `SANS ISC <https://isc.sans.edu/api/>`_
   - `ShadowServer Sandbox API <http://www.shadowserver.org/wiki/pmwiki.php/Services/Sandboxapi>`_
   - `Shodan search API <https://shodan.readthedocs.io/en/latest/tutorial.html#searching-shodan>`_
   - `Snort <http://labs.snort.org/feeds/ip-filter.blf>`_
   - `stopforumspam Toxic IP addresses and domains <https://www.stopforumspam.com/downloads>`_
-  - `Spamhaus BGP feed (BGPf) <https://www.spamhaus.org/bgpf/>`_
+  - `Spamhaus Botnet Controller List <https://www.spamhaus.org/bcl/>`_
   - `SteveBlack Hosts File <https://github.com/StevenBlack/hosts>`_
-  - `TheCyberThreat <http://thecyberthreat.com/cyber-threat-intelligence-feeds/>`_
   - `The Haleys <http://charles.the-haleys.org/ssh_dico_attack_hdeny_format.php/hostsdeny.txt>`_
   - `Threat Crowd <https://www.threatcrowd.org/feeds/hashes.txt>`_
   - `Threat Grid <http://www.threatgrid.com/>`_
@@ -106,5 +102,4 @@ This list evolved from the issue :issue:`Contribute: Feeds List (#384) <384>`.
   - `Virustotal <https://www.virustotal.com/gui/home/search>`_
   - `virustream <https://github.com/ntddk/virustream>`_
   - `VoIP Blacklist <http://www.voipbl.org/update/>`_
-  - `Wordpress Callback Domains <http://callbackdomains.wordpress.com>`_
   - `YourCMC <http://vmx.yourcmc.ru/BAD_HOSTS.IP4>`_

docs/user/bots.rst

@@ -314,6 +314,7 @@ Generic Mail Attachment Fetcher
 * `ssl_ca_certificate`: Optional string of path to trusted CA certificate. Applies only to IMAP connections, not HTTP. If the provided certificate is not found, the IMAP connection will fail on handshake. By default, no certificate is used.
 
 The resulting reports contains the following special fields:
+
 * `extra.email_date`: The content of the email's `Date` header
 * `extra.email_subject`: The subject of the email
 * `extra.email_from`: The email's from address
@@ -353,6 +354,7 @@ Generic Mail Body Fetcher
   - `string`, e.g. `'plain'`
 
 The resulting reports contains the following special fields:
+
 * `extra.email_date`: The content of the email's `Date` header
 * `extra.email_subject`: The subject of the email
 * `extra.email_from`: The email's from address
@@ -545,6 +547,7 @@ MISP Generic
 * `misp_tag_processed`: MISP tag for processed events, optional
 
 Generic parameters used in this bot:
+
 * `http_verify_cert`: Verify the TLS certificate of the server, boolean (default: `true`)
 
 **Workflow**
@@ -1788,6 +1791,7 @@ Aggregate
 **Configuration Parameters**
 
 * **Cache parameters** (see in section :ref:`common-parameters`)
+
   * TTL is not used, using it would result in data loss.
 * **fields** Given fields which are used to aggregate like `classification.type, classification.identifier`
 * **threshold** If the aggregated event is lower than the given threshold after the timespan, the event will get dropped.
@@ -1989,10 +1993,16 @@ Deduplicator
 **Parameters for "fine-grained" deduplication**
 
 * `filter_type`: type of the filtering which can be "blacklist" or "whitelist". The filter type will be used to define how Deduplicator bot will interpret the parameter `filter_keys` in order to decide whether an event has already been seen or not, i.e., duplicated event or a completely new event.
+
   * "whitelist" configuration: only the keys listed in `filter_keys` will be considered to verify if an event is duplicated or not.
   * "blacklist" configuration: all keys except those in `filter_keys` will be considered to verify if an event is duplicated or not.
 * `filter_keys`: string with multiple keys separated by comma. Please note that `time.observation` key will not be considered even if defined, because the system always ignore that key.
 
+When using a whitelist field pattern and a small number of fields (keys), it becomes more important, that these fields exist in the events themselves.
+If a field does not exist, but is part of the hashing/deduplication, this field will be ignored.
+If such events should not get deduplicated, you need to filter them out before the deduplication process, e.g. using a sieve expert.
+See also `this discussion thread <https://lists.cert.at/pipermail/intelmq-users/2021-July/000370.html>`_ on the mailing-list.
+
 **Parameters Configuration Example**
 
 *Example 1*
@@ -2049,6 +2059,7 @@ DO Portal Expert Bot
 * `description:` The DO portal retrieves the contact information from a DO portal instance: http://github.com/certat/do-portal/
 
 **Configuration Parameters**
+
 * `mode` - Either `replace` or `append` the new abuse contacts in case there are existing ones.
 * `portal_url` - The URL to the portal, without the API-path. The used URL is `$portal_url + '/api/1.0/ripe/contact?cidr=%s'`.
 * `portal_api_key` - The API key of the user to be used. Must have sufficient privileges.
@@ -2068,6 +2079,7 @@ Field Reducer Bot
 * `description:` The field reducer bot is capable of removing fields from events.
 
 **Configuration Parameters**
+
 * `type` - either `"whitelist"` or `"blacklist"`
 * `keys` - Can be a JSON-list of field names (`["raw", "source.account"]`) or a string with a comma-separated list of field names (`"raw,source.account"`).
 
@@ -2093,17 +2105,18 @@ The filter bot is capable of filtering specific events.
 * `lookup:` none
 * `public:` yes
 * `cache (redis db):` none
-* `description:` filter messages (drop or pass messages) FIXME
+* `description:` A simple filter for messages (drop or pass) based on a exact string comparison or regular expression
 
 **Configuration Parameters**
 
 *Parameters for filtering with key/value attributes*
 
-* `filter_key` - key from data format
-* `filter_value` - value for the key
-* `filter_action` - action when a message match to the criteria (possible actions: keep/drop)
-* `filter_regex` - attribute determines if the `filter_value` shall be treated as regular expression or not.
-   If this attribute is not empty, the bot uses python's "search" function to evaluate the filter.
+* ``filter_key`` - key from data format
+* ``filter_value`` - value for the key
+* ``filter_action`` - action when a message match to the criteria (possible actions: keep/drop)
+* ``filter_regex`` - attribute determines if the ``filter_value`` shall be treated as regular expression or not.
+   If this attribute is not empty (can be ``true``, ``yes`` or whatever), the bot uses python's ```re.search`` <https://docs.python.org/3/library/re.html#re.search>`_ function to evaluate the filter with regular expressions.
+   If this attribute is empty or evaluates to false, an exact string comparison is performed. A check on string *inequality* can be achieved with the usage of *Paths* described below.
 
 *Parameters for time based filtering*
 
@@ -2175,17 +2188,19 @@ Format Field
 
    .. code-block:: json
 
-   "columns": "malware.name,extra.tags"
+      "columns": "malware.name,extra.tags"
 
-* `strip_chars` -  a set of characters to remove as leading/trailing characters(default: ` ` or whitespace)
+* `strip_chars` -  a set of characters to remove as leading/trailing characters(default: space)
 
 *Parameters for replacing chars*
+
 * `replace_column` - key from data format
 * `old_value` - the string to search for
 * `new_value` - the string to replace the old value with
 * `replace_count` - number specifying how many occurrences of the old value you want to replace(default: `1`)
 
 *Parameters for splitting string to list of string*
+
 * `split_column` - key from data format
 * `split_separator` - specifies the separator to use when splitting the string(default: `,`)
 
@@ -2725,13 +2740,15 @@ Sources:
 **Configuration Parameters**
 
 * `fields`: string, comma-separated list of fields e.g. `destination.ip,source.asn,source.url`. Supported fields are:
+
   * `destination.asn` & `source.asn`
   * `destination.fqdn` & `source.fqdn`
   * `destination.ip` & `source.ip`
   * `destination.url` & `source.url`
 * `policy`: string, comma-separated list of policies, e.g. `del,drop,drop`. `drop` will cause that the the entire event to be removed if the field is , `del` causes the field to be removed.
 
 With the example parameter values given above, this means that:
+
 * If a `destination.ip` value is part of a reserved network block, the field will be removed (policy "del").
 * If a `source.asn` value is in the range of reserved AS numbers, the event will be removed altogether (policy "drop).
 * If a `source.url` value contains a host with either an IP address part of a reserved network block, or a reserved domain name (or with a reserved TLD), the event will be dropped (policy "drop")
@@ -3150,6 +3167,7 @@ Threshold
 **Limitations**
 
 This bot has certain limitations and is not a true threshold filter (yet). It works like this:
+
 1. Every incoming message is hashed according to the `filter_*` parameters.
 2. The hash is looked up in the cache and the count is incremented by 1, and the TTL of the key is (re-)set to the timeout.
 3. If the new count matches the threshold exactly, the message is forwarded. Otherwise it is dropped.
@@ -3319,6 +3337,7 @@ Events without `source.url`, `source.fqdn`, `source.ip`, or `source.asn`, are ig
 only contains the domain. uWhoisd will automatically strip the subdomain part if it is present in the request.
 
 Example: `https://www.theguardian.co.uk`
+
 * TLD: `co.uk` (uWhoisd uses the `Mozilla public suffix list <https://publicsuffix.org/list/>`_ as a reference)
 * Domain: `theguardian.co.uk`
 * Subdomain: `www`
@@ -3877,6 +3896,7 @@ The parameters marked with 'PostgreSQL' will be sent to libpq via psycopg2. Chec
 **PostgreSQL**
 
 You have two basic choices to run PostgreSQL:
+
 1. on the same machine as intelmq, then you could use Unix sockets if available on your platform
 2. on a different machine. In which case you would need to use a TCP connection and make sure you give the right connection parameters to each psql or client call.
 

intelmq/bots/collectors/http/collector_http_stream.py

@@ -23,8 +23,9 @@
 
 from intelmq.lib.bot import CollectorBot
 from intelmq.lib.mixins import HttpMixin
-from intelmq.lib.utils import decode, create_request_session
-from intelmq.lib.exceptions import MissingDependencyError
+from intelmq.lib.utils import decode
+
+import requests.exceptions
 
 
 class HTTPStreamCollectorBot(CollectorBot, HttpMixin):
@@ -73,12 +74,12 @@ def process(self):
                     IncompleteRead,
                     ReadTimeoutError) as exc:
                 self.__error_count += 1
-                if (self.__error_count > self.parameters.error_max_retries):
+                if (self.__error_count > self.error_max_retries):
                     self.__error_count = 0
                     raise
                 else:
                     self.logger.info('Got exception %r, retrying (consecutive error count %d <= %d).',
-                                     exc, self.__error_count, self.parameters.error_max_retries)
+                                     exc, self.__error_count, self.error_max_retries)
 
             self.logger.info('Stream stopped.')
 

intelmq/bots/outputs/mcafee/output_esm_ip.py

@@ -43,7 +43,7 @@ def init(self):
 
         self.esm = ESM()
         try:
-            self.esm.login(self.parameters.esm_ip, self.parameters.esm_user, self.parameters.esm_password)
+            self.esm.login(self.esm_ip, self.esm_user, self.esm_password)
         except Exception:
             raise ValueError('Could not Login to ESM.')
 
@@ -53,7 +53,7 @@ def init(self):
             retVal = self.esm.post('sysGetWatchlists?hidden=false&dynamic=false&writeOnly=false&indexedOnly=false',
                                    watchlist_filter)
             for WL in retVal:
-                if (WL['name'] == self.parameters.esm_watchlist):
+                if (WL['name'] == self.esm_watchlist):
                     self.watchlist_id = WL['id']
         except TypeError:
             self.logger.error('Watchlist not found. Please verify name of the watchlist.')
@@ -64,7 +64,7 @@ def process(self):
         self.logger.info('Message received.')
         try:
             self.esm.post('sysAddWatchlistValues', {'watchlist': {'value': self.watchlist_id},
-                                                    'values': '["' + event.get(self.parameters.field) + '"]'},
+                                                    'values': '["' + event.get(self.field) + '"]'},
                           raw=True)
             self.logger.info('ESM Watchlist updated')
             self.acknowledge_message()