FIX: threshold expert: configure timeout using Redis cache mixin

CHANGELOG.md

@@ -75,12 +75,16 @@ CHANGELOG
 - `intelmq.bots.experts.truncate_by_delimiter.expert`: Cut string if its length is higher than a maximum length (PR#1967 by Marius Karotkis).
 - `intelmq.bots.experts.remove_affix`: Remove prefix or postfix strings from a field (PR#1965 by Marius Karotkis).
 - `intelmq.bots.experts.asn_lookup.expert`: Fixes update-database script on the last few days of a month (PR#2121 by Filip Pokorný, fixes #2088).
+- `intelmq.bots.experts.threshold.expert`: Correctly use the standard parameter `redis_cache_ttl` instead of the previously used parameter `timeout` (PR#2155 by Karl-Johan Karlsson).
+- `intelmq.bots.experts.jinja2.expert`: Lift restriction on requirement jinja2 < 3 (PR#2158 by Sebastian Wagner).
 
 #### Outputs
 - Removed `intelmq.bots.outputs.postgresql`: this bot was marked as deprecated in 2019 announced to be removed in version 3 of IntelMQ (PR#2045 by Birger Schacht).
 - Added `intelmq.bots.outputs.rpz_file.output` to create RPZ files (PR#1962 by Marius Karotkis).
 - Added `intelmq.bots.outputs.bro_file.output` to create Bro intel formatted files (PR#1963 by Marius Karotkis).
-- `intelmq.bots.outputs.templated_smtp.output`: Add new function `from_json()` (which just calls `json.loads()` in the standard Python environment), meaning the Templated SMTP output bot can take strings containing JSON documents and do the formatting itself (PR#2120 by Karl-Johan Karlsson).
+- `intelmq.bots.outputs.templated_smtp.output`:
+  - Add new function `from_json()` (which just calls `json.loads()` in the standard Python environment), meaning the Templated SMTP output bot can take strings containing JSON documents and do the formatting itself (PR#2120 by Karl-Johan Karlsson).
+  - Lift restriction on requirement jinja2 < 3 (PR#2158 by Sebastian Wagner).
 
 ### Documentation
 - Feeds: Add documentation for newly supported dataplane feeds, see above (PR#2102 by Mikk Margus Möll).

NEWS.md

@@ -1,5 +1,5 @@
 <!-- comment
-   SPDX-FileCopyrightText: 2015-2021 Sebastian Wagner
+   SPDX-FileCopyrightText: 2015-2022 Sebastian Wagner
    SPDX-License-Identifier: AGPL-3.0-or-later
 -->
 
@@ -31,6 +31,8 @@ sudo find /var/log/ -user intelmq ! -path \*intelmq\*
 ```
 
 ### Configuration
+#### Threshold Expert
+The parameter `timeout` has been merged into `redis_cache_ttl`.
 
 ### Libraries
 

docs/user/bots.rst

@@ -3194,8 +3194,6 @@ Threshold
 
 **Information**
 
-
-* **Cache parameters** (see section :ref:`common-parameters`)
 * `name`: `intelmq.bots.experts.threshold.expert`
 * `lookup`: redis cache
 * `public`: no
@@ -3204,9 +3202,9 @@ Threshold
 
 **Configuration Parameters**
 
+* **Cache parameters** (see section :ref:`common-parameters`), especially ``redis_cache_ttl`` as number of seconds before threshold counter is reset. Since version 3.1 (until 3.1 `timeout` was used).
 * `filter_keys`: String, comma-separated list of field names to consider or ignore when determining which messages are similar.
 * `filter_type`: String, `whitelist` (consider only the fields in `filter_keys`) or `blacklist` (consider everything but the fields in `filter_keys`).
-* `timeout`: Integer, number of seconds before threshold counter is reset.
 * `threshold`: Integer, number of messages required before propagating one. In forwarded messages, the threshold is saved in the message as `extra.count`.
 * `add_keys`: Array of string->string, optional, fields and values to add (or update) to propagated messages. Example:
 

intelmq/bots/experts/jinja/REQUIREMENTS.txt

@@ -0,0 +1,4 @@
+# SPDX-FileCopyrightText: 2022 Sebastian Wagner <sebix@sebix.at>
+# SPDX-License-Identifier: AGPL-3.0-or-later
+
+jinja2>=2.11

intelmq/bots/experts/jinja/expert.py

@@ -32,7 +32,7 @@ class JinjaExpertBot(Bot):
 
     def init(self):
         if not Template:
-            raise MissingDependencyError("Library 'jinja2' is required, please install it.")
+            raise MissingDependencyError("jinja2")
 
         for field, template in self.fields.items():
             if template.startswith("file:///"):

intelmq/bots/experts/threshold/expert.py

@@ -20,6 +20,9 @@
 
     redis_cache_password: string.  default: {None}
 
+    redis_cache_ttl: int, number of seconds to keep counts of similar
+                     messages.
+
     filter_type: string ["whitelist", "blacklist"], when determining
                  whether two messages are similar, consider either
                  only the named fields, or all but the named fields
@@ -35,11 +38,6 @@
                long as the count is above the threshold, no new
                messages will be sent.
 
-    timeout: int, number of seconds to keep counts of similar
-             messages. After this many seconds have elapsed, the count
-             is deleted and "threshold" number of new messages will
-             result in a new message being sent.
-
     add_keys: optional, array of strings to strings, keys to add to
               forwarded messages. Regardless of this setting, the
               field "extra.count" will be set to the number of
@@ -59,19 +57,16 @@ class ThresholdExpertBot(ExpertBot, CacheMixin):
     filter_keys: Iterable = ["raw", "time.observation"]
     filter_type: str = "blacklist"
     redis_cache_db: int = 11
-    redis_cache_host: str = "127.0.0.1"  # TODO: could be ipaddress
-    redis_cache_password: Optional[str] = None
-    redis_cache_port: int = 6379
+    redis_cache_ttl: int = 3600
     threshold: int = 100
-    timeout: int = 3600
 
     _message_processed_verb = 'Forwarded'
 
     _is_multithreadable = False
     bypass = False
 
     def init(self):
-        if self.timeout <= 0:
+        if self.redis_cache_ttl <= 0:
             raise ConfigurationError('Timeout', 'Invalid timeout specified, use positive integer seconds.')
         if self.threshold <= 0:
             raise ConfigurationError('Threshold', 'Invalid threshold specified, use positive integer count.')

intelmq/bots/outputs/templated_smtp/REQUIREMENTS.txt

@@ -1,4 +1,4 @@
 # SPDX-FileCopyrightText: 2021 Linköping University <https://liu.se/>
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
-jinja2>=2.11,<3
+jinja2>=2.11