Merge branch 'maintenance' into develop

Author: Sebastian Wagner

CHANGELOG.md

@@ -98,6 +98,7 @@ CHANGELOG
 ### Documentation
 - Various formatting fixes (by Sebastian Wagner).
 - Removed the malwaredomains feed from the feeds list because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
+- Update Docker installation instructions (PR#2035 by Sebastian Waldbauer).
 
 ### Packaging
 - intelmq-update-database crontab: Add missing `recordedfuture_iprisk` update call (by Sebastian Wagner).
@@ -108,6 +109,7 @@ CHANGELOG
 - `intelmq.tests.bots.collectors.mail.test_collector_attach`: Test text attachment (by Sebastian Wagner).
 
 ### Tools
+- `intelmqctl`: Also honour parameters from environment variables (PR#2068 by Sebastian Wagner, fixes #2063).
 
 ### Contrib
 

docs/user/bots.rst

@@ -3279,7 +3279,12 @@ Trusted Introducer Lookup Expert
 
 **Configuration Parameters**
 
-* **order**: Possible values are 'domain', 'asn'
+* **order**: Possible values are 'domain', 'asn'. You can set multiple values, so first match wins.
+* If 'domain' is set, it will lookup the `source.fqdn` field. It will go from high-order to low-order, i.e. 1337.super.example.com -> super.example.com -> example.com -> `.com`
+* If 'asn' is set, it will lookup `source.asn`.
+
+After a match, the abuse contact will be fetched from the trusted introducer teams list and will be stored in the event as `source.abuse_contact`.
+If there is no match, the event will not be enriched and will be sent to the next configured step.
 
 
 .. _intelmq.bots.experts.tuency.expert:

docs/user/configuration-management.rst

@@ -129,13 +129,16 @@ Miscellaneous
     * **false** - duplicates the messages into each queue
     * When using AMQP as message broker, take a look at the :ref:`multithreading` section and the ``instances_threads`` parameter.
 
-* **broker** - select which broker intelmq can use. Use the following values:
-    * **redis** - Redis allows some persistence but is not so fast as ZeroMQ (in development). But note that persistence has to be manually activated. See http://redis.io/topics/persistence
-
 * **rate_limit** - time interval (in seconds) between messages processing.  int value.
 
 * **ssl_ca_certificate** - trusted CA certificate for IMAP connections (supported by some bots).
 
+* **source_pipeline_broker** & **destination_pipeline_broker** - select which broker IntelMQ should use. There are two options
+    * **redis** (default) - Please note that persistence has to be `manually activated <http://redis.io/topics/persistence>`_.
+    * **amqp** - The AMQP pipeline is currently beta but there are no known issues. A popular AMQP broker is `RabbitMQ <https://www.rabbitmq.com/>`_. See :ref:`aqmp pipeline broker` for more details.
+
+  * As these parameters can be set per bot, this allows usage of different broker systems and hosts, as well as switching between them on the same IntelMQ instance.
+
 * **source_pipeline_host** - broker IP, FQDN or Unix socket that the bot will use to connect and receive messages.
 
 * **source_pipeline_port** - broker port that the bot will use to connect and receive messages. Can be empty for Unix socket.
@@ -290,6 +293,8 @@ In case of errors during processing, and the optional path ``"_on_error"`` is sp
 Other destination queues can be explicitly addressed by the bots, e.g. bots with filtering capabilities. Some expert bots are capable of sending messages to paths, this feature is explained in their documentation, e.g. the :ref:`intelmq.bots.experts.filter.expert` expert and the :ref:`intelmq.bots.experts.sieve.expert` expert.
 The named queues need to be explicitly addressed by the bot (e.g. filtering) or the core (``_on_error``) to be used. Setting arbitrary paths has no effect.
 
+.. _aqmp pipeline broker:
+
 AMQP (Beta)
 -----------
 

intelmq/bin/intelmqctl.py

@@ -1017,14 +1017,20 @@ def load_defaults_configuration(self, silent=False):
         for option, value in utils.get_global_settings().items():
             setattr(self.parameters, option, value)
 
-        # TODO: Rewrite variables with env. variables ( CURRENT IMPLEMENTATION NOT FINAL )
-        # "destination_pipeline_host": "127.0.0.1",
-        # "source_pipeline_host": "127.0.0.1",
-        if os.getenv('INTELMQ_IS_DOCKER', None):
-            pipeline_host = os.getenv('INTELMQ_PIPELINE_HOST')
-            if pipeline_host:
-                setattr(self.parameters, 'destination_pipeline_host', pipeline_host)
-                setattr(self.parameters, 'source_pipeline_host', pipeline_host)
+        # copied from intelmq.lib.bot, should be refactored to e.g. intelmq.lib.config
+        intelmq_environment = [elem for elem in os.environ if elem.startswith('INTELMQ_')]
+        for elem in intelmq_environment:
+            option = elem[8:].lower()
+            value = os.environ[elem]
+            # do some conversions:
+            if value == 'True':
+                value = True
+            elif value == 'False':
+                value = False
+            elif value.isnumeric():
+                value = int(value)
+
+            setattr(self.parameters, option, value)
 
     def run(self):
         results = None