Merge pull request #2414 from zuo/jk/ssl-improvements-related-to-stomp-and-n6

STOMP+SSL-related changes: security fixes + convenience enhancement (+ doc updates/improvements)

Author: sebix

CHANGELOG.md

@@ -19,6 +19,10 @@
     if `auth_by_ssl_client_certificate` is *false*);
   - `password` (STOMP authentication passcode, default: "guest"; to be used only
     if `auth_by_ssl_client_certificate` is *false*).
+- Add the possibility to set the `ssl_ca_certificate` configuration parameter for
+  `intelmq.bots.collectors.stomp.collector` and/or `intelmq.bots.outputs.stomp.output`
+  to an empty string - which means that the SSL machinery used for STOMP communication
+  will attempt to load the system’s default CA certificates (PR#2414 by Jan Kaliszewski).
 
 ### Core
 - `intelmq.lib.message`: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).
@@ -27,7 +31,7 @@
 - `intelmq.lib.mixins`: Add a new class, `StompMixin` (defined in a new submodule: `stomp`),
   which provides certain common STOMP-bot-specific operations, factored out from
   `intelmq.bots.collectors.stomp.collector` and `intelmq.bots.outputs.stomp.output`
-  (PR#2408 by Jan Kaliszewski).
+  (PR#2408 and PR#2414 by Jan Kaliszewski).
 
 ### Development
 - Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).
@@ -36,11 +40,16 @@
 
 ### Bots
 #### Collectors
-- `intelmq.bots.collectors.stomp.collector` (PR#2408 by Jan Kaliszewski):
-  - Add support for authentication based on STOMP login and passcode,
-    introducing 3 new configuration parameters (see above: *Configuration*).
+- `intelmq.bots.collectors.stomp.collector` (PR#2408 and PR#2414 by Jan Kaliszewski):
+  - Drop support for versions of `stomp.py` older than `4.1.12`.
   - Update the code to support new versions of `stomp.py`, including the latest (`8.1.0`);
     fixes [#2342](https://github.com/certtools/intelmq/issues/2342).
+  - Add support for authentication based on STOMP login and passcode, introducing three
+    new configuration parameters (see above: *Configuration*).
+  - Add support for loading the system’s default CA certificates, as an alternative to
+    specifying the CA certificate(s) file path explicitly (see above: *Configuration*).
+  - Fix (by carefully targeted monkey patching) certain security problems caused by
+    SSL-related weaknesses that some versions of `stomp.py` suffer from.
   - Fix the reconnection behavior: do not attempt to reconnect after `shutdown`. Also,
     never attempt to reconnect if the version of `stomp.py` is older than `4.1.21` (it
     did not work properly anyway).
@@ -59,27 +68,35 @@
     file permissions on socket file, if it is in use.
 
 #### Outputs
-- `intelmq.bots.outputs.stomp.output` (PR#2408 by Jan Kaliszewski):
-  - Add support for authentication based on STOMP login and passcode,
-    introducing 3 new configuration parameters (see above: *Configuration*).
+- `intelmq.bots.outputs.stomp.output` (PR#2408 and PR#2414 by Jan Kaliszewski):
+  - Drop support for versions of `stomp.py` older than `4.1.12`.
   - Update the code to support new versions of `stomp.py`, including the latest (`8.1.0`).
+  - Add support for authentication based on STOMP login and passcode, introducing three
+    new configuration parameters (see above: *Configuration*).
+  - Add support for loading the system’s default CA certificates, as an alternative to
+    specifying the CA certificate(s) file path explicitly (see above: *Configuration*).
+  - Fix (by carefully targeted monkey patching) certain security problems caused by
+    SSL-related weaknesses that some versions of `stomp.py` suffer from.
   - Fix `AttributeError` caused by attempts to get unset attributes of `StompOutputBot`
     (`ssl_ca_cert` et consortes).
   - Add coercion of the `port` config parameter to `int`.
   - Add implementation of the `check` hook (verifying, in particular, accessibility
     of necessary file(s)).
-  - Add `stomp.py` version check (raise `MissingDependencyError` if not `>=4.1.8`).
+  - Add `stomp.py` version check (raise `MissingDependencyError` if not `>=4.1.12`).
   - Minor fixes/improvements and some refactoring (see also above: *Core*...).
 
 ### Documentation
 - Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
 - Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
 - Update/fix/improve the stuff related to the STOMP bots and integration with the *n6*'s
-  Stream API (PR#2408 by Jan Kaliszewski).
+  Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
 - Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný).
 
 ### Packaging
 - Add `pendulum` to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner).
+- `debian/control`: in `Suggests` field, replace ``python3-stomp.py (>= 4.1.9)`` with
+  ``python3-stomp (>= 4.1.12)``, i.e., fix the package name by removing the `.py`
+  suffix and bump the minimum version to `4.1.12` (PR#2414 by Jan Kaliszewski).
 
 ### Tests
 

debian/control

@@ -53,7 +53,7 @@ Suggests: python3-geoip2 (>= 2.2.0),
           python3-pyasn (>= 1.5.0),
           python3-pymongo (>= 2.7.1),
           python3-sleekxmpp (>= 1.3.1),
-          python3-stomp.py (>= 4.1.9),
+          python3-stomp (>= 4.1.12),
           python3-pendulum
 Description: Solution for IT security teams for collecting and processing security feeds
  IntelMQ is a solution for IT security teams (CERTs, CSIRTs, abuse

docs/user/bots.md

@@ -1184,35 +1184,49 @@ Install the `stomp.py` library from PyPI:
 pip3 install -r intelmq/bots/collectors/stomp/REQUIREMENTS.txt
 ```
 
+Alternatively, you may want to install it using your OS's native
+packaging tools, e.g.:
+
+```bash
+apt install python3-stomp
+```
+
+Apart from that, depending on what STOMP server you connect to, you may
+need to obtain, from the organization or company owning the server, one
+or more of the following security/authentication-related resources:
+
+* CA certificate file;
+* either: *client certificate* and *client certificate's key* files,
+  or: *username* (STOMP *login*) and *password* (STOMP *passcode*).
+
+Also, you will need to know an appropriate STOMP *destination* (aka
+*exchange point*), e.g. `/exchange/my.example.org/*.*.*.*`.
+
 **Parameters (also expects [feed parameters](#feed-parameters)):**
 
 **`server`**
 
-(required, string) Hostname of the STOMP server.
+(required, string) STOMP server's hostname or IP, e.g. "n6stream.cert.pl" (which is default)
 
 **`port`**
 
-(optional, integer) Defaults to 61614.
+(optional, integer) STOMP server's port number (default: 61614)
 
 **`exchange`**
 
-(required, string) STOMP *destination* to subscribe to, e.g. "/exchange/my.org/*.*.*.*"
+(required, string) STOMP *destination* to subscribe to, e.g. `"/exchange/my.org/*.*.*.*"`
 
-**`username`**
-
-(optional, string) Username to use.
+**`heartbeat`**
 
-**`password`**
-
-(optional, string) Password to use.
+(optional, integer) default: 6000
 
 **`ssl_ca_certificate`**
 
-(optional, string) Path to trusted CA certificate.
+(optional, string) Path to CA file, or empty string to load system's default CA certificates
 
 **`auth_by_ssl_client_certificate`**
 
-(optional, boolean) Whether to authenticate using TLS certificate. (Set to false for new *n6* auth.) Defaults to true.
+(optional, boolean) Default: true (note: false is needed for new *n6* auth)
 
 **`ssl_client_certificate`**
 
@@ -1222,6 +1236,14 @@ pip3 install -r intelmq/bots/collectors/stomp/REQUIREMENTS.txt
 
 (optional, string) Path to client private key to use for TLS connections.
 
+**`username`**
+
+(optional, string) Username to use.
+
+**`password`**
+
+(optional, string) Password to use.
+
 ---
 
 ### Twitter (REMOVE?) <div id="intelmq.bots.collectors.twitter.collector_twitter" />
@@ -5135,72 +5157,87 @@ This bot pushes data to any STOMP stream. STOMP stands for Streaming Text Orient
 
 **Requirements**
 
-Install the stomp.py library, e.g. [apt install python3-stomp.py] or [pip install stomp.py].
+Install the `stomp.py` library from PyPI:
 
-You need a CA certificate, client certificate and key file from the organization / server you are connecting to. Also
-you will need a so called "exchange point".
+```bash
+pip3 install -r intelmq/bots/outputs/stomp/REQUIREMENTS.txt
+```
 
-**Parameters:**
+Alternatively, you may want to install it using your OS's native
+packaging tools, e.g.:
 
-**`exchange`**
+```bash
+apt install python3-stomp
+```
 
-(optional, string) The exchange to push to. Defaults to `/exchange/_push`.
+Apart from that, depending on what STOMP server you connect to, you may
+need to obtain, from the organization or company owning the server, one
+or more of the following security/authentication-related resources:
 
-**`username`**
+* CA certificate file;
+* either: *client certificate* and *client certificate's key* files,
+  or: *username* (STOMP *login*) and *password* (STOMP *passcode*).
 
-(optional, string) Username to use.
+Also, you will need to know an appropriate STOMP *destination* (aka
+*exchange point*), e.g. `/exchange/_push`.
 
-**`password`**
+**Parameters:**
 
-(optional, string) Password to use.
+**`server`**
 
-**`ssl_ca_certificate`**
+(optional, string) STOMP server's hostname or IP, e.g. "n6stream.cert.pl" or "127.0.0.1" (which is default)
 
-(optional, string) Path to trusted CA certificate.
+**`port`**
 
-**`auth_by_ssl_client_certificate`**
+(optional, integer) STOMP server's port number (default: 61614)
 
-(optional, boolean) Whether to authenticate using TLS certificate. (Set to false for new *n6* auth.) Defaults to true.
+**`exchange`**
+
+(optional, string) STOMP *destination* to push at, e.g. ``"/exchange/_push"`` (which is default)
 
 **`heartbeat`**
 
 (optional, integer) Defaults to 60000.
 
-**`message_hierarchical_output`**
+**`ssl_ca_certificate`**
 
-(optional, boolean) Defaults to false.
+(optional, string) path to CA file, or empty string to load system's default CA certificates
 
-**`message_jsondict_as_string`**
+**`auth_by_ssl_client_certificate`**
 
-(optional, boolean) Defaults to false.
+(optional, boolean) default: true (note: false is needed for new *n6* auth)
 
-**`message_with_type`**
+**`ssl_client_certificate`**
 
-(optional, boolean) Defaults to false.
+(optional, string) Path to client certificate to use for TLS connections.
 
-**`port`**
+**`ssl_client_certificate_key`**
 
-(optional, integer) Defaults to 61614.
+(optional, string) Path to client private key to use for TLS connections.
 
-**`server`**
+**`username`**
 
-(optional, string) Hostname of the STOMP server.
+(optional, string) STOMP *login* (e.g., *n6* user login), used only if `auth_by_ssl_client_certificate` is false
 
-**`single_key`**
+**`password`**
 
-(optional, string) Output only a single specified key. In case of `raw` key the data is base64 decoded. Defaults to null (output the whole message).
+(optional, string) STOMP *passcode* (e.g., *n6* user API key), used only if `auth_by_ssl_client_certificate` is false
 
-**`ssl_ca_certificate`**
+**`message_hierarchical_output`**
 
-(optional, string) Path to trusted CA certificate.
+(optional, boolean) Defaults to false.
 
-**`ssl_client_certificate`**
+**`message_jsondict_as_string`**
 
-(optional, string) Path to client certificate to use for TLS connections.
+(optional, boolean) Defaults to false.
 
-**`ssl_client_certificate_key`**
+**`message_with_type`**
 
-(optional, string) Path to client private key to use for TLS connections.
+(optional, boolean) Defaults to false.
+
+**`single_key`**
+
+(optional, string) Output only a single specified key. In case of `raw` key the data is base64 decoded. Defaults to null (output the whole message).
 
 ---
 

docs/user/feeds.md

@@ -719,15 +719,15 @@ parameters:
 
 ### N6 Stomp Stream
 
-N6 Collector - CERT.pl's N6 Collector - N6 feed via STOMP interface. Note that rate_limit does not apply for this bot as it is waiting for messages on a stream.
+N6 Collector - CERT.pl's *n6* Stream API feed (via STOMP interface). Note that 'rate_limit' does not apply to this bot, as it is waiting for messages on a stream.
 
 **Public:** no
 
-**Revision:** 2023-09-23
+**Revision:** 2023-10-08
 
 **Documentation:** <https://n6.readthedocs.io/usage/streamapi/>
 
-**Additional Information:** Contact cert.pl to get access to the feed.
+**Additional Information:** Contact CERT.pl to get access to the feed. Note that the configuration parameter values suggested here are suitable for the new *n6* Stream API variant (with authentication based on 'username' and 'password'); for this variant, typically you can leave the 'ssl_ca_certificate' parameter's value empty - then the system's default CA certificates will be used; however, if that does not work, you need to set 'ssl_ca_certificate' to the path to a file containing CA certificates eligible to verify "*.cert.pl" server certificates (to be found among the publicly available CA certs distributed with modern web browsers/OSes). Also, note that the 'server' parameter's value (for the *new API variant*) suggested here, "n6stream-new.cert.pl", is a temporary domain; ultimately, it will be changed back to "stream.cert.pl". When it comes to the *old API variant* (turned off in November 2023!), you need to have the 'server' parameter set to the name "n6stream.cert.pl", 'auth_by_ssl_client_certificate' set to true, 'ssl_ca_certificate' set to the path to a file containing the *n6*'s legacy self-signed CA certificate (which is stored in file "intelmq/bots/collectors/stomp/ca.pem"), and the parameters 'ssl_client_certificate' and 'ssl_client_certificate_key' set to the paths to your-*n6*-client-specific certificate and key files (note that the 'username' and 'password' parameters are then irrelevant and can be omitted).
 
 
 **Collector configuration**
@@ -736,14 +736,14 @@ N6 Collector - CERT.pl's N6 Collector - N6 feed via STOMP interface. Note that r
 module: intelmq.bots.collectors.stomp.collector
 parameters:
   auth_by_ssl_client_certificate: False
-  exchange: {insert your exchange point as given by CERT.pl}
+  exchange: {insert your STOMP *destination* to subscribe to, as given by CERT.pl, e.g. /exchange/my.example.org/*.*.*.*}
   name: N6 Stomp Stream
-  password: {insert n6 user's API key}
+  password: {insert your *n6* API key}
   port: 61614
   provider: CERT.PL
-  server: n6stream.cert.pl
-  ssl_ca_certificate: {insert path to CA file for CERT.pl's n6}
-  username: {insert n6 user's login}
+  server: n6stream-new.cert.pl
+  ssl_ca_certificate:
+  username: {insert your *n6* login, e.g. someuser@my.example.org}
 ```
 
 **Parser configuration**

intelmq/bots/collectors/stomp/REQUIREMENTS.txt

@@ -1,4 +1,4 @@
 # SPDX-FileCopyrightText: 2017 Sebastian Wagner
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
-stomp.py>=4.1.8
+stomp.py>=4.1.12

intelmq/bots/collectors/stomp/collector.py

@@ -4,15 +4,19 @@
 
 # -*- coding: utf-8 -*-
 
-from intelmq.lib.bot import CollectorBot
-from intelmq.lib.mixins import StompMixin
-
 try:
     import stomp
-    import stomp.exception
 except ImportError:
     stomp = None
 else:
+    import stomp.exception
+
+from intelmq.lib.bot import CollectorBot
+from intelmq.lib.mixins import StompMixin
+
+
+if stomp is not None:
+
     class StompListener(stomp.PrintingListener):
         """
         the stomp listener gets called asynchronously for
@@ -74,17 +78,33 @@ def connect_and_subscribe(conn, logger, destination, start=False, connect_kwargs
 class StompCollectorBot(CollectorBot, StompMixin):
     """Collect data from a STOMP Interface"""
     """ main class for the STOMP protocol collector """
-    exchange: str = ''
+
+    server: str = 'n6stream.cert.pl'
     port: int = 61614
-    server: str = "n6stream.cert.pl"
-    auth_by_ssl_client_certificate: bool = True
-    username: str = 'guest'  # ignored if `auth_by_ssl_client_certificate` is true
-    password: str = 'guest'  # ignored if `auth_by_ssl_client_certificate` is true
-    ssl_ca_certificate: str = 'ca.pem'  # TODO pathlib.Path
-    ssl_client_certificate: str = 'client.pem'  # TODO pathlib.Path
-    ssl_client_certificate_key: str = 'client.key'  # TODO pathlib.Path
+    exchange: str = ''
     heartbeat: int = 6000
 
+    # Note: the `ssl_ca_certificate` configuration parameter must be set:
+    # * *either* to the server's CA certificate(s) file path,
+    # * *or* to an empty string -- dictating that the SSL tools employed
+    #   by the `stomp.py`'s machinery will attempt to load the system’s
+    #   default CA certificates.
+    # The latter, if applicable, is more convenient -- by avoiding the
+    # need to manually update the CA certificate(s) file.
+    ssl_ca_certificate: str = 'ca.pem'  # <- TODO: change to '' (+ remove "ca.pem*" legacy files)
+    # (^ TODO: could also be pathlib.Path)
+
+    auth_by_ssl_client_certificate: bool = True
+
+    # Used if `auth_by_ssl_client_certificate` is true (otherwise ignored):
+    ssl_client_certificate: str = 'client.pem'       # (cert file path)
+    ssl_client_certificate_key: str = 'client.key'   # (cert's key file path)
+    # (^ TODO: could also be pathlib.Path)
+
+    # Used if `auth_by_ssl_client_certificate` is false (otherwise ignored):
+    username: str = 'guest'   # (STOMP auth *login*)
+    password: str = 'guest'   # (STOMP auth *passcode*)
+
     _collector_empty_process: bool = True
     __conn = False  # define here so shutdown method can check for it