Add Anomali ThreatStream support to threatintel module

This adds a new dataset, `anomali_threatstream` to the threatintel
module. It allows to ingest indicators from Anomali ThreatStream
Integrator via a custom SDK output.

filebeat/docs/fields.asciidoc

@@ -152754,6 +152754,191 @@ type: keyword
 The STIX reference object.
 
 
+type: keyword
+
+--
+
+[float]
+=== anomali_threatstream
+
+Fields for Anomali ThreatStream
+
+
+
+*`threatintel.anomali_threatstream.classification`*::
++
+--
+Indicates whether an indicator is private or from a public feed and available publicly. Possible values: private, public.
+
+
+type: keyword
+
+example: private
+
+--
+
+*`threatintel.anomali_threatstream.confidence`*::
++
+--
+The measure of the accuracy (from 0 to 100) assigned by ThreatStream's predictive analytics technology to indicators.
+
+
+type: short
+
+--
+
+*`threatintel.anomali_threatstream.detail2`*::
++
+--
+Detail text for indicator.
+
+
+type: text
+
+example: Imported by user 42.
+
+--
+
+*`threatintel.anomali_threatstream.id`*::
++
+--
+The ID of the indicator.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomali_threatstream.import_session_id`*::
++
+--
+ID of the import session that created the indicator on ThreatStream.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomali_threatstream.itype`*::
++
+--
+Indicator type. Possible values: "apt_domain", "apt_email", "apt_ip", "apt_url", "bot_ip", "c2_domain", "c2_ip", "c2_url", "i2p_ip", "mal_domain", "mal_email", "mal_ip", "mal_md5", "mal_url", "parked_ip", "phish_email", "phish_ip", "phish_url", "scan_ip", "spam_domain", "ssh_ip", "suspicious_domain", "tor_ip" and "torrent_tracker_url".
+
+
+type: keyword
+
+--
+
+*`threatintel.anomali_threatstream.maltype`*::
++
+--
+Information regarding a malware family, a CVE ID, or another attack or threat, associated with the indicator.
+
+
+type: wildcard
+
+--
+
+*`threatintel.anomali_threatstream.md5`*::
++
+--
+Hash for the indicator.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomali_threatstream.resource_uri`*::
++
+--
+Relative URI for the indicator details.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomali_threatstream.severity`*::
++
+--
+Criticality associated with the threat feed that supplied the indicator. Possible values: low, medium, high, very-high.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomali_threatstream.source`*::
++
+--
+Source for the indicator.
+
+
+type: keyword
+
+example: Analyst
+
+--
+
+*`threatintel.anomali_threatstream.source_feed_id`*::
++
+--
+ID for the integrator source.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomali_threatstream.state`*::
++
+--
+State for this indicator.
+
+
+type: keyword
+
+example: active
+
+--
+
+*`threatintel.anomali_threatstream.trusted_circle_ids`*::
++
+--
+ID of the trusted circle that imported the indicator.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomali_threatstream.update_id`*::
++
+--
+Update ID.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomali_threatstream.url`*::
++
+--
+URL for the indicator.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomali_threatstream.value_type`*::
++
+--
+Data type of the indicator. Possible values: ip, domain, url, email, md5.
+
+
 type: keyword
 
 --

filebeat/docs/modules/threatintel.asciidoc

@@ -22,15 +22,17 @@ fields.
 
 The available filesets are:
 
-* `abuseurl`: Supports gathering URL entities from Abuse.ch.
-* `abusemalware`: Supports gathering Malware/Payload entities from Abuse.ch.
-* `misp`: Supports gathering threat intel attributes from MISP (replaces MISP module).
-* `malwarebazaar`: Supports gathering Malware/Payload entities from Malware Bazaar.
-* `otx`: Supports gathering threat intel attributes from AlientVault OTX.
-* `anomali`: Supports gathering threat intel attributes from Anomali.
+* <<abuseurl,abuseurl>>: Supports gathering URL entities from Abuse.ch.
+* <<abusemalware,abusemalware>>: Supports gathering Malware/Payload entities from Abuse.ch.
+* <<misp,misp>>: Supports gathering threat intel attributes from MISP (replaces MISP module).
+* <<malwarebazaar,malwarebazaar>>: Supports gathering Malware/Payload entities from Malware Bazaar.
+* <<otx,otx>>: Supports gathering threat intel attributes from AlientVault OTX.
+* <<anomali,anomali>>: Supports gathering threat intel attributes from Anomali Limo.
+* <<anomali_threatstream,anomali_threatstream>>: Supports gathering threat intel attributes from Anomali ThreatStream.
 
 include::../include/gs-link.asciidoc[]
 
+[[abuseurl]]
 [float]
 ==== `abuseurl` fileset settings
 
@@ -70,6 +72,7 @@ Abuse.ch URL Threat Intel is mapped to the following ECS fields.
 | host                       | threatintel.indicator.ip/domain
 |==============================================================
 
+[[abusemalware]]
 [float]
 ==== `abusemalware` fileset settings
 
@@ -109,6 +112,7 @@ Abuse.ch Malware Threat Intel is mapped to the following ECS fields.
 | file_size                  | threatintel.indicator.file.size
 |================================================================
 
+[[malwarebazaar]]
 [float]
 ==== `malwarebazaar` fileset settings
 
@@ -163,6 +167,7 @@ Malware Bazaar Threat Intel is mapped to the following ECS fields.
 | code_sign.serial_number    | threatintel.indicator.file.x509.serial_number
 |================================================================
 
+[[misp]]
 [float]
 ==== `misp` fileset settings
 
@@ -240,6 +245,7 @@ MISP Threat Intel is mapped to the following ECS fields.
 
 `misp.value` is mapped to the appropriate field dependent on attribute type.
 
+[[otx]]
 [float]
 ==== `otx` fileset settings
 
@@ -315,6 +321,7 @@ OTX Threat Intel is mapped to the following ECS fields.
 
 `otx.indicator` is mapped to the appropriate field dependent on attribute type.
 
+[[anomali]]
 [float]
 ==== `anomali` fileset settings
 
@@ -396,6 +403,101 @@ Anomali Threat Intel is mapped to the following ECS fields.
 
 `anomali.pattern` is mapped to the appropriate field dependent on attribute type.
 
+[[anomali_threatstream]]
+[float]
+==== `anomali_threatstream` fileset settings
+
+To configure the ThreatStream integration you first need to define an output
+in the Anomali ThreatStream Integrator using the Elastic SDK provided by Anomali.
+It will deliver indicators via HTTP or HTTPS to a Filebeat instance running as
+a server.
+
+Configure an Integrator output with the following settings:
+
+* Indicator Filter: `*` (or use any desired filter).
+* SDK Executable Command: `/path/to/python /path/to/anomali-sdk/main.py`.
+  Adjust the paths to the python executable and the directory where the Elastic SDK
+  has been unpacked.
+* Metadata in JSON Format: `{"url": "https://filebeat:8080/", "server_certificate": "/path/to/cert.pem", "secret": "my secret"}`.
+    - `url`: Use the host and port where Filebeat will be running, and `http` or `https` accordingly.
+    - `server_certificate`: If using HTTPS, absolute path to the server certificate. Otherwise don't set
+        this field.
+    - `secret`: A shared secret string to authenticate messages between the SDK and Filebeat.
+
+Then configure the `anomali_threatstream` fileset in Filebeat accordingly:
+[source,yaml]
+----
+- module: threatintel
+  anomali_threatstream:
+    enabled: true
+    var.input: http_endpoint
+    var.listen_address: 0.0.0.0 # Listen on all interfaces.
+    var.listen_port: 8080
+    var.secret: 'Same secret as used above'
+    var.ssl_certificate: path/to/server_ssl_cert.pem
+    var.ssl_key: path/to/ssl_key.pem
+----
+
+To filter on specific types, you can define `var.types` as a comma delimited list
+of object types. This defaults to "indicators".
+
+
+*`var.input`*::
+
+The Filebeat input to use. Use either `http_endpoint` for an HTTP Server or `file`
+for ingesting indicators from a file.
+
+*`var.listen_address`*::
+
+Local address to bind the HTTP server to. Use `0.0.0.0` to accept connections
+from all interfaces.
+
+*`var.listen_port`*::
+
+Port number to use for the HTTP server.
+
+*`var.secret`*::
+
+Shared secret string between the SDK and Filebeat to authenticate messages.
+
+*`var.ssl_certificate`*::
+
+Path to the public SSL certificate for the HTTPS server. If unset, Filebeat
+will use unsecure HTTP connections.
+
+*`var.ssl_key`*::
+
+Path to the certificate's private key.
+
+include::../include/var-paths.asciidoc[]
+
+Anomali ThreatStream fields are mapped to the following ECS fields:
+
+[options="header"]
+|=============================================================
+| ThreatStream fields | ECS Fields
+| asn         | threatintel.indicator.as.number
+| classification<<a,[1]>> | threatintel.indicator.marking.tlp
+| confidence<<a,[1]>> | threatintel.indicator.confidence
+| country     | threatintel.indicator.geo.country_iso_code
+| date_first  | threatintel.indicator.first_seen
+| date_last   | threatintel.indicator.last_seen
+| detail      | tags
+| domain      | threatintel.indicator.domain
+| email       | threatintel.indicator.email.address
+| itype<<a,[1]>> | threatintel.indicator.type
+| lat         | threatintel.indicator.geo.location.lat
+| lon         | threatintel.indicator.geo.location.lon
+| md5         | threatintel.indicator.file.hash
+| org         | threatintel.indicator.as.organization.name
+| severity<<a,[1]>> | event.severity
+| srcip       | threatintel.indicator.ip
+| url         | threatintel.indicator.url.original
+|=============================================================
+
+[[a]]
+[small]#[1]: Field is used to derive a value for the ECS field but its original value is kept under `threatintel.anomali_threatstream`.#
+
 :has-dashboards!:
 
 [float]

filebeat/tests/system/test_modules.py

@@ -277,6 +277,7 @@ def clean_keys(obj):
         "threatintel.abuseurl",
         "threatintel.abusemalware",
         "threatintel.anomali",
+        "threatintel.anomali_threatstream",
         "threatintel.malwarebazaar",
         "snyk.vulnerabilities",
         "snyk.audit",

x-pack/filebeat/filebeat.reference.yml

@@ -2247,6 +2247,31 @@ filebeat.modules:
     # The interval to poll the API for updates
     var.interval: 5m
 
+  anomali_threatstream:
+    enabled: true
+
+    # Input used for ingesting threat intel data
+    var.input: http_endpoint
+
+    # Address to bind to in order to receive HTTP requests
+    # from the Integrator SDK. Use 0.0.0.0 to bind to all
+    # existing interfaces.
+    var.listen_address: localhost
+
+    # Port to use to receive HTTP requests from the
+    # Integrator SDK.
+    var.listen_port: 8080
+
+    # Secret key to authenticate requests from the SDK.
+    var.secret: '<Add your secret here>'
+
+    # Uncomment the following and set the absolute paths
+    # to the server SSL certificate and private key to
+    # enable HTTPS secure connections.
+    #
+    # var.ssl_certificate: path/to/server_ssl_cert.pem
+    # var.ssl_key: path/to/ssl_key.pem
+
 #---------------------------- Apache Tomcat Module ----------------------------
 - module: tomcat
   log: