[FLASHPOINT] Deprecate malware and APT mport options

external-import/flashpoint/README.md

@@ -0,0 +1,203 @@
+# OpenCTI Flashpoint Connector
+
+This connector integrates Flashpoint with the OpenCTI platform. It pulls various threat intelligence data from Flashpoint and imports it into OpenCTI, providing enhanced visibility into security threats.
+
+See [Flashpoint API Documentation](https://flashpoint.io/resources/datasheets/ignite-platform-datasheet/) for more details.
+
+## Table of Contents
+
+- [Introduction](#introduction)
+- [Installation](#installation)
+    - [Requirements](#requirements)
+- [Configuration variables](#configuration-variables)
+    - [OpenCTI environment variables](#opencti-environment-variables)
+    - [Base connector environment variables](#base-connector-environment-variables)
+    - [Connector extra parameters environment variables](#connector-extra-parameters-environment-variables)
+- [Deployment](#deployment)
+    - [Docker Deployment](#docker-deployment)
+    - [Manual Deployment](#manual-deployment)
+- [Usage](#usage)
+- [Behavior](#behavior)
+- [Debugging](#debugging)
+- [Additional information](#additional-information)
+
+## Introduction
+
+The Flashpoint Connector for OpenCTI imports threat intelligence data from Flashpoint and maps it to relevant entities in the OpenCTI ecosystem. This includes alerts, indicators, malware reports, vulnerabilities, and other intelligence, enhancing threat detection and response capabilities.
+
+## Installation
+
+### Requirements
+
+- OpenCTI Platform >= 6.x
+- Flashpoint API access (API key required)
+
+## Configuration variables
+
+The configuration variables for the connector can be set in `docker-compose.yml` for Docker deployments or `config.yml` for manual deployments.
+
+### OpenCTI environment variables
+
+| Parameter     | config.yml | Docker environment variable | Mandatory | Description                                          |
+|---------------|------------|-----------------------------|-----------|------------------------------------------------------|
+| OpenCTI URL   | url        | `OPENCTI_URL`               | Yes       | The URL of the OpenCTI platform.                     |
+| OpenCTI Token | token      | `OPENCTI_TOKEN`             | Yes       | The default admin token set in the OpenCTI platform. |
+
+### Base connector environment variables
+
+| Parameter       | config.yml | Docker environment variable | Default         | Mandatory | Description                                                                      |
+|-----------------|------------|-----------------------------|-----------------|-----------|----------------------------------------------------------------------------------|
+| Connector ID    | id         | `CONNECTOR_ID`              | /               | Yes       | A unique `UUIDv4` identifier for this connector instance.                        |
+| Connector Type  | type       | `CONNECTOR_TYPE`            | EXTERNAL_IMPORT | Yes       | Should always be set to `EXTERNAL_IMPORT` for this connector.                    |
+| Connector Name  | name       | `CONNECTOR_NAME`            |                 | Yes       | Name of the connector.                                                           |
+| Connector Scope | scope      | `CONNECTOR_SCOPE`           | flashpoint      | Yes       | The scope or type of data the connector is importing (e.g., flashpoint).        |
+| Log Level       | log_level  | `CONNECTOR_LOG_LEVEL`       | info            | Yes       | Determines the verbosity of the logs. Options: `debug`, `info`, `warn`, `error`. |
+
+### Connector extra parameters environment variables
+
+| Parameter                       | config.yml                     | Docker environment variable          | Default       | Mandatory | Description                                               |
+|---------------------------------|--------------------------------|-------------------------------------|---------------|-----------|-----------------------------------------------------------|
+| API base URL                    | api_base_url                   | `FLASHPOINT_API_BASE_URL`           | https://api.flashpoint.io | Yes       | Base URL for the Flashpoint API.                          |
+| API access key                  | api_access_key                 | `FLASHPOINT_API_KEY`                 |               | Yes       | Flashpoint API access key.                                |
+| Import interval (minutes)       | interval                       | `FLASHPOINT_INTERVAL`                | 5             | No        | Interval (in minutes) to import data from Flashpoint.     |
+| Import start date               | import_start_date              | `FLASHPOINT_IMPORT_START_DATE`       |               | No        | The date from which to start importing data.              |
+| Import reports                  | import_reports                 | `FLASHPOINT_IMPORT_REPORTS`          | true          | No        | Import reports from Flashpoint.                           |
+| Import indicators                | import_indicators              | `FLASHPOINT_IMPORT_INDICATORS`       | true          | No        | Import indicators of compromise (IoCs).                   |
+| Import communities               | import_communities             | `FLASHPOINT_IMPORT_COMMUNITIES`      | false         | No        | Import community data.                                    |
+| Communities queries              | communities_queries            | `FLASHPOINT_COMMUNITIES_QUERIES`     | ""            | No        | Comma-separated list of community queries to execute.     |
+| Import alerts                   | import_alerts                 | `FLASHPOINT_IMPORT_ALERTS`           | true          | No        | Import alert data from Flashpoint.                        |
+| Indicators in reports           | indicators_in_reports          | `FLASHPOINT_INDICATORS_IN_REPORTS`   | false         | No        | Include indicators in the reports imported from MispFeed. |
+
+## Deployment
+
+### Docker Deployment
+
+Before building the Docker container, ensure you have set the version of `pycti` in `requirements.txt` to match the version of OpenCTI you are running. For example, `pycti==6.3.8`. Failing to do so may result in initialization issues.
+
+Build a Docker Image using the provided `Dockerfile`.
+
+Example:
+
+```shell
+# Replace the IMAGE NAME with the appropriate value
+docker build . -t [IMAGE NAME]:latest
+```
+
+Make sure to replace the environment variables in docker-compose.yml with the appropriate configurations for your environment. Then, start the docker container with the provided docker-compose.yml.
+
+```shell
+docker compose up -d
+```
+
+### Manual Deployment
+Create a file config.yml based on the provided config.yml.sample.
+
+Replace the configuration variables (especially the "ChangeMe" variables) with the appropriate configurations for your environment.
+
+Install the required Python dependencies (preferably in a virtual environment):
+
+```shell
+pip install -r requirements.txt
+```
+
+Then, start the connector from the /src directory:
+```shell
+python flashpoint.py
+```
+
+## Usage
+After installation, the connector requires minimal interaction and should update automatically at a regular interval specified in your docker-compose.yml or config.yml under FLASHPOINT_INTERVAL.
+
+To force an immediate download of a new batch of entities, navigate to:
+
+Data management -> Ingestion -> Connectors in the OpenCTI platform.
+
+Find the connector and click on the refresh button to reset the connector's state and force a new download of data by re-running the connector.
+
+## Behavior
+
+### Mapping details
+
+Reports
+```mermaid
+graph LR
+  subgraph Flashpoint
+    direction TB
+    FlashpointReport[Reports]
+
+  end
+
+  subgraph OpenCTI
+    direction LR
+    OpenCTIReport[STIX Report]
+    OpenCTIThreatActor[STIX Threat Actor]
+    OpenCTIIntrusionSet[STIX Intrusion Set]
+    OpenCTIMalware[STIX Malware]
+    OpenCTIAttackPattern[STIX Attack Pattern]
+    OpenCTICountry[STIX Country]
+    OpenCTIRegion[STIX Region]
+    OpenCTISector[STIX Sector]
+    OpenCTITool[STIX Tool]
+    OpenCTIExternalReference[STIX External Reference]
+  end
+
+%% Relationships
+  FlashpointReport --> OpenCTIReport --> OpenCTIExternalReference
+
+
+%% Relationships to reports
+  OpenCTIReport -- references --> OpenCTIAttackPattern
+  OpenCTIAttackPattern -- uses --> OpenCTIIntrusionSet & OpenCTIThreatActor & OpenCTIMalware
+  OpenCTIIntrusionSet & OpenCTIThreatActor -- uses --> OpenCTIMalware
+  OpenCTIIntrusionSet & OpenCTIThreatActor -- uses --> OpenCTITool
+  OpenCTIIntrusionSet & OpenCTIThreatActor & OpenCTIMalware -- targets --> OpenCTICountry & OpenCTIRegion & OpenCTISector
+
+```
+
+Communities
+
+```mermaid
+graph LR
+    subgraph Flashpoint
+        direction TB
+        FlashpointCommunity[Communities]
+    end
+
+    subgraph OpenCTI
+        direction LR
+        OpenCTIChannel[Custom Channel]
+        OpenCTIMediaContent[Custom Media Content]
+        OpenCTIExternalReference[STIX External Reference]
+    end
+
+    %% Relationships
+    FlashpointCommunity --> OpenCTIChannel --> OpenCTIExternalReference
+
+    %% Relationships to communities
+    OpenCTIChannel -- publishes --> OpenCTIMediaContent
+```
+
+Alerts
+```mermaid
+graph LR
+    subgraph Flashpoint
+        direction TB
+        FlashpointAlert[Alerts]
+    end
+
+    subgraph OpenCTI
+        direction LR
+        OpenCTIIncident[STIX Incident]
+        OpenCTIChannel[Custom  Channel]
+        OpenCTIMediaContent[Custom Media Content]
+        OpenCTIExternalReference[STIX External Reference]
+    end
+
+    %% Relationships
+    FlashpointAlert --> OpenCTIIncident --> OpenCTIExternalReference
+
+    %% Relationships to incidents
+    OpenCTIIncident -- uses --> OpenCTIChannel
+    OpenCTIChannel -- publishes --> OpenCTIMediaContent
+    OpenCTIMediaContent -- related-to --> OpenCTIIncident
+```

external-import/flashpoint/docker-compose.yml

@@ -12,11 +12,8 @@ services:
       - CONNECTOR_LOG_LEVEL=error
       - FLASHPOINT_API_KEY=ChangeMe
       - FLASHPOINT_IMPORT_START_DATE=2024-01-01
-      - FLASHPOINT_IMPORT_APT=false
-      - FLASHPOINT_IMPORT_MALWARE=false
       - FLASHPOINT_IMPORT_REPORTS=true
       - FLASHPOINT_IMPORT_INDICATORS=true
-      - FLASHPOINT_IMPORT_VULNERABILITIES=true
       - FLASHPOINT_IMPORT_COMMUNITIES=false
       - FLASHPOINT_IMPORT_ALERTS=true
       - FLASHPOINT_COMMUNITIES_QUERIES=cybersecurity,cyberattack

external-import/flashpoint/src/config.yml.sample

@@ -13,11 +13,8 @@ connector:
 flashpoint:
   api_key: 'ChangeMe'
   import_start_date: '2024-01-01'
-  import_apt: false
-  import_malware: false
   import_reports: true
   import_indicators: true
-  import_vulnerabilities: true
   import_communities: false
   communities_queries: 'cybersecurity,cyberattack'
   import_alerts: true

external-import/flashpoint/src/flashpoint.py

@@ -5,6 +5,7 @@
 import os
 import sys
 import time
+import warnings
 
 import html2text
 import pytz
@@ -84,12 +85,6 @@ def __init__(self):
             config,
             default=True,
         )
-        self.flashpoint_import_vulnerabilities = get_config_variable(
-            "FLASHPOINT_IMPORT_VULNERABILITIES",
-            ["flashpoint", "import_vulnerabilities"],
-            config,
-            default=True,
-        )
         self.flashpoint_import_communities = get_config_variable(
             "FLASHPOINT_IMPORT_COMMUNITIES",
             ["flashpoint", "import_communities"],
@@ -356,68 +351,30 @@ def _convert_to_markdown(self, content):
 
     def _import_apt(self, work_id):
         # Query params
-        url = self.flashpoint_api_app_url + "/documents/apt/wiki"
-        headers = {
-            "Content-Type": "application/json",
-            "Authorization": "Bearer " + self.flashpoint_api_key,
-        }
-        response = requests.get(url, headers=headers)
-        data = json.loads(response.content)
-        objects = []
-        try:
-            if "data" in data:
-                for apt in data["data"]:
-                    intrusion_set_stix = stix2.IntrusionSet(
-                        id=IntrusionSet.generate_id(apt["apt_group"]),
-                        name=apt["apt_group"],
-                        aliases=apt["aliases"],
-                        description=self._convert_to_markdown(apt["body"]["raw"]),
-                        created_by_ref=self.identity["standard_id"],
-                        object_marking_refs=[stix2.TLP_GREEN.get("id")],
-                    )
-                    objects.append(intrusion_set_stix)
-            self.helper.send_stix2_bundle(
-                stix2.Bundle(
-                    objects=objects,
-                    allow_custom=True,
-                ).serialize(),
-                work_id=work_id,
-            )
-        except Exception as e:
-            self.helper.log_error(str(e))
+        msg = (
+            "/document/apt/wiki has been deprecated by Flashpoint Ignite, the option import apt does not "
+            "work anymore and will be removed in a future version."
+        )
+        warnings.warn(
+            message=msg,
+            category=DeprecationWarning,
+            stacklevel=2,
+        )
+        self.helper.connector_logger.warning(msg)  # warns connector user
+        self.helper.api.work.to_processed(work_id, msg)  # warns OpenCTI user
 
     def _import_malware(self, work_id):
-        # Query params
-        url = self.flashpoint_api_app_url + "/documents/malware/wiki"
-        headers = {
-            "Content-Type": "application/json",
-            "Authorization": "Bearer " + self.flashpoint_api_key,
-        }
-        response = requests.get(url, headers=headers)
-        data = json.loads(response.content)
-        objects = []
-        try:
-            if "data" in data:
-                for malware in data["data"]:
-                    malware_stix = stix2.Malware(
-                        id=Malware.generate_id(malware["malware_family_name"]),
-                        name=malware["malware_family_name"],
-                        is_family=True,
-                        aliases=malware["aliases"],
-                        description=self._convert_to_markdown(malware["body"]["raw"]),
-                        created_by_ref=self.identity["standard_id"],
-                        object_marking_refs=[stix2.TLP_AMBER.get("id")],
-                    )
-                    objects.append(malware_stix)
-            self.helper.send_stix2_bundle(
-                stix2.Bundle(
-                    objects=objects,
-                    allow_custom=True,
-                ).serialize(),
-                work_id=work_id,
-            )
-        except Exception as e:
-            self.helper.log_error(str(e))
+        msg = (
+            "/document/malware/wiki has been deprecated by Flashpoint Ignite, the option import malware does not "
+            "work anymore and will be removed in a future version."
+        )
+        warnings.warn(
+            message=msg,
+            category=DeprecationWarning,
+            stacklevel=2,
+        )
+        self.helper.connector_logger.warning(msg)  # warns connector user
+        self.helper.api.work.to_processed(work_id, msg)  # warns OpenCTI user
 
     def _import_reports(self, work_id, start_date):
         # Query params