Fix honeypots visualizer + merge develop_public (#596)

* Vulners#1257 (#2340) * vulners * vulners wrapper * docs * lesser variables * migrations * code quality * migration * code --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * bump 6.0.3 * bump 6.0.3 * updated docs * Bump django-ses from 4.0.0 to 4.1.0 in /requirements (#2342) Bumps [django-ses](https://github.com/django-ses/django-ses) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/django-ses/django-ses/releases) - [Changelog](https://github.com/django-ses/django-ses/blob/main/CHANGES.md) - [Commits](django-ses/django-ses@v4.0.0...v4.1.0) --- updated-dependencies: - dependency-name: django-ses dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * migrate (#2353) Co-authored-by: g4ze <bhaiyajionline@gmail.com> * incrementing uwsgi start-up period to due to migration time * adjusting doc + https nginx file * ailtyposquatting (#2341) * ailtyposquatting * restore a file that was deleted * fix * fix * changes * tests * no files * logs * files * variables * test * test * enum * tests * tests * dns_resolve * migration * a log :p --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * supported sh tld * bump * removed initialize.sh from start script * Fix phoneinfoga name Signed-off-by: 0ssigeno <s.berni@certego.net> * Start with --traefik/--traefik_local option. Closes #2305 (#2351) * add traefik config and options for dev and prod working config with traefik finish traefik config prod/dev add documentation * Update traefik_local.override.yml - remove comment * rework prod/local traefik and add deletion of get-docker.sh * split traefik compose into base, prod and local * remove print of compose files * parent c45c84a author David Mihajlovic <david.mihajlovic@protonmail.com> 1716908101 +0200 committer David Mihajlovic <david.mihajlovic@protonmail.com> 1717135119 +0200 add traefik config and options for dev and prod working config with traefik finish traefik config prod/dev add documentation Vulners#1257 (#2340) * vulners * vulners wrapper * docs * lesser variables * migrations * code quality * migration * code --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> bump 6.0.3 updated docs Bump django-ses from 4.0.0 to 4.1.0 in /requirements (#2342) Bumps [django-ses](https://github.com/django-ses/django-ses) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/django-ses/django-ses/releases) - [Changelog](https://github.com/django-ses/django-ses/blob/main/CHANGES.md) - [Commits](django-ses/django-ses@v4.0.0...v4.1.0) --- updated-dependencies: - dependency-name: django-ses dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> rework prod/local traefik and add deletion of get-docker.sh split traefik compose into base, prod and local get-docker.sh deletion without sudo change traefik compose naming * remove unnecessary files * remove print of compose files * change doc --------- Co-authored-by: Ubuntu <ubuntu@intelowldev.novalocal> * Fix url Signed-off-by: 0ssigeno <s.berni@certego.net> * Visualizer improvements (#2366) * table visualizer improvements * adjusted tests * prettier * changes * fixed start script * Split folder creation into two parts removing sudo (#2373) * Bump elasticsearch-dsl from 8.13.0 to 8.14.0 in /requirements (#2370) Bumps [elasticsearch-dsl](https://github.com/elasticsearch/elasticsearch-dsl-py) from 8.13.0 to 8.14.0. - [Release notes](https://github.com/elasticsearch/elasticsearch-dsl-py/releases) - [Changelog](https://github.com/elastic/elasticsearch-dsl-py/blob/main/Changelog.rst) - [Commits](elastic/elasticsearch-dsl-py@v8.13.0...v8.14.0) --- updated-dependencies: - dependency-name: elasticsearch-dsl dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump quark-engine from 24.5.1 to 24.6.1 in /requirements (#2371) Bumps [quark-engine](https://github.com/quark-engine/quark-engine) from 24.5.1 to 24.6.1. - [Release notes](https://github.com/quark-engine/quark-engine/releases) - [Commits](quark-engine/quark-engine@v24.5.1...v24.6.1) --- updated-dependencies: - dependency-name: quark-engine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Auto creation default test user with debug=true#1189 (#2369) * create super user * env files :p --------- Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: g4ze <bhaiyajionline@gmail.com> * Bump library/nginx from 1.26.0-alpine to 1.27.0-alpine in /docker (#2358) Bumps library/nginx from 1.26.0-alpine to 1.27.0-alpine. --- updated-dependencies: - dependency-name: library/nginx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump authlib from 1.3.0 to 1.3.1 in /requirements (#2368) Bumps [authlib](https://github.com/lepture/authlib) from 1.3.0 to 1.3.1. - [Release notes](https://github.com/lepture/authlib/releases) - [Changelog](https://github.com/lepture/authlib/blob/master/docs/changelog.rst) - [Commits](lepture/authlib@v1.3.0...v1.3.1) --- updated-dependencies: - dependency-name: authlib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * detect-it-easy analyzer, closes #1590 (#2354) * die * tweeks * codefactor * codefactor * ypo * gitignore * typo fix * detectiteasyyyyy * tests * supported files * msdos * logs, file support, soft t/o, poll * migrate * for all files * docker_based_true * params * tests debug[1] * Update api_app/analyzers_manager/migrations/0094_analyzer_config_detectiteasy.py * Update api_app/analyzers_manager/file_analyzers/detectiteasy.py --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> * Bi update (#2326) * added bi document * update bi interface * update bi interface * fix bi serializer * update certego-saas version * mign fix (#2375) Co-authored-by: g4ze <bhaiyajionline@gmail.com> * watchman adjusts test (#2349) * watchman adjusts test * watchman right version * test * adjust * right watchman version * Malprob analyzer, closes #1521 (#2357) * init updates works, weirdly new flow updates tests deepsrc * tests * disable_ratelimit(), t/o * timeout,reform response,TLP:CLEAR,logs,no raise,disableRatelimit * migrations * reponse format * t/o * t/o(agn) * api_key * ratelimit,migrations,healthcheck --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * fix columns * changes --------- Signed-off-by: 0ssigeno <s.berni@certego.net> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Shivam Purohit <shivampurohit900@gmail.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: Daniele Rosetti <d.rosetti@certego.net> Co-authored-by: 0ssigeno <s.berni@certego.net> Co-authored-by: Daniele Rosetti <55402684+drosetti@users.noreply.github.com> Co-authored-by: fgibertoni <152909479+fgibertoni@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Simone Berni <simone.berni2@studio.unibo.it> Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com> Co-authored-by: Shivam Purohit <shivampurohit900@gmail.com> Co-authored-by: Moon Patel <moonpatel2003@gmail.com> Co-authored-by: Cristina Ascari <95929371+cristinaascari@users.noreply.github.com> Co-authored-by: IP2Location <support@ip2location.com> Co-authored-by: suryapavan1611 <160897639+suryapavan1611@users.noreply.github.com> Co-authored-by: Your Name <you@example.com> Co-authored-by: Nilay Gupta <102874321+g4ze@users.noreply.github.com> Co-authored-by: g4ze <bhaiyajionline@gmail.com> Co-authored-by: David Mihajlovic <47985423+agnorance@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@intelowldev.novalocal>
intelowlproject · Jun 17, 2024 · 38ef87b · 38ef87b
1 parent 3a0d5bc
commit 38ef87b
Show file tree

Hide file tree

Showing 57 changed files with 2,035 additions and 195 deletions.
diff --git a/.github/CHANGELOG.md b/.github/CHANGELOG.md
@@ -2,7 +2,10 @@
 
 [**Upgrade Guide**](https://intelowl.readthedocs.io/en/latest/Installation.md#update-to-the-most-recent-version)
 
-## [v6.0.2](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.0.1)
+## [v6.0.4](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.0.4)
+Mostly adjusts and fixes with few new analyzers: Vulners and AILTypoSquatting Library.
+
+## [v6.0.2](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.0.2)
 Major fixes and adjustments. We improved the documentation to help the transition to the new major version.
 
 We added **Pivot** buttons to enable manual Pivoting from an Observable/File analysis to another. See [Doc](https://intelowl.readthedocs.io/en/latest/Usage.html#pivots) for more info

diff --git a/.github/workflows/pull_request_automation.yml b/.github/workflows/pull_request_automation.yml
@@ -87,7 +87,6 @@ jobs:
           BUILDKIT_PROGRESS: "plain"
           STAGE: "ci"
           REPO_DOWNLOADER_ENABLED: false
-          WATCHMAN: false
 
       - name: Startup script launch (Fast)
         if: "!contains(github.base_ref, 'master')"
@@ -98,7 +97,6 @@ jobs:
           BUILDKIT_PROGRESS: "plain"
           STAGE: "ci"
           REPO_DOWNLOADER_ENABLED: false
-          WATCHMAN: false
 
       - name: Docker debug
         if: always()

diff --git a/.gitignore b/.gitignore
@@ -50,3 +50,6 @@ coverage.xml
 *.cover
 .hypothesis/
 /.env
+
+# post run dev
+integrations/malware_tools_analyzers/clamav/sigs
diff --git a/api_app/analyzers_manager/file_analyzers/detectiteasy.py b/api_app/analyzers_manager/file_analyzers/detectiteasy.py
@@ -0,0 +1,60 @@
+import logging
+
+from api_app.analyzers_manager.classes import DockerBasedAnalyzer, FileAnalyzer
+from tests.mock_utils import MockUpResponse
+
+logger = logging.getLogger(__name__)
+
+
+class DetectItEasy(FileAnalyzer, DockerBasedAnalyzer):
+    name: str = "executable_analyzer"
+    url: str = "http://malware_tools_analyzers:4002/die"
+    # http request polling max number of tries
+    max_tries: int = 10
+    # interval between http request polling (in secs)
+    poll_distance: int = 1
+
+    def update(self):
+        pass
+
+    def run(self):
+        fname = str(self.filename).replace("/", "_").replace(" ", "_")
+        # get the file to send
+        binary = self.read_file_bytes()
+        args = [f"@{fname}", "--json"]
+        req_data = {
+            "args": args,
+        }
+        req_files = {fname: binary}
+        logger.info(
+            f"Running {self.analyzer_name} on {self.filename} with args: {args}"
+        )
+        report = self._docker_run(req_data, req_files, analyzer_name=self.analyzer_name)
+        if not report:
+            self.report.errors.append("DIE did not detect the file type")
+            return {}
+        return report
+
+    @staticmethod
+    def mocked_docker_analyzer_get(*args, **kwargs):
+        return MockUpResponse(
+            {
+                "report": {
+                    "arch": "NOEXEC",
+                    "mode": "Unknown",
+                    "type": "Unknown",
+                    "detects": [
+                        {
+                            "name": "Zip",
+                            "type": "archive",
+                            "string": "archive: Zip(2.0)[38.5%,1 file]",
+                            "options": "38.5%,1 file",
+                            "version": "2.0",
+                        }
+                    ],
+                    "filetype": "Binary",
+                    "endianess": "LE",
+                }
+            },
+            200,
+        )
diff --git a/api_app/analyzers_manager/file_analyzers/malprob.py b/api_app/analyzers_manager/file_analyzers/malprob.py
@@ -0,0 +1,79 @@
+import logging
+
+import requests
+
+from api_app.analyzers_manager.classes import FileAnalyzer
+from api_app.analyzers_manager.exceptions import AnalyzerRunException
+from tests.mock_utils import MockUpResponse, if_mock_connections, patch
+
+logger = logging.getLogger(__name__)
+
+
+class MalprobScan(FileAnalyzer):
+    url: str = "https://malprob.io/api"
+    private: bool = False
+    timeout: int = 60
+    _api_key_name: str
+
+    def update(self):
+        pass
+
+    def run(self):
+        file_name = str(self.filename).replace("/", "_").replace(" ", "_")
+        headers = {"Authorization": f"Token {self._api_key_name}"}
+        binary_file = self.read_file_bytes()
+
+        if self._job.tlp == self._job.TLP.CLEAR.value:
+            logger.info(f"uploading {file_name}:{self.md5} to MalProb.io for analysis")
+            scan = requests.post(
+                f"{self.url}/scan/",
+                files={"file": binary_file},
+                data={"name": file_name, "private": self.private},
+                headers=headers,
+                timeout=self.timeout,
+            )
+            scan.raise_for_status()
+            if scan.status_code == 204:
+                self.disable_for_rate_limit()
+                raise AnalyzerRunException("Limit reached for API")
+            elif scan.status_code == 302:
+                logger.info(
+                    f"status 302: file already exists | Rescanning the file: {self.md5}"
+                )
+            else:
+                return scan.json()
+
+        logger.info(f"rescanning {file_name} using {self.md5} on MalProb.io")
+        rescan = requests.post(
+            f"{self.url}/rescan/",
+            data={"hashcode": self.md5},
+            headers=headers,
+            timeout=self.timeout,
+        )
+        rescan.raise_for_status()
+        if rescan.status_code == 204:
+            self.disable_for_rate_limit()
+            raise AnalyzerRunException("Limit reached for API")
+        return rescan.json()
+
+    @classmethod
+    def _monkeypatch(cls):
+        patches = [
+            if_mock_connections(
+                patch(
+                    "requests.post",
+                    return_value=MockUpResponse(
+                        {
+                            "report": {
+                                "md5": "8a05a189e58ccd7275f7ffdf88c2c191",
+                                "sha1": "a7a70f2f482e6b26eedcf1781b277718078c743a",
+                                "sha256": """ac24043d48dadc390877a6151515565b1fdc1da
+                                b028ee2d95d80bd80085d9376""",
+                            },
+                        },
+                        200,
+                    ),
+                ),
+            )
+        ]
+        return super()._monkeypatch(patches=patches)