'Private_Passive_DNS' playbook + 'Quokka - Passive DNS' page (#598)

* Vulners#1257 (#2340) * vulners * vulners wrapper * docs * lesser variables * migrations * code quality * migration * code --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * bump 6.0.3 * bump 6.0.3 * updated docs * Bump django-ses from 4.0.0 to 4.1.0 in /requirements (#2342) Bumps [django-ses](https://github.com/django-ses/django-ses) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/django-ses/django-ses/releases) - [Changelog](https://github.com/django-ses/django-ses/blob/main/CHANGES.md) - [Commits](django-ses/django-ses@v4.0.0...v4.1.0) --- updated-dependencies: - dependency-name: django-ses dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * migrate (#2353) Co-authored-by: g4ze <bhaiyajionline@gmail.com> * incrementing uwsgi start-up period to due to migration time * adjusting doc + https nginx file * ailtyposquatting (#2341) * ailtyposquatting * restore a file that was deleted * fix * fix * changes * tests * no files * logs * files * variables * test * test * enum * tests * tests * dns_resolve * migration * a log :p --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * supported sh tld * bump * removed initialize.sh from start script * Fix phoneinfoga name Signed-off-by: 0ssigeno <s.berni@certego.net> * Start with --traefik/--traefik_local option. Closes #2305 (#2351) * add traefik config and options for dev and prod working config with traefik finish traefik config prod/dev add documentation * Update traefik_local.override.yml - remove comment * rework prod/local traefik and add deletion of get-docker.sh * split traefik compose into base, prod and local * remove print of compose files * parent c45c84a author David Mihajlovic <david.mihajlovic@protonmail.com> 1716908101 +0200 committer David Mihajlovic <david.mihajlovic@protonmail.com> 1717135119 +0200 add traefik config and options for dev and prod working config with traefik finish traefik config prod/dev add documentation Vulners#1257 (#2340) * vulners * vulners wrapper * docs * lesser variables * migrations * code quality * migration * code --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> bump 6.0.3 updated docs Bump django-ses from 4.0.0 to 4.1.0 in /requirements (#2342) Bumps [django-ses](https://github.com/django-ses/django-ses) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/django-ses/django-ses/releases) - [Changelog](https://github.com/django-ses/django-ses/blob/main/CHANGES.md) - [Commits](django-ses/django-ses@v4.0.0...v4.1.0) --- updated-dependencies: - dependency-name: django-ses dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> rework prod/local traefik and add deletion of get-docker.sh split traefik compose into base, prod and local get-docker.sh deletion without sudo change traefik compose naming * remove unnecessary files * remove print of compose files * change doc --------- Co-authored-by: Ubuntu <ubuntu@intelowldev.novalocal> * Fix url Signed-off-by: 0ssigeno <s.berni@certego.net> * Visualizer improvements (#2366) * table visualizer improvements * adjusted tests * prettier * changes * fixed start script * Split folder creation into two parts removing sudo (#2373) * Bump elasticsearch-dsl from 8.13.0 to 8.14.0 in /requirements (#2370) Bumps [elasticsearch-dsl](https://github.com/elasticsearch/elasticsearch-dsl-py) from 8.13.0 to 8.14.0. - [Release notes](https://github.com/elasticsearch/elasticsearch-dsl-py/releases) - [Changelog](https://github.com/elastic/elasticsearch-dsl-py/blob/main/Changelog.rst) - [Commits](elastic/elasticsearch-dsl-py@v8.13.0...v8.14.0) --- updated-dependencies: - dependency-name: elasticsearch-dsl dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump quark-engine from 24.5.1 to 24.6.1 in /requirements (#2371) Bumps [quark-engine](https://github.com/quark-engine/quark-engine) from 24.5.1 to 24.6.1. - [Release notes](https://github.com/quark-engine/quark-engine/releases) - [Commits](quark-engine/quark-engine@v24.5.1...v24.6.1) --- updated-dependencies: - dependency-name: quark-engine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Auto creation default test user with debug=true#1189 (#2369) * create super user * env files :p --------- Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: g4ze <bhaiyajionline@gmail.com> * Bump library/nginx from 1.26.0-alpine to 1.27.0-alpine in /docker (#2358) Bumps library/nginx from 1.26.0-alpine to 1.27.0-alpine. --- updated-dependencies: - dependency-name: library/nginx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump authlib from 1.3.0 to 1.3.1 in /requirements (#2368) Bumps [authlib](https://github.com/lepture/authlib) from 1.3.0 to 1.3.1. - [Release notes](https://github.com/lepture/authlib/releases) - [Changelog](https://github.com/lepture/authlib/blob/master/docs/changelog.rst) - [Commits](lepture/authlib@v1.3.0...v1.3.1) --- updated-dependencies: - dependency-name: authlib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * detect-it-easy analyzer, closes #1590 (#2354) * die * tweeks * codefactor * codefactor * ypo * gitignore * typo fix * detectiteasyyyyy * tests * supported files * msdos * logs, file support, soft t/o, poll * migrate * for all files * docker_based_true * params * tests debug[1] * Update api_app/analyzers_manager/migrations/0094_analyzer_config_detectiteasy.py * Update api_app/analyzers_manager/file_analyzers/detectiteasy.py --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> * Bi update (#2326) * added bi document * update bi interface * update bi interface * fix bi serializer * update certego-saas version * mign fix (#2375) Co-authored-by: g4ze <bhaiyajionline@gmail.com> * watchman adjusts test (#2349) * watchman adjusts test * watchman right version * test * adjust * right watchman version * Malprob analyzer, closes #1521 (#2357) * init updates works, weirdly new flow updates tests deepsrc * tests * disable_ratelimit(), t/o * timeout,reform response,TLP:CLEAR,logs,no raise,disableRatelimit * migrations * reponse format * t/o * t/o(agn) * api_key * ratelimit,migrations,healthcheck --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * Passive_DNS playbook and visualizer (#2374) * created 'passive_dns' playbook and visualizer * dnsdb * validin * changes * refactor * changes * refactor + tests * changes * changes * added private_passive_dns playbook and visualizer * tests * changes * changes * added test --------- Signed-off-by: 0ssigeno <s.berni@certego.net> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Shivam Purohit <shivampurohit900@gmail.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: Daniele Rosetti <d.rosetti@certego.net> Co-authored-by: 0ssigeno <s.berni@certego.net> Co-authored-by: Daniele Rosetti <55402684+drosetti@users.noreply.github.com> Co-authored-by: fgibertoni <152909479+fgibertoni@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Simone Berni <simone.berni2@studio.unibo.it> Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com> Co-authored-by: Shivam Purohit <shivampurohit900@gmail.com> Co-authored-by: Moon Patel <moonpatel2003@gmail.com> Co-authored-by: Cristina Ascari <95929371+cristinaascari@users.noreply.github.com> Co-authored-by: IP2Location <support@ip2location.com> Co-authored-by: suryapavan1611 <160897639+suryapavan1611@users.noreply.github.com> Co-authored-by: Your Name <you@example.com> Co-authored-by: Nilay Gupta <102874321+g4ze@users.noreply.github.com> Co-authored-by: g4ze <bhaiyajionline@gmail.com> Co-authored-by: David Mihajlovic <47985423+agnorance@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@intelowldev.novalocal>
intelowlproject · Jun 27, 2024 · c309036 · c309036
1 parent d09b1d8
commit c309036
Show file tree

Hide file tree

Showing 15 changed files with 2,521 additions and 215 deletions.
diff --git a/api_app/playbooks_manager/migrations/0045_playbook_config_passive_dns.py b/api_app/playbooks_manager/migrations/0045_playbook_config_passive_dns.py
@@ -0,0 +1,125 @@
+from django.db import migrations
+from django.db.models.fields.related_descriptors import (
+    ForwardManyToOneDescriptor,
+    ForwardOneToOneDescriptor,
+    ManyToManyDescriptor,
+)
+
+plugin = {
+    "analyzers": [
+        "CIRCLPassiveDNS",
+        "DNSDB",
+        "Mnemonic_PassiveDNS",
+        "OTXQuery",
+        "Robtex",
+        "Threatminer",
+        "Validin",
+    ],
+    "connectors": [],
+    "pivots": [],
+    "for_organization": False,
+    "name": "Passive_DNS",
+    "description": "A playbook that retrieve information from Passive DNS",
+    "disabled": False,
+    "type": ["ip", "url", "domain"],
+    "runtime_configuration": {
+        "pivots": {},
+        "analyzers": {},
+        "connectors": {},
+        "visualizers": {},
+    },
+    "scan_mode": 2,
+    "scan_check_time": "1 00:00:00",
+    "tlp": "AMBER",
+    "starting": True,
+    "owner": None,
+    "tags": [],
+    "model": "playbooks_manager.PlaybookConfig",
+}
+
+params = []
+
+values = []
+
+
+def _get_real_obj(Model, field, value):
+    def _get_obj(Model, other_model, value):
+        if isinstance(value, dict):
+            real_vals = {}
+            for key, real_val in value.items():
+                real_vals[key] = _get_real_obj(other_model, key, real_val)
+            value = other_model.objects.get_or_create(**real_vals)[0]
+        # it is just the primary key serialized
+        else:
+            if isinstance(value, int):
+                if Model.__name__ == "PluginConfig":
+                    value = other_model.objects.get(name=plugin["name"])
+                else:
+                    value = other_model.objects.get(pk=value)
+            else:
+                value = other_model.objects.get(name=value)
+        return value
+
+    if (
+        type(getattr(Model, field))
+        in [ForwardManyToOneDescriptor, ForwardOneToOneDescriptor]
+        and value
+    ):
+        other_model = getattr(Model, field).get_queryset().model
+        value = _get_obj(Model, other_model, value)
+    elif type(getattr(Model, field)) in [ManyToManyDescriptor] and value:
+        other_model = getattr(Model, field).rel.model
+        value = [_get_obj(Model, other_model, val) for val in value]
+    return value
+
+
+def _create_object(Model, data):
+    mtm, no_mtm = {}, {}
+    for field, value in data.items():
+        value = _get_real_obj(Model, field, value)
+        if type(getattr(Model, field)) is ManyToManyDescriptor:
+            mtm[field] = value
+        else:
+            no_mtm[field] = value
+    try:
+        o = Model.objects.get(**no_mtm)
+    except Model.DoesNotExist:
+        o = Model(**no_mtm)
+        o.full_clean()
+        o.save()
+        for field, value in mtm.items():
+            attribute = getattr(o, field)
+            if value is not None:
+                attribute.set(value)
+        return False
+    return True
+
+
+def migrate(apps, schema_editor):
+    Parameter = apps.get_model("api_app", "Parameter")
+    PluginConfig = apps.get_model("api_app", "PluginConfig")
+    python_path = plugin.pop("model")
+    Model = apps.get_model(*python_path.split("."))
+    if not Model.objects.filter(name=plugin["name"]).exists():
+        exists = _create_object(Model, plugin)
+        if not exists:
+            for param in params:
+                _create_object(Parameter, param)
+            for value in values:
+                _create_object(PluginConfig, value)
+
+
+def reverse_migrate(apps, schema_editor):
+    python_path = plugin.pop("model")
+    Model = apps.get_model(*python_path.split("."))
+    Model.objects.get(name=plugin["name"]).delete()
+
+
+class Migration(migrations.Migration):
+    atomic = False
+    dependencies = [
+        ("api_app", "0062_alter_parameter_python_module"),
+        ("playbooks_manager", "0044_add_cycat_to_free_to_use"),
+    ]
+
+    operations = [migrations.RunPython(migrate, reverse_migrate)]
diff --git a/api_app/playbooks_manager/migrations/0046_playbook_config_dns_intel.py b/api_app/playbooks_manager/migrations/0046_playbook_config_dns_intel.py
@@ -0,0 +1,124 @@
+from django.db import migrations
+from django.db.models.fields.related_descriptors import (
+    ForwardManyToOneDescriptor,
+    ForwardOneToOneDescriptor,
+    ManyToManyDescriptor,
+)
+
+plugin = {
+    "analyzers": [
+        "DNS0_dns",
+        "DNSDB_SIE",
+        "Quokka_PDNS",
+        "Quokka_PDNS_Wildcard_Left",
+        "Quokka_RDNS",
+        "Quokka_RDNS_Names",
+        "Quokka_RDNS_Network",
+    ],
+    "connectors": [],
+    "pivots": [],
+    "for_organization": False,
+    "name": "DNS_Intel",
+    "description": "A playbook that retrieve information from Quokka Passive DNS and DNS0",  # noqa E501
+    "disabled": False,
+    "type": ["ip", "url", "domain"],
+    "runtime_configuration": {
+        "pivots": {},
+        "analyzers": {},
+        "connectors": {},
+        "visualizers": {},
+    },
+    "scan_mode": 2,
+    "scan_check_time": "1 00:00:00",
+    "tlp": "RED",
+    "starting": True,
+    "owner": None,
+    "tags": [],
+    "model": "playbooks_manager.PlaybookConfig",
+}
+
+params = []
+
+values = []
+
+
+def _get_real_obj(Model, field, value):
+    def _get_obj(Model, other_model, value):
+        if isinstance(value, dict):
+            real_vals = {}
+            for key, real_val in value.items():
+                real_vals[key] = _get_real_obj(other_model, key, real_val)
+            value = other_model.objects.get_or_create(**real_vals)[0]
+        # it is just the primary key serialized
+        else:
+            if isinstance(value, int):
+                if Model.__name__ == "PluginConfig":
+                    value = other_model.objects.get(name=plugin["name"])
+                else:
+                    value = other_model.objects.get(pk=value)
+            else:
+                value = other_model.objects.get(name=value)
+        return value
+
+    if (
+        type(getattr(Model, field))
+        in [ForwardManyToOneDescriptor, ForwardOneToOneDescriptor]
+        and value
+    ):
+        other_model = getattr(Model, field).get_queryset().model
+        value = _get_obj(Model, other_model, value)
+    elif type(getattr(Model, field)) in [ManyToManyDescriptor] and value:
+        other_model = getattr(Model, field).rel.model
+        value = [_get_obj(Model, other_model, val) for val in value]
+    return value
+
+
+def _create_object(Model, data):
+    mtm, no_mtm = {}, {}
+    for field, value in data.items():
+        value = _get_real_obj(Model, field, value)
+        if type(getattr(Model, field)) is ManyToManyDescriptor:
+            mtm[field] = value
+        else:
+            no_mtm[field] = value
+    try:
+        o = Model.objects.get(**no_mtm)
+    except Model.DoesNotExist:
+        o = Model(**no_mtm)
+        o.full_clean()
+        o.save()
+        for field, value in mtm.items():
+            attribute = getattr(o, field)
+            if value is not None:
+                attribute.set(value)
+        return False
+    return True
+
+
+def migrate(apps, schema_editor):
+    Parameter = apps.get_model("api_app", "Parameter")
+    PluginConfig = apps.get_model("api_app", "PluginConfig")
+    python_path = plugin.pop("model")
+    Model = apps.get_model(*python_path.split("."))
+    if not Model.objects.filter(name=plugin["name"]).exists():
+        exists = _create_object(Model, plugin)
+        if not exists:
+            for param in params:
+                _create_object(Parameter, param)
+            for value in values:
+                _create_object(PluginConfig, value)
+
+
+def reverse_migrate(apps, schema_editor):
+    python_path = plugin.pop("model")
+    Model = apps.get_model(*python_path.split("."))
+    Model.objects.get(name=plugin["name"]).delete()
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("api_app", "0062_alter_parameter_python_module"),
+        ("playbooks_manager", "0045_playbook_config_passive_dns"),
+    ]
+
+    operations = [migrations.RunPython(migrate, reverse_migrate)]
diff --git a/api_app/visualizers_manager/migrations/0038_0002_visualizer_config_passive_dns.py b/api_app/visualizers_manager/migrations/0038_0002_visualizer_config_passive_dns.py
@@ -0,0 +1,110 @@
+from django.db import migrations
+from django.db.models.fields.related_descriptors import (
+    ForwardManyToOneDescriptor,
+    ForwardOneToOneDescriptor,
+    ManyToManyDescriptor,
+)
+
+plugin = {
+    "python_module": {
+        "health_check_schedule": None,
+        "update_schedule": None,
+        "module": "passive_dns.visualizer.PassiveDNS",
+        "base_path": "api_app.visualizers_manager.visualizers",
+    },
+    "playbooks": ["Passive_DNS"],
+    "name": "Passive_DNS",
+    "description": "Visualize Passive DNS informations",
+    "disabled": False,
+    "soft_time_limit": 60,
+    "routing_key": "default",
+    "health_check_status": True,
+    "model": "visualizers_manager.VisualizerConfig",
+}
+
+params = []
+
+values = []
+
+
+def _get_real_obj(Model, field, value):
+    def _get_obj(Model, other_model, value):
+        if isinstance(value, dict):
+            real_vals = {}
+            for key, real_val in value.items():
+                real_vals[key] = _get_real_obj(other_model, key, real_val)
+            value = other_model.objects.get_or_create(**real_vals)[0]
+        # it is just the primary key serialized
+        else:
+            if isinstance(value, int):
+                if Model.__name__ == "PluginConfig":
+                    value = other_model.objects.get(name=plugin["name"])
+                else:
+                    value = other_model.objects.get(pk=value)
+            else:
+                value = other_model.objects.get(name=value)
+        return value
+
+    if (
+        type(getattr(Model, field))
+        in [ForwardManyToOneDescriptor, ForwardOneToOneDescriptor]
+        and value
+    ):
+        other_model = getattr(Model, field).get_queryset().model
+        value = _get_obj(Model, other_model, value)
+    elif type(getattr(Model, field)) in [ManyToManyDescriptor] and value:
+        other_model = getattr(Model, field).rel.model
+        value = [_get_obj(Model, other_model, val) for val in value]
+    return value
+
+
+def _create_object(Model, data):
+    mtm, no_mtm = {}, {}
+    for field, value in data.items():
+        value = _get_real_obj(Model, field, value)
+        if type(getattr(Model, field)) is ManyToManyDescriptor:
+            mtm[field] = value
+        else:
+            no_mtm[field] = value
+    try:
+        o = Model.objects.get(**no_mtm)
+    except Model.DoesNotExist:
+        o = Model(**no_mtm)
+        o.full_clean()
+        o.save()
+        for field, value in mtm.items():
+            attribute = getattr(o, field)
+            if value is not None:
+                attribute.set(value)
+        return False
+    return True
+
+
+def migrate(apps, schema_editor):
+    Parameter = apps.get_model("api_app", "Parameter")
+    PluginConfig = apps.get_model("api_app", "PluginConfig")
+    python_path = plugin.pop("model")
+    Model = apps.get_model(*python_path.split("."))
+    if not Model.objects.filter(name=plugin["name"]).exists():
+        exists = _create_object(Model, plugin)
+        if not exists:
+            for param in params:
+                _create_object(Parameter, param)
+            for value in values:
+                _create_object(PluginConfig, value)
+
+
+def reverse_migrate(apps, schema_editor):
+    python_path = plugin.pop("model")
+    Model = apps.get_model(*python_path.split("."))
+    Model.objects.get(name=plugin["name"]).delete()
+
+
+class Migration(migrations.Migration):
+    atomic = False
+    dependencies = [
+        ("api_app", "0062_alter_parameter_python_module"),
+        ("visualizers_manager", "0038_0001_visualizer_config_threathunting"),
+    ]
+
+    operations = [migrations.RunPython(migrate, reverse_migrate)]