-
-
Notifications
You must be signed in to change notification settings - Fork 440
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Malware bazaar ingestor #2259
Malware bazaar ingestor #2259
Conversation
fixed json serialization for types: bytes and File
in this way you can set an health checker if you want
…estor # Conflicts: # api_app/serializers/job.py
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some food for thought. View full project report here.
api_app/classes.py
Outdated
if all(isinstance(n, bytes) for n in report_content): | ||
report_content = [ | ||
base64.b64encode(b).decode("utf-8") for b in report_content | ||
] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
elif
we do not want to check the content 2 times
api_app/classes.py
Outdated
base64.b64encode(b).decode("utf-8") for b in report_content | ||
] | ||
|
||
self.content = content |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not necessary
api_app/ingestors_manager/classes.py
Outdated
def after_run_success(self, content): | ||
super().after_run_success(content) | ||
self._config: IngestorConfig | ||
# exhaust generator | ||
deque( | ||
self._config.create_jobs( | ||
# every job created from an ingestor | ||
content, | ||
self.content, | ||
TLP.CLEAR.value, | ||
self._user, | ||
self._config.delay, | ||
), | ||
maxlen=0, | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
def after_run_success(self, content):
pre_parsing_content = content # <------------
super().after_run_success(content)
self._config: IngestorConfig
deque(
self._config.create_jobs(
# every job created from an ingestor
pre_parsing_content, # <-----------
TLP.CLEAR.value,
self._user,
self._config.delay,
),
maxlen=0,
)
```
|
||
class MalwareBazaar(Ingestor): | ||
url: str | ||
hours: int |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add as comment the same description that you added in the Parameter
model?
Just so that if someone only reads this code, understand what is this parameter for
|
||
plugin = {'id': 2, 'python_module': {'health_check_schedule': None, 'update_schedule': {'minute': '0', 'hour': '*', 'day_of_week': '*', 'day_of_month': '*', 'month_of_year': '*'}, 'module': 'malware_bazaar.MalwareBazaar', 'base_path': 'api_app.ingestors_manager.ingestors'}, 'schedule': {'minute': '0', 'hour': '*', 'day_of_week': '*', 'day_of_month': '*', 'month_of_year': '*'}, 'periodic_task': {'crontab': {'minute': '0', 'hour': '*', 'day_of_week': '*', 'day_of_month': '*', 'month_of_year': '*'}, 'name': 'MalwarebazaarIngestor', 'task': 'intel_owl.tasks.execute_ingestor', 'kwargs': '{"config_name": "MalwareBazaar"}', 'queue': 'default', 'enabled': True}, 'user': {'username': 'MalwarebazaarIngestor', 'first_name': '', 'last_name': '', 'email': ''}, 'name': 'MalwareBazaar', 'description': 'MalwareBazaar ingestor', 'disabled': False, 'soft_time_limit': 60, 'routing_key': 'default', 'health_check_status': True, 'maximum_jobs': 30, 'delay': '00:00:30', 'health_check_task': None, 'playbook_to_execute': 2, 'model': 'ingestors_manager.IngestorConfig'} | ||
|
||
params = [{'python_module': {'module': 'malware_bazaar.MalwareBazaar', 'base_path': 'api_app.ingestors_manager.ingestors'}, 'name': 'url', 'type': 'str', 'description': 'API endpoint', 'is_secret': False, 'required': True}, {'python_module': {'module': 'malware_bazaar.MalwareBazaar', 'base_path': 'api_app.ingestors_manager.ingestors'}, 'name': 'hours', 'type': 'int', 'description': 'Download samples that are up to X hours old', 'is_secret': False, 'required': True}, {'python_module': {'module': 'malware_bazaar.MalwareBazaar', 'base_path': 'api_app.ingestors_manager.ingestors'}, 'name': 'signatures', 'type': 'list', 'description': 'Download samples from chosen signatures (aka malware families)', 'is_secret': True, 'required': True}] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does it make sense that signatures
is a secret?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe you dont't want to revealt that information, dunno. Anyway I changed that as NOT secret parameter
class MalwareBazaar(Ingestor): | ||
url: str | ||
hours: int | ||
signatures: str |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if the Parameter
called signatures
it is actually a secret, the config()
function would have assigned it to _signatures
in this class.
Idk how this is working on your side
class ThreatFox(Ingestor): | ||
url: str | ||
days: int | ||
|
||
BASE_URL = "https://threatfox-api.abuse.ch/api/v1/" | ||
|
||
def run(self) -> Iterable[Any]: | ||
result = requests.post( | ||
self.BASE_URL, json={"query": "get_iocs", "days": self.days} | ||
) | ||
result = requests.post(self.url, json={"query": "get_iocs", "days": self.days}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you change the URL? Why someone should be able to change the URL to something differently?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed the BASE_URL
to url
in order to enable the healthcheck feature (it's present, why disable it completely?).
I made it to do things the same way, as if it were a standard. If it doesn't go well I'll put it back as before.
…estor # Conflicts: # requirements/project-requirements.txt
fixed threatfox migration
result = requests.post( | ||
self.url, | ||
data={"query": "get_siginfo", "signature": signature, "limit": 100}, | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add a timeout to the post request? Just in case
sample_archive = requests.post( | ||
self.url, | ||
data={ | ||
"query": "get_file", | ||
"sha256_hash": h, | ||
}, | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a timeout
# Download samples from chosen signatures (aka malware families) | ||
signatures: str | ||
|
||
def run(self) -> Iterable[Any]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This function imho should be split in 3 parts:
- Where we are retrieving the results of each signatures
- Where we are parsing the results and adding stuff in the
hashes
variable - Where we are downloading the file from the hashes
I would split this function in these 3 parts, to decrease its complexity
For the error, you have to cast the datetime retrieved to aware (or viceversa) |
* Fix phoneinfoga name Signed-off-by: 0ssigeno <s.berni@certego.net> * Start with --traefik/--traefik_local option. Closes #2305 (#2351) * add traefik config and options for dev and prod working config with traefik finish traefik config prod/dev add documentation * Update traefik_local.override.yml - remove comment * rework prod/local traefik and add deletion of get-docker.sh * split traefik compose into base, prod and local * remove print of compose files * parent c45c84a author David Mihajlovic <david.mihajlovic@protonmail.com> 1716908101 +0200 committer David Mihajlovic <david.mihajlovic@protonmail.com> 1717135119 +0200 add traefik config and options for dev and prod working config with traefik finish traefik config prod/dev add documentation Vulners#1257 (#2340) * vulners * vulners wrapper * docs * lesser variables * migrations * code quality * migration * code --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> bump 6.0.3 updated docs Bump django-ses from 4.0.0 to 4.1.0 in /requirements (#2342) Bumps [django-ses](https://github.com/django-ses/django-ses) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/django-ses/django-ses/releases) - [Changelog](https://github.com/django-ses/django-ses/blob/main/CHANGES.md) - [Commits](django-ses/django-ses@v4.0.0...v4.1.0) --- updated-dependencies: - dependency-name: django-ses dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> rework prod/local traefik and add deletion of get-docker.sh split traefik compose into base, prod and local get-docker.sh deletion without sudo change traefik compose naming * remove unnecessary files * remove print of compose files * change doc --------- Co-authored-by: Ubuntu <ubuntu@intelowldev.novalocal> * Fix url Signed-off-by: 0ssigeno <s.berni@certego.net> * Visualizer improvements (#2366) * table visualizer improvements * adjusted tests * prettier * changes * fixed start script * Split folder creation into two parts removing sudo (#2373) * Bump elasticsearch-dsl from 8.13.0 to 8.14.0 in /requirements (#2370) Bumps [elasticsearch-dsl](https://github.com/elasticsearch/elasticsearch-dsl-py) from 8.13.0 to 8.14.0. - [Release notes](https://github.com/elasticsearch/elasticsearch-dsl-py/releases) - [Changelog](https://github.com/elastic/elasticsearch-dsl-py/blob/main/Changelog.rst) - [Commits](elastic/elasticsearch-dsl-py@v8.13.0...v8.14.0) --- updated-dependencies: - dependency-name: elasticsearch-dsl dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump quark-engine from 24.5.1 to 24.6.1 in /requirements (#2371) Bumps [quark-engine](https://github.com/quark-engine/quark-engine) from 24.5.1 to 24.6.1. - [Release notes](https://github.com/quark-engine/quark-engine/releases) - [Commits](quark-engine/quark-engine@v24.5.1...v24.6.1) --- updated-dependencies: - dependency-name: quark-engine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Auto creation default test user with debug=true#1189 (#2369) * create super user * env files :p --------- Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: g4ze <bhaiyajionline@gmail.com> * Bump library/nginx from 1.26.0-alpine to 1.27.0-alpine in /docker (#2358) Bumps library/nginx from 1.26.0-alpine to 1.27.0-alpine. --- updated-dependencies: - dependency-name: library/nginx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump authlib from 1.3.0 to 1.3.1 in /requirements (#2368) Bumps [authlib](https://github.com/lepture/authlib) from 1.3.0 to 1.3.1. - [Release notes](https://github.com/lepture/authlib/releases) - [Changelog](https://github.com/lepture/authlib/blob/master/docs/changelog.rst) - [Commits](lepture/authlib@v1.3.0...v1.3.1) --- updated-dependencies: - dependency-name: authlib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * detect-it-easy analyzer, closes #1590 (#2354) * die * tweeks * codefactor * codefactor * ypo * gitignore * typo fix * detectiteasyyyyy * tests * supported files * msdos * logs, file support, soft t/o, poll * migrate * for all files * docker_based_true * params * tests debug[1] * Update api_app/analyzers_manager/migrations/0094_analyzer_config_detectiteasy.py * Update api_app/analyzers_manager/file_analyzers/detectiteasy.py --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> * Bi update (#2326) * added bi document * update bi interface * update bi interface * fix bi serializer * update certego-saas version * mign fix (#2375) Co-authored-by: g4ze <bhaiyajionline@gmail.com> * watchman adjusts test (#2349) * watchman adjusts test * watchman right version * test * adjust * right watchman version * Malprob analyzer, closes #1521 (#2357) * init updates works, weirdly new flow updates tests deepsrc * tests * disable_ratelimit(), t/o * timeout,reform response,TLP:CLEAR,logs,no raise,disableRatelimit * migrations * reponse format * t/o * t/o(agn) * api_key * ratelimit,migrations,healthcheck --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * Passive_DNS playbook and visualizer (#2374) * created 'passive_dns' playbook and visualizer * dnsdb * validin * changes * refactor * changes * refactor + tests * changes * changes * Add create user docs (#2381) * docs for test user creation docs for test user creation * typo :"( --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * fixed capesandbox short analysis time limit (#2364) * fixed capesandbox short analysis time limit * added url to soft time limit error * fixed code doctor * added update method * added info installation process * Orkl_search analyzer, closes #1274 (#2380) * orkl search * docs * migrations * free to use * typo --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * Frontend - no more required analyzer in scan form (#2397) * no more requried analyzer in scan form * fix test * removed bad migration * Bump whitenoise from 6.6.0 to 6.7.0 in /requirements (#2396) Bumps [whitenoise](https://github.com/evansd/whitenoise) from 6.6.0 to 6.7.0. - [Changelog](https://github.com/evansd/whitenoise/blob/main/docs/changelog.rst) - [Commits](evansd/whitenoise@6.6.0...6.7.0) --- updated-dependencies: - dependency-name: whitenoise dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump greynoise from 2.1.0 to 2.2.0 in /requirements (#2389) Bumps [greynoise](https://github.com/GreyNoise-Intelligence/pygreynoise) from 2.1.0 to 2.2.0. - [Release notes](https://github.com/GreyNoise-Intelligence/pygreynoise/releases) - [Changelog](https://github.com/GreyNoise-Intelligence/pygreynoise/blob/master/CHANGELOG.rst) - [Commits](GreyNoise-Intelligence/pygreynoise@v2.1.0...v2.2.0) --- updated-dependencies: - dependency-name: greynoise dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * crtsh (#2379) crt_sh Passive_DNS playbook and visualizer (#2374) * created 'passive_dns' playbook and visualizer * dnsdb * validin * changes * refactor * changes * refactor + tests * changes * changes Add create user docs (#2381) * docs for test user creation docs for test user creation * typo :"( --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> fixed capesandbox short analysis time limit (#2364) * fixed capesandbox short analysis time limit * added url to soft time limit error * fixed code doctor * added update method added info installation process Orkl_search analyzer, closes #1274 (#2380) * orkl search * docs * migrations * free to use * typo --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> Frontend - no more required analyzer in scan form (#2397) * no more requried analyzer in scan form * fix test docs, migrations and corrections ci Co-authored-by: g4ze <bhaiyajionline@gmail.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> * Added external link to output (#2399) * Fixed load_env() parsing of .env files (#2400) * Spamhaus_WQS Analyzer, closes #1526 (#2378) * init * init * migration * docs * python * better code * code handling and migrations * better code * docs link * docs link --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * Pdf uri extractor and pivoting (#2391) * uri extraction * added download file analyzer and pivot configs * fixed code review doctor * made code review changes added job creation check to avoid graph related issues * added abstract update method * fixed migration order * fixed validated_data dict access * fixed migrations order * fixed migrations order * Malware bazaar ingestor (#2259) * added malware bazaar ingestor fixed json serialization for types: bytes and File * typo * added support to delayed celery jobs startup for ingestors * moved url to config parameter in this way you can set an health checker if you want * fixed wrong access to observable name * changed timedelta from class to object * added _monkeypatch() * omitted full_name field and generate ingestors plugin config * added threatfox url migration * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * updated threatfox migration * changed migration order * fixed reverse migrations * fixed default signatures * fixed default signatures * added malware bazaar userprofile fixed threatfox migration * isort * added default value to timedelta * fixed delay parameter default value and int conversion * fixed userprofile dumpplugin * reduced code complexity and fixed generator job creation * fixed deepsource warnings * fixed deepsoruce cyclic import * changed order PivotConfigurationException * made code review changes * fixed errors * fixed errors * fixed empty analyzer report * Adguard dns analyzer, closes #1361 (#2363) * adguard * adguard * bad query * tests * adguard works now :p * adguard * docs+mign * ci * ci * ci * tests * ci * ci * playbook * ci try * ci try * mign * mign --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * Fix ingestor (#2405) * Update Usage.md * disabled ingestors by default * fixed migration ingestors (#2406) * ja4db analyzer, closes #2361 (#2402) * adguard * adguard * bad query * ja4db * ci fixes * ci fix * ci fix * ci * cro tests * tests * adguard works now :p * adguard * docs+mign * ci * ci * ci * tests * ci * ci * playbook * ci try * ci try * mign * mign * mign upate * checks and amber * more precise * little refactor * added docstring --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> * Goresym analyzer, fixes#1451 and fixes executable file support (#2401) * file support * goresym * mign * ci fix * mign update * file types * file-types * mign * revert * pass tests * executable support * params mign * params add in run * params * tests * test files * tests * fix * fix * ci * tests files * main.out for Goresym * test files * comment * mign * changes * errorssss * update * conflict * tests --------- Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: g4ze <bhaiyajionline@gmail.com> * Fix ingestor dump Signed-off-by: 0ssigeno <s.berni@certego.net> * Bump djangorestframework from 3.15.1 to 3.15.2 in /requirements (#2398) Bumps [djangorestframework](https://github.com/encode/django-rest-framework) from 3.15.1 to 3.15.2. - [Release notes](https://github.com/encode/django-rest-framework/releases) - [Commits](encode/django-rest-framework@3.15.1...3.15.2) --- updated-dependencies: - dependency-name: djangorestframework dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump jsonschema from 4.22.0 to 4.23.0 in /requirements (#2409) Bumps [jsonschema](https://github.com/python-jsonschema/jsonschema) from 4.22.0 to 4.23.0. - [Release notes](https://github.com/python-jsonschema/jsonschema/releases) - [Changelog](https://github.com/python-jsonschema/jsonschema/blob/main/CHANGELOG.rst) - [Commits](python-jsonschema/jsonschema@v4.22.0...v4.23.0) --- updated-dependencies: - dependency-name: jsonschema dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump quark-engine from 24.6.1 to 24.7.1 in /requirements (#2410) Bumps [quark-engine](https://github.com/quark-engine/quark-engine) from 24.6.1 to 24.7.1. - [Release notes](https://github.com/quark-engine/quark-engine/releases) - [Commits](quark-engine/quark-engine@v24.6.1...v24.7.1) --- updated-dependencies: - dependency-name: quark-engine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump flake8 from 7.0.0 to 7.1.0 in /requirements (#2388) Bumps [flake8](https://github.com/pycqa/flake8) from 7.0.0 to 7.1.0. - [Commits](PyCQA/flake8@7.0.0...7.1.0) --- updated-dependencies: - dependency-name: flake8 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix duplicated ingestor users (#2412) * fix * removed if condition and added defaults * removed .title() * fixed test due to .title() change * fixed test due to .title() change * Bump django from 4.2.11 to 4.2.14 in /requirements (#2415) Bumps [django](https://github.com/django/django) from 4.2.11 to 4.2.14. - [Commits](django/django@4.2.11...4.2.14) --- updated-dependencies: - dependency-name: django dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Playbook choice for pivot and ingestor (#2411) * As title Signed-off-by: 0ssigeno <s.berni@certego.net> * Blake Signed-off-by: 0ssigeno <s.berni@certego.net> * Fixes Signed-off-by: 0ssigeno <s.berni@certego.net> * Fix tests Signed-off-by: 0ssigeno <s.berni@certego.net> * Fixes Signed-off-by: 0ssigeno <s.berni@certego.net> * Fixes Signed-off-by: 0ssigeno <s.berni@certego.net> * Fixes Signed-off-by: 0ssigeno <s.berni@certego.net> * Fix migrations after rebase Signed-off-by: 0ssigeno <s.berni@certego.net> --------- Signed-off-by: 0ssigeno <s.berni@certego.net> * Fix serializer Signed-off-by: 0ssigeno <s.berni@certego.net> * Fixes Signed-off-by: 0ssigeno <s.berni@certego.net> * Fix Signed-off-by: 0ssigeno <s.berni@certego.net> * Fix Signed-off-by: 0ssigeno <s.berni@certego.net> * Frontend - Replaced the time picker with a date picker (#2413) * created TimePicker component + useTimePickerSore * test * fix * Fix Signed-off-by: 0ssigeno <s.berni@certego.net> * updated frontend dependencies * Speed up query Signed-off-by: 0ssigeno <s.berni@certego.net> * Leakix analyzer, closes#1256 (#2423) * LeakIx * tests * lint * fix tasks duplicates (#2424) * fix tasks duplicates * reformatted out of scope fie * Apivoid analyzer, closes 1245 (#2428) * apivoid * tests * tests * qa * qa * qa * tests * Iocextract analyzer#1228 (#2426) * iocextract * iocextract * iocextract * ioc * iocextract * logs * mign * IocFinder Analyzer, closes #1229 (#2427) * IocFinder * bool * mign * mign * spamhaus_drop analyzer, closes #2408 (#2422) * spamhaus_drop * spamhaus_drop * ip matching * migratiuons * migrations * tests * tests * tests * tests * tests * tests * IocFinder * bool * mign * docs * mign * mign * mign * Criminalip analyzer closes#1240 (#2435) * cip * criminalip * criminalip * criminalip * criminalip * variables * Bump checkdmarc from 5.3.1 to 5.4.0 in /requirements (#2433) Bumps [checkdmarc](https://github.com/domainaware/checkdmarc) from 5.3.1 to 5.4.0. - [Changelog](https://github.com/domainaware/checkdmarc/blob/master/CHANGELOG.md) - [Commits](https://github.com/domainaware/checkdmarc/commits) --- updated-dependencies: - dependency-name: checkdmarc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump setuptools in /integrations/malware_tools_analyzers/requirements (#2416) Bumps [setuptools](https://github.com/pypa/setuptools) from 67.6.0 to 70.0.0. - [Release notes](https://github.com/pypa/setuptools/releases) - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst) - [Commits](pypa/setuptools@v67.6.0...v70.0.0) --- updated-dependencies: - dependency-name: setuptools dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Default value Signed-off-by: 0ssigeno <s.berni@certego.net> * Enable/disable admin actions Signed-off-by: 0ssigeno <s.berni@certego.net> * removed recaptcha (#2437) * removed recaptcha * removed useless import * prettier * removed env.js setup * Criminalip_Scan analyzer closes#1240 (#2438) * cip * criminalip * criminalip * criminalip * criminalip * variables * init * tests * mign * fixes * fixes * mign * mign * mign * mign * mign * adjusted investigation filters (#2440) * adjusted investigation filters * fixed 'playbook to execute' column * fix * fix deepsource * Bump greynoise from 2.2.0 to 2.3.0 in /requirements (#2446) Bumps [greynoise](https://github.com/GreyNoise-Intelligence/pygreynoise) from 2.2.0 to 2.3.0. - [Release notes](https://github.com/GreyNoise-Intelligence/pygreynoise/releases) - [Changelog](https://github.com/GreyNoise-Intelligence/pygreynoise/blob/master/CHANGELOG.rst) - [Commits](GreyNoise-Intelligence/pygreynoise@v2.2.0...v2.3.0) --- updated-dependencies: - dependency-name: greynoise dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * job_id BI (#2449) * added job_id field to bi * removed name from JobBISerializer * Polyswarm analyzer closes#1255 (#2439) * polyswarm * polyswarm * polyswarm * polyswarm * polyswarm * mign * logs * logs * mign * obs * obs * obs * tests * modular * Knock analyzer (#2448) * knock * migration * knock but no deletion reqed * t/o test * rmv log * timeout tests * t/o * mock * mock * tests * tests * t/o * typo * tlp * pypi * works now * log * mign --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * Fix triage (#2452) * fix triage manage submission response * fix * Bump quark-engine from 24.7.1 to 24.8.1 in /requirements (#2459) Bumps [quark-engine](https://github.com/quark-engine/quark-engine) from 24.7.1 to 24.8.1. - [Release notes](https://github.com/quark-engine/quark-engine/releases) - [Commits](quark-engine/quark-engine@v24.7.1...v24.8.1) --- updated-dependencies: - dependency-name: quark-engine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump polyswarm-api from 3.8.0 to 3.9.0 in /requirements (#2458) Bumps [polyswarm-api](https://github.com/polyswarm/polyswarm-api) from 3.8.0 to 3.9.0. - [Release notes](https://github.com/polyswarm/polyswarm-api/releases) - [Commits](polyswarm/polyswarm-api@3.8.0...3.9.0) --- updated-dependencies: - dependency-name: polyswarm-api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump django-filter from 24.2 to 24.3 in /requirements (#2457) Bumps [django-filter](https://github.com/carltongibson/django-filter) from 24.2 to 24.3. - [Release notes](https://github.com/carltongibson/django-filter/releases) - [Changelog](https://github.com/carltongibson/django-filter/blob/main/CHANGES.rst) - [Commits](carltongibson/django-filter@24.2...24.3) --- updated-dependencies: - dependency-name: django-filter dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump dnstwist[full] from 20240116 to 20240812 in /requirements (#2456) Bumps [dnstwist[full]](https://github.com/elceef/dnstwist) from 20240116 to 20240812. - [Release notes](https://github.com/elceef/dnstwist/releases) - [Commits](https://github.com/elceef/dnstwist/commits) --- updated-dependencies: - dependency-name: dnstwist[full] dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump elasticsearch-dsl from 8.14.0 to 8.15.0 in /requirements (#2455) Bumps [elasticsearch-dsl](https://github.com/elasticsearch/elasticsearch-dsl-py) from 8.14.0 to 8.15.0. - [Release notes](https://github.com/elasticsearch/elasticsearch-dsl-py/releases) - [Changelog](https://github.com/elastic/elasticsearch-dsl-py/blob/main/Changelog.rst) - [Commits](elastic/elasticsearch-dsl-py@v8.14.0...v8.15.0) --- updated-dependencies: - dependency-name: elasticsearch-dsl dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump django from 4.2.11 to 4.2.15 in /requirements (#2450) Bumps [django](https://github.com/django/django) from 4.2.11 to 4.2.15. - [Commits](django/django@4.2.11...4.2.15) --- updated-dependencies: - dependency-name: django dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated blint * removing documentation in favor of new doc repo * removing documentation in favor of new doc repo * update test-requirements and pr template * chore: update pluginTableColumns.jsx (#2466) fitler -> filter Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> * removed quotes * get back images for the ReadME * updated frontend dependencies * Improved PE_info analyzer (#2464) * update * update * init * init * blint fix * black and flake8 * upgraded lief * complexity --------- Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> * [WIP] Adding docstrings in IntelOwl Codebase. (#2430) * Added docstrings in Authentication Signed-off-by: aryan <aryan1bhokare@gmail.com> * Added docstrings in api_app module. Signed-off-by: aryan <aryan1bhokare@gmail.com> * fixed linters Signed-off-by: aryan <aryan1bhokare@gmail.com> --------- Signed-off-by: aryan <aryan1bhokare@gmail.com> * fixed frontend issues * updated PR automation * Bump django-iam-dbauth from 0.1.4 to 0.2.0 in /requirements (#2476) Bumps [django-iam-dbauth](https://github.com/LabD/django-iam-dbauth) from 0.1.4 to 0.2.0. - [Release notes](https://github.com/LabD/django-iam-dbauth/releases) - [Changelog](https://github.com/labd/django-iam-dbauth/blob/main/CHANGES) - [Commits](labd/django-iam-dbauth@0.1.4...0.2.0) --- updated-dependencies: - dependency-name: django-iam-dbauth dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump checkdmarc from 5.4.0 to 5.5.0 in /requirements (#2475) Bumps [checkdmarc](https://github.com/domainaware/checkdmarc) from 5.4.0 to 5.5.0. - [Changelog](https://github.com/domainaware/checkdmarc/blob/master/CHANGELOG.md) - [Commits](https://github.com/domainaware/checkdmarc/commits) --- updated-dependencies: - dependency-name: checkdmarc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fixed wrong task deletion (#2477) * email cc sender (#2468) * email sender Signed-off-by: 0ssigeno <s.berni@certego.net> * Fix Signed-off-by: 0ssigeno <s.berni@certego.net> * Added case for list Signed-off-by: 0ssigeno <s.berni@certego.net> * Blake Signed-off-by: 0ssigeno <s.berni@certego.net> --------- Signed-off-by: 0ssigeno <s.berni@certego.net> * removed obsolete docker compose version * updated readme * improved quad9 analyzers (#2453) * improved quad9 analyzers * fix * Refactor old documentation link. (#2465) * Refactor old documentation link. Signed-off-by: Aryan Bhokare <92683836+aryan-bhokare@users.noreply.github.com> * fix prettier test Signed-off-by: Aryan Bhokare <92683836+aryan-bhokare@users.noreply.github.com> * Fixed formatting Signed-off-by: Aryan Bhokare <92683836+aryan-bhokare@users.noreply.github.com> * Improved PE_info analyzer (#2464) * update * update * init * init * blint fix * black and flake8 * upgraded lief * complexity --------- Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> * Fixed flake8 errors. Signed-off-by: Aryan Bhokare <92683836+aryan-bhokare@users.noreply.github.com> * [WIP] Adding docstrings in IntelOwl Codebase. (#2430) * Added docstrings in Authentication Signed-off-by: aryan <aryan1bhokare@gmail.com> * Added docstrings in api_app module. Signed-off-by: aryan <aryan1bhokare@gmail.com> * fixed linters Signed-off-by: aryan <aryan1bhokare@gmail.com> --------- Signed-off-by: aryan <aryan1bhokare@gmail.com> * fixed frontend issues * updated PR automation * Bump django-iam-dbauth from 0.1.4 to 0.2.0 in /requirements (#2476) Bumps [django-iam-dbauth](https://github.com/LabD/django-iam-dbauth) from 0.1.4 to 0.2.0. - [Release notes](https://github.com/LabD/django-iam-dbauth/releases) - [Changelog](https://github.com/labd/django-iam-dbauth/blob/main/CHANGES) - [Commits](labd/django-iam-dbauth@0.1.4...0.2.0) --- updated-dependencies: - dependency-name: django-iam-dbauth dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump checkdmarc from 5.4.0 to 5.5.0 in /requirements (#2475) Bumps [checkdmarc](https://github.com/domainaware/checkdmarc) from 5.4.0 to 5.5.0. - [Changelog](https://github.com/domainaware/checkdmarc/blob/master/CHANGELOG.md) - [Commits](https://github.com/domainaware/checkdmarc/commits) --- updated-dependencies: - dependency-name: checkdmarc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fixed wrong task deletion (#2477) * email cc sender (#2468) * email sender Signed-off-by: 0ssigeno <s.berni@certego.net> * Fix Signed-off-by: 0ssigeno <s.berni@certego.net> * Added case for list Signed-off-by: 0ssigeno <s.berni@certego.net> * Blake Signed-off-by: 0ssigeno <s.berni@certego.net> --------- Signed-off-by: 0ssigeno <s.berni@certego.net> * removed obsolete docker compose version * updated readme * Refactor old documentation link. Signed-off-by: Aryan Bhokare <92683836+aryan-bhokare@users.noreply.github.com> * fix prettier test Signed-off-by: Aryan Bhokare <92683836+aryan-bhokare@users.noreply.github.com> * Fixed formatting Signed-off-by: Aryan Bhokare <92683836+aryan-bhokare@users.noreply.github.com> * Fixed flake8 errors. Signed-off-by: Aryan Bhokare <92683836+aryan-bhokare@users.noreply.github.com> * linters again Signed-off-by: Aryan Bhokare <92683836+aryan-bhokare@users.noreply.github.com> --------- Signed-off-by: Aryan Bhokare <92683836+aryan-bhokare@users.noreply.github.com> Signed-off-by: aryan <aryan1bhokare@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: 0ssigeno <s.berni@certego.net> Co-authored-by: Nilay Gupta <102874321+g4ze@users.noreply.github.com> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: Daniele Rosetti <d.rosetti@certego.net> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Federico Fantini <122294644+federicofantini@users.noreply.github.com> Co-authored-by: Simone Berni <simone.berni2@studio.unibo.it> * misp fix: invalid json (#2481) * fix1 * adjusted pymisp and upgraded * update pymisp --------- Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> * updated readme * Added docstrings in API_APP for api documentation (#2484) Signed-off-by: aryan <aryan1bhokare@gmail.com> * mobsf (#2461) * mobsf * MobSF * tests * req:p * typo:p * extra file * mign * files * seperate migns * fix * fixes * no mock * mock * mock * mock * comments * Droidlysis analyzer closes#1591 (#2454) * droid * droid * droid * config fixes * config fixes * fixes * mobsf * fixes * MobSF * tests * req:p * typo:p * extra file * mign * files * fixes * mign * test * tests * tests * add imgs * seperate migns * fix * fixes * no mock * mock * mock * mock * mock * comments * comments * comms * mign * merge conflict * Bump pefile from 2023.2.7 to 2024.8.26 in /requirements (#2489) Bumps [pefile](https://github.com/erocarrera/pefile) from 2023.2.7 to 2024.8.26. - [Release notes](https://github.com/erocarrera/pefile/releases) - [Commits](erocarrera/pefile@v2023.2.7...v2024.8.26) --- updated-dependencies: - dependency-name: pefile dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump django-silk from 5.1.0 to 5.2.0 in /requirements (#2487) Bumps [django-silk](https://github.com/jazzband/django-silk) from 5.1.0 to 5.2.0. - [Release notes](https://github.com/jazzband/django-silk/releases) - [Changelog](https://github.com/jazzband/django-silk/blob/master/CHANGELOG.md) - [Commits](jazzband/django-silk@5.1.0...5.2.0) --- updated-dependencies: - dependency-name: django-silk dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump django-celery-beat from 2.6.0 to 2.7.0 in /requirements (#2488) Bumps [django-celery-beat](https://github.com/celery/django-celery-beat) from 2.6.0 to 2.7.0. - [Release notes](https://github.com/celery/django-celery-beat/releases) - [Changelog](https://github.com/celery/django-celery-beat/blob/main/Changelog) - [Commits](celery/django-celery-beat@v2.6.0...v2.7.0) --- updated-dependencies: - dependency-name: django-celery-beat dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * apk artifacts analyzer closes#2444 + upgraded stringsifter (#2469) * init * init * works * docker * comments * apk_artifacts * apk_artifacts * file supp --------- Signed-off-by: 0ssigeno <s.berni@certego.net> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: aryan <aryan1bhokare@gmail.com> Signed-off-by: Aryan Bhokare <92683836+aryan-bhokare@users.noreply.github.com> Co-authored-by: 0ssigeno <s.berni@certego.net> Co-authored-by: David Mihajlovic <47985423+agnorance@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@intelowldev.novalocal> Co-authored-by: Martina Carella <m.carella@certego.net> Co-authored-by: Daniele Rosetti <d.rosetti@certego.net> Co-authored-by: fgibertoni <152909479+fgibertoni@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nilay Gupta <102874321+g4ze@users.noreply.github.com> Co-authored-by: g4ze <bhaiyajionline@gmail.com> Co-authored-by: Cristina Ascari <95929371+cristinaascari@users.noreply.github.com> Co-authored-by: Federico Fantini <122294644+federicofantini@users.noreply.github.com> Co-authored-by: Simone Berni <simone.berni2@studio.unibo.it> Co-authored-by: Daniele Rosetti <55402684+drosetti@users.noreply.github.com> Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com> Co-authored-by: Aryan Bhokare <92683836+aryan-bhokare@users.noreply.github.com>
* added malware bazaar ingestor fixed json serialization for types: bytes and File * typo * added support to delayed celery jobs startup for ingestors * moved url to config parameter in this way you can set an health checker if you want * fixed wrong access to observable name * changed timedelta from class to object * added _monkeypatch() * omitted full_name field and generate ingestors plugin config * added threatfox url migration * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * updated threatfox migration * changed migration order * fixed reverse migrations * fixed default signatures * fixed default signatures * added malware bazaar userprofile fixed threatfox migration * isort * added default value to timedelta * fixed delay parameter default value and int conversion * fixed userprofile dumpplugin * reduced code complexity and fixed generator job creation * fixed deepsource warnings * fixed deepsoruce cyclic import * changed order PivotConfigurationException * made code review changes * fixed errors * fixed errors
* added malware bazaar ingestor fixed json serialization for types: bytes and File * typo * added support to delayed celery jobs startup for ingestors * moved url to config parameter in this way you can set an health checker if you want * fixed wrong access to observable name * changed timedelta from class to object * added _monkeypatch() * omitted full_name field and generate ingestors plugin config * added threatfox url migration * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * updated threatfox migration * changed migration order * fixed reverse migrations * fixed default signatures * fixed default signatures * added malware bazaar userprofile fixed threatfox migration * isort * added default value to timedelta * fixed delay parameter default value and int conversion * fixed userprofile dumpplugin * reduced code complexity and fixed generator job creation * fixed deepsource warnings * fixed deepsoruce cyclic import * changed order PivotConfigurationException * made code review changes * fixed errors * fixed errors
* added malware bazaar ingestor fixed json serialization for types: bytes and File * typo * added support to delayed celery jobs startup for ingestors * moved url to config parameter in this way you can set an health checker if you want * fixed wrong access to observable name * changed timedelta from class to object * added _monkeypatch() * omitted full_name field and generate ingestors plugin config * added threatfox url migration * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * fixed linter * updated threatfox migration * changed migration order * fixed reverse migrations * fixed default signatures * fixed default signatures * added malware bazaar userprofile fixed threatfox migration * isort * added default value to timedelta * fixed delay parameter default value and int conversion * fixed userprofile dumpplugin * reduced code complexity and fixed generator job creation * fixed deepsource warnings * fixed deepsoruce cyclic import * changed order PivotConfigurationException * made code review changes * fixed errors * fixed errors
Description
Addition of MalwareBazaar Ingestor and minor bug fixing
Type of change
Please delete options that are not relevant.
Checklist
develop
_monkeypatch()
was used in its class to apply the necessary decorators.dumpplugin
command and added it in the project as a data migration. ("How to share a plugin with the community")test_files.zip
and you added the default tests for that mimetype in test_classes.py.FREE_TO_USE_ANALYZERS
playbook by following this guide.MockUpResponse
of the_monkeypatch()
method. This serves us to provide a valid sample for testing.Black
,Flake
,Isort
) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.tests
folder). All the tests (new and old ones) gave 0 errors.DeepSource
,Django Doctors
or other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.