Skip to content

Commit ae08e12

Browse files
kiblikdependabot[bot]renovate[bot]coheigeaCharlieSears
authored
Refactor API parsers (#7002)
* Rename "..._api" parsers to "api_..." * Reorder + categorise parsers in documentation * Group same step to one definition * Fix some imports * Add Edgescan and fix some imports * Fix mocks * Fix next unittests * Bump packageurl-python from 0.10.3 to 0.10.4 (#6997) Bumps [packageurl-python](https://github.com/package-url/packageurl-python) from 0.10.3 to 0.10.4. - [Release notes](https://github.com/package-url/packageurl-python/releases) - [Changelog](https://github.com/package-url/packageurl-python/blob/main/CHANGELOG.rst) - [Commits](package-url/packageurl-python@v0.10.3...v0.10.4) --- updated-dependencies: - dependency-name: packageurl-python dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump sqlalchemy from 1.4.41 to 1.4.42 (#6996) Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 1.4.41 to 1.4.42. - [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases) - [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/main/CHANGES.rst) - [Commits](https://github.com/sqlalchemy/sqlalchemy/commits) --- updated-dependencies: - dependency-name: sqlalchemy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update rabbitmq:3.11.1-alpine Docker digest from 3.11.1 to 3.11.1-alpine (docker-compose.yml) (#6993) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update release-drafter/release-drafter action from v5.21.0 to v5.21.1 (.github/workflows/release-drafter.yml) (#7000) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Parse Veracode library_id for SCA to get the maven component name (#6995) * Enable filtering Findings on steps_to_reproduce (#6970) * Bump python from 3.8.14-slim-bullseye to 3.8.15-slim-bullseye (#6998) Bumps python from 3.8.14-slim-bullseye to 3.8.15-slim-bullseye. --- updated-dependencies: - dependency-name: python dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update asset and findings retrieval for Cobalt API parser (#7005) Update the Cobalt API parser's `CobaltAPI` client to fetch the maximum allowed number of findings and assets. * Update tj-actions/changed-files action from v32 to v33 (.github/workflows/submodule-update.yml) (#7014) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update rabbitmq Docker tag from 3.11.1 to v3.11.2 (docker-compose.yml) (#7008) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * [FIX] Issues on disconnection and connection using Social Auth (#6066) * [FIX] Issues on disconnection and connection using OAuth Fix two issues: - When disconnecting using OAuth and "SHOW_LOGIN_FORM = False", as return URL is not /login and does not contains next parameter, an error 500 is trigerred. - When connecing using OAuth and "SHOW_LOGIN_FORM = False", message "You have logged out" is displayed after logging in. * Fixing Flake8 issues * Update package.json * Update __init__.py * Update views.py * Update pipeline.py * Update Chart.yaml * Update __init__.py * Update views.py * Update pipeline.py * Update pipeline.py * Update pipeline.py * Fix out of SLA time (#7017) * Add a HTML link in the references back to the Bugcrowd finding (#7018) * Bump boto3 from 1.24.55 to 1.25.0 (#7022) Bumps [boto3](https://github.com/boto/boto3) from 1.24.55 to 1.25.0. - [Release notes](https://github.com/boto/boto3/releases) - [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst) - [Commits](boto/boto3@1.24.55...1.25.0) --- updated-dependencies: - dependency-name: boto3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update tj-actions/changed-files action from v33 to v34 (.github/workflows/submodule-update.yml) (#7026) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update gcr.io/cloudsql-docker/gce-proxy Docker tag from 1.32.0 to v1.33.0 (helm/defectdojo/values.yaml) (#7025) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Bump uwsgi from 2.0.20 to 2.0.21 (#7023) Bumps [uwsgi](https://github.com/unbit/uwsgi-docs) from 2.0.20 to 2.0.21. - [Release notes](https://github.com/unbit/uwsgi-docs/releases) - [Commits](https://github.com/unbit/uwsgi-docs/commits) --- updated-dependencies: - dependency-name: uwsgi dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update postgres:14.5-alpine Docker digest from 14.5 to v (docker-compose.yml) (#7024) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Bump google-auth-oauthlib from 0.5.3 to 0.6.0 (#7021) Bumps [google-auth-oauthlib](https://github.com/GoogleCloudPlatform/google-auth-library-python-oauthlib) from 0.5.3 to 0.6.0. - [Release notes](https://github.com/GoogleCloudPlatform/google-auth-library-python-oauthlib/releases) - [Changelog](https://github.com/googleapis/google-auth-library-python-oauthlib/blob/main/CHANGELOG.md) - [Commits](googleapis/google-auth-library-python-oauthlib@v0.5.3...v0.6.0) --- updated-dependencies: - dependency-name: google-auth-oauthlib dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update rabbitmq:3.11.2-alpine Docker digest from 3.11.2 to 3.11.2-alpine (docker-compose.yml) (#7020) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Bump nginx from 1.23.1-alpine to 1.23.2-alpine (#7019) Bumps nginx from 1.23.1-alpine to 1.23.2-alpine. --- updated-dependencies: - dependency-name: nginx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update stefanzweifel/git-auto-commit-action action from v4.15.1 to v4.15.2 (.github/workflows/release-3-master-into-dev.yml) (#7016) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Add support for ZAP "XML with requests and responses" format (#7013) * Support ZAP XML with requests and responses Signed-off-by: Max Maass <max.maass@iteratec.com> * Update ZAP parser with final XMLplus format Signed-off-by: Max Maass <max.maass@iteratec.com> * Update ZAP parser docs Signed-off-by: Max Maass <max.maass@iteratec.com> Signed-off-by: Max Maass <max.maass@iteratec.com> * Bump pyjwt from 2.5.0 to 2.6.0 (#7010) Bumps [pyjwt](https://github.com/jpadilla/pyjwt) from 2.5.0 to 2.6.0. - [Release notes](https://github.com/jpadilla/pyjwt/releases) - [Changelog](https://github.com/jpadilla/pyjwt/blob/master/CHANGELOG.rst) - [Commits](https://github.com/jpadilla/pyjwt/commits) --- updated-dependencies: - dependency-name: pyjwt dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update mysql:5.7.40 Docker digest from 5.7.40 to v (docker-compose.yml) (#7007) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Bump pytz from 2022.4 to 2022.5 (#7003) Bumps [pytz](https://github.com/stub42/pytz) from 2022.4 to 2022.5. - [Release notes](https://github.com/stub42/pytz/releases) - [Commits](stub42/pytz@release_2022.4...release_2022.5) --- updated-dependencies: - dependency-name: pytz dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Don't save vulnerability Ids on a re-import if they're already defined for the finding (#7012) * Bump gitpython from 3.1.28 to 3.1.29 (#6966) Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.28 to 3.1.29. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](gitpython-developers/GitPython@3.1.28...3.1.29) --- updated-dependencies: - dependency-name: gitpython dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add the ability to only create finding groups when you have more than one finding in the group (#6916) * Bump google-auth from 2.12.0 to 2.13.0 (#7004) Bumps [google-auth](https://github.com/googleapis/google-auth-library-python) from 2.12.0 to 2.13.0. - [Release notes](https://github.com/googleapis/google-auth-library-python/releases) - [Changelog](https://github.com/googleapis/google-auth-library-python/blob/main/CHANGELOG.md) - [Commits](googleapis/google-auth-library-python@v2.12.0...v2.13.0) --- updated-dependencies: - dependency-name: google-auth dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update postgres Docker tag from 14.5 to v15 (docker-compose.yml) (#6994) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Bump google-api-python-client from 2.64.0 to 2.65.0 (#7006) Bumps [google-api-python-client](https://github.com/googleapis/google-api-python-client) from 2.64.0 to 2.65.0. - [Release notes](https://github.com/googleapis/google-api-python-client/releases) - [Changelog](https://github.com/googleapis/google-api-python-client/blob/main/CHANGELOG.md) - [Commits](googleapis/google-api-python-client@v2.64.0...v2.65.0) --- updated-dependencies: - dependency-name: google-api-python-client dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Forms: allow to prefill values * Fix class names - to be able to load them * Better logging in tools.factory * add 'get_api_scan_configuration_hints' + render hints * Flake8 * Cleanup of parsers * Fix unittests * Fix hint rendering * Add tool config unittests * PoC: ToolConfig tests * Add int test to all tests * Tests: Fix IDs * Move scans in tests * Revert "Tests: Fix IDs" This reverts commit dddefdd. * Fix intTest login, flake8 * Add BlackDuck to SCAN_APIS * Fix intTest path * Fix test * Fix edit page * Avoid 2 useless tests * Rebase Vulners * Fix double docs * add VulnersAPI to factory * Small typos * Fix VulnersAPI in factory * Fix unittests * add tests * Documentation - how to write API parsers * Fix integration test * Docs: Add test_connection and test_product_connection [skip action] Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Max Maass <max.maass@iteratec.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Colm O hEigeartaigh <coheigea@users.noreply.github.com> Co-authored-by: CharlieSears <charlie.sears@gmail.com> Co-authored-by: Eric Cornelissen <eric.cornelissen@cobalt.io> Co-authored-by: Ludovic Courgnaud <ludovic.courgnaud@gmail.com> Co-authored-by: Max Maass <1688580+malexmave@users.noreply.github.com>
1 parent 1b16700 commit ae08e12

File tree

83 files changed

+772
-514
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

83 files changed

+772
-514
lines changed

.github/workflows/integration-tests.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,7 @@ jobs:
100100
"tests/dedupe_test.py",
101101
"tests/check_various_pages.py",
102102
"tests/notifications_test.py",
103+
"tests/tool_config.py",
103104
]
104105
profile: ["mysql-rabbitmq", "postgres-redis"]
105106
fail-fast: false

docker/entrypoint-integration-tests.sh

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -238,5 +238,14 @@ else
238238
# else
239239
# echo "Error: Zap integration test failed"; exit 1
240240
# fi
241+
242+
test="Tool Config integration tests"
243+
echo "Running: $test"
244+
if python3 tests/tool_config.py ; then
245+
success $test
246+
else
247+
fail $test
248+
fi
249+
241250
exec echo "Done Running all configured integration tests."
242251
fi

docs/content/en/contributing/how-to-write-a-parser.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ $ docker-compose build --build-arg uid=1000
3939
|`unittests/scans/<parser_dir>/{many_vulns,no_vuln,one_vuln}.json` | Sample files containing meaningful data for unit tests. The minimal set.
4040
|`unittests/tools/test_<parser_name>_parser.py` | Unit tests of the parser.
4141
|`dojo/settings/settings.dist.py` | If you want to use a modern hashcode based deduplication algorithm
42+
|`doc/content/en/integrations/parsers` | Documentation, what kind of file format is required and how it should be obtained
4243

4344
## Factory contract
4445

@@ -91,6 +92,35 @@ class MyToolParser(object):
9192

9293
```
9394

95+
## API Parsers
96+
97+
Some reports are not reachable as a file that the user or pipeline can upload but the results of the scans have to be downloaded via API (or we just want to add support for multiple methods).
98+
In that case, an "API parser" is needed. Core code is the same as a regular parser but there are some additional requirements.
99+
100+
### Which files do you need to modify? (API Parsers only)
101+
102+
| File | Purpose
103+
|------- |--------
104+
|`dojo/tools/api_<parser_dir>/api_client.py` | API client should perform all HTTP API calls and JSON with data from the API
105+
|`dojo/tools/api_<parser_dir>/importer.py` | Importer should prepare the API client and process its results
106+
|`dojo/tools/api_<parser_dir>/parser.py` | Parser should fetch processed data from the importer
107+
|`unittests/tools/test_api_<parser_name>_parser.py` | Unit tests of the parser.
108+
|`unittests/tools/test_api_<parser_name>_importer.py` | Unit tests of the importer.
109+
|`dojo/tool_config/factory.py` | Parser must be listed in `SCAN_APIS`
110+
|`unittests/test_tool_config.py` | Unit tests for content of hints and other metadata
111+
112+
### Factory contract (API Parsers only)
113+
114+
1. Parser directory *MUST* starts with `api_`
115+
- ex: `dojo/tools/api_mytool`
116+
2. class-name of parser *MUST* starts with `Api`
117+
- ex: `ApiMytoolParser`
118+
3. Parser *MUST* implements function `def api_scan_configuration_hint(self)` which returns a string with a hint, on how to configure service keys in Product ...TODO. Using of HTML tag `<b>` is required. Help will be rendered on the website.
119+
- ex: `return 'the field <b>Service key 1</b> has to be set to ID of the project. <b>Service key 2</b> has to be set to the version of the project'`
120+
4. Parser *MUST* implemets function `def requires_tool_type(self, scan_type)` which returns name of the required `Tool_Type`.
121+
5. Parser *MUST NOT* create related `Tool_Type`. It will be created automatically based on the function `requires_tool_type`.
122+
6. API client *SHOULD* implemets `def test_connection(self):` and `def test_product_connection(self, api_scan_configuration):` to be able to test connectivity and test permissions. It should return string with a sucessfull status (like _you have access to 125 projects_) or raise an exception.
123+
94124
## Template Generator
95125

96126
Use the [template](https://github.com/DefectDojo/cookiecutter-scanner-parser) parser to quickly generate the files required. To get started you will need to install [cookiecutter](https://github.com/cookiecutter/cookiecutter).
@@ -284,6 +314,12 @@ for finding in findings:
284314
endpoint.clean()
285315
```
286316

317+
### Tests API Parsers
318+
319+
Not only parser but also importer should be tested.
320+
`patch` method from `unittest.mock` is usualy usefull for simulating API responses.
321+
It is highly recommeded to use it.
322+
287323
## Other files that could be involved
288324

289325
### Change to the model

0 commit comments

Comments
 (0)