Refactor API parsers #7002

kiblik · 2022-10-17T22:02:18Z

PR is based on my question in Slack: https://owasp.slack.com/archives/C014H3ZV9U6/p1661299601825669

Summary

The number of tools/parsers which do not only parse uploaded files but communicate with API is increasing. This PR does some small clean-up and defines a set of rules to keep it maintainable and improve UX for using API parsers (simplified way to add Tool Type, Tool Config and ApiScanConfig).

a parser which supports API is placed in the directory dojo/tools/api_<tool> - yes, this means renames like sonarqube_api to api_sonarqube
in documentation, in a list of parsers, a subsection/subheader only for these parsers was created. At the beginning of this section is a manual “how to set ToolType, ToolConfig, ApiScanConfig” to avoid repetition in all parsers. Each parser defines only which values and how should be filled
add function to parsers api_scan_configuration_hint which returns string with a hint, which service_key and how should be set up
add function which collects hints from api_scan_configuration_hint and print it in add and edit view
Add to documentation: “how to write a parser which communicates with API”. Some rules:
Required parts: api_client, importer, parser
Write tests
Optional but recommened functions for api_client: test_connection, test_product_connection
Update template for “Template Generator”: Add support for API parsers cookiecutter-scanner-parser#8
In almost all api_clients is used Tool_Type.objects.get_or_create(name='XYZ'). Parsers should be stateless (they shouldn't create these entries).
Integration test for creation of ToolType, ToolConfig and ApiScanConfig

Example of UI changes

Dynamic hints. Tool Type for BlackDuck, Cobalt.io and Vulners exist. There is no Toll Type for Bugcrowd, Edgescan and SonarQube - I removed them only because of this test. They are by default there with a new or upgraded instance.

Description Parser XXX requires created tool type XXX. is <a>, it is referencing to page where the user can create missing Tool Type

After clicking on the link, a user is redirected to the page with the prefield form

Updated description after creating Tool Type. The link is now pointing to the page for the Tool Config creation

Tool Type is automatically fielded

New Tool Config is ready. The description is updated to hint from api_scan_configuration_hint

github-actions · 2022-10-20T14:32:39Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

Bumps [packageurl-python](https://github.com/package-url/packageurl-python) from 0.10.3 to 0.10.4. - [Release notes](https://github.com/package-url/packageurl-python/releases) - [Changelog](https://github.com/package-url/packageurl-python/blob/main/CHANGELOG.rst) - [Commits](package-url/packageurl-python@v0.10.3...v0.10.4) --- updated-dependencies: - dependency-name: packageurl-python dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 1.4.41 to 1.4.42. - [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases) - [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/main/CHANGES.rst) - [Commits](https://github.com/sqlalchemy/sqlalchemy/commits) --- updated-dependencies: - dependency-name: sqlalchemy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ine (docker-compose.yml) (DefectDojo#6993) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

… (.github/workflows/release-drafter.yml) (DefectDojo#7000) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…fectDojo#6995)

…Dojo#6998) Bumps python from 3.8.14-slim-bullseye to 3.8.15-slim-bullseye. --- updated-dependencies: - dependency-name: python dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#7005) Update the Cobalt API parser's `CobaltAPI` client to fetch the maximum allowed number of findings and assets.

github-actions · 2022-10-21T20:17:15Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

github-actions · 2022-10-26T00:39:47Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

…lows/submodule-update.yml) (DefectDojo#7014) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

DefectDojo#7008) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…tDojo#6066) * [FIX] Issues on disconnection and connection using OAuth Fix two issues: - When disconnecting using OAuth and "SHOW_LOGIN_FORM = False", as return URL is not /login and does not contains next parameter, an error 500 is trigerred. - When connecing using OAuth and "SHOW_LOGIN_FORM = False", message "You have logged out" is displayed after logging in. * Fixing Flake8 issues * Update package.json * Update __init__.py * Update views.py * Update pipeline.py * Update Chart.yaml * Update __init__.py * Update views.py * Update pipeline.py * Update pipeline.py * Update pipeline.py

github-actions · 2022-12-05T22:27:49Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

github-actions · 2022-12-07T22:46:42Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2023-01-06T11:49:13Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

kiblik · 2023-01-12T13:46:22Z

Hi, I would like to ask about your opinion about this PR (and testing if it is possible).

Thanks in advance.

damiencarol · 2023-01-13T14:29:30Z

This is good. @kiblik

ericcornelissen

Everything looks good and works correctly from Cobalt's side, thanks @kiblik

Maffooch

Excellent work as always 🤙🏼

kiblik added 3 commits October 17, 2022 21:32

Rename "..._api" parsers to "api_..."

d99baa4

Reorder + categorise parsers in documentation

c2a9faa

Group same step to one definition

8c00b2e

github-actions bot added docs parser unittests labels Oct 17, 2022

kiblik added 4 commits October 17, 2022 22:12

Fix some imports

a5297f0

Add Edgescan and fix some imports

d7988ec

Fix mocks

809736c

Fix next unittests

b9e097b

github-actions bot added the conflicts-detected label Oct 20, 2022

dependabot bot and others added 8 commits October 21, 2022 20:15

Update rabbitmq:3.11.1-alpine Docker digest from 3.11.1 to 3.11.1-alp…

ea481e8

…ine (docker-compose.yml) (DefectDojo#6993) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update release-drafter/release-drafter action from v5.21.0 to v5.21.1…

868ac25

… (.github/workflows/release-drafter.yml) (DefectDojo#7000) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Parse Veracode library_id for SCA to get the maven component name (De…

0abbd1f

…fectDojo#6995)

Enable filtering Findings on steps_to_reproduce (DefectDojo#6970)

5b65e65

Update asset and findings retrieval for Cobalt API parser (DefectDojo…

2ef1ed0

…#7005) Update the Cobalt API parser's `CobaltAPI` client to fetch the maximum allowed number of findings and assets.

github-actions bot added the docker label Oct 21, 2022

Merge remote-tracking branch 'upstream/dev' into refactor_api_parsers

43f1465

github-actions bot added conflicts-detected and removed conflicts-detected docker labels Oct 21, 2022

renovate bot and others added 3 commits October 26, 2022 12:11

Update tj-actions/changed-files action from v32 to v33 (.github/workf…

122d313

…lows/submodule-update.yml) (DefectDojo#7014) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update rabbitmq Docker tag from 3.11.1 to v3.11.2 (docker-compose.yml) (

64db9a2

DefectDojo#7008) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

github-actions bot removed the conflicts-detected label Dec 5, 2022

kiblik added 2 commits December 5, 2022 22:29

Fix double docs

4827683

add VulnersAPI to factory

68a984b

github-actions bot added the conflicts-detected label Dec 7, 2022

kiblik added 2 commits January 5, 2023 17:12

Small typos

bc33b15

Merge remote-tracking branch 'upstream/dev' into refactor_api_parsers

fb1f7ec

github-actions bot removed the conflicts-detected label Jan 6, 2023

kiblik added 2 commits January 10, 2023 18:23

Fix VulnersAPI in factory

dcb272d

Fix unittests

73e5c23

kiblik mentioned this pull request Jan 10, 2023

Add support for API parsers DefectDojo/cookiecutter-scanner-parser#8

Merged

kiblik force-pushed the refactor_api_parsers branch from 1368f28 to d548a75 Compare January 10, 2023 22:28

add tests

6cd86d8

kiblik force-pushed the refactor_api_parsers branch from d548a75 to 6cd86d8 Compare January 10, 2023 22:44

kiblik added 3 commits January 11, 2023 00:15

Documentation - how to write API parsers

426df4e

Fix integration test

ecf98d7

Docs: Add test_connection and test_product_connection [skip action]

7780112

kiblik marked this pull request as ready for review January 11, 2023 21:11

Merge remote-tracking branch 'upstream/dev' into refactor_api_parsers

8e31e35

kiblik closed this Jan 13, 2023

kiblik reopened this Jan 13, 2023

damiencarol approved these changes Jan 13, 2023

View reviewed changes

ericcornelissen approved these changes Jan 13, 2023

View reviewed changes

Maffooch approved these changes Jan 13, 2023

View reviewed changes

Maffooch merged commit ae08e12 into DefectDojo:dev Jan 13, 2023

kiblik deleted the refactor_api_parsers branch January 13, 2023 19:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor API parsers #7002

Refactor API parsers #7002

Uh oh!

kiblik commented Oct 17, 2022 •

edited

Loading

Uh oh!

github-actions bot commented Oct 20, 2022

Uh oh!

github-actions bot commented Oct 21, 2022

Uh oh!

github-actions bot commented Oct 26, 2022

Uh oh!

github-actions bot commented Dec 5, 2022

Uh oh!

github-actions bot commented Dec 7, 2022

Uh oh!

github-actions bot commented Jan 6, 2023

Uh oh!

kiblik commented Jan 12, 2023 •

edited

Loading

Uh oh!

damiencarol commented Jan 13, 2023

Uh oh!

ericcornelissen left a comment

Uh oh!

Maffooch left a comment

Uh oh!

Uh oh!

Refactor API parsers #7002

Refactor API parsers #7002

Uh oh!

Conversation

kiblik commented Oct 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Example of UI changes

Uh oh!

github-actions bot commented Oct 20, 2022

Uh oh!

github-actions bot commented Oct 21, 2022

Uh oh!

github-actions bot commented Oct 26, 2022

Uh oh!

github-actions bot commented Dec 5, 2022

Uh oh!

github-actions bot commented Dec 7, 2022

Uh oh!

github-actions bot commented Jan 6, 2023

Uh oh!

kiblik commented Jan 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

damiencarol commented Jan 13, 2023

Uh oh!

ericcornelissen left a comment

Choose a reason for hiding this comment

Uh oh!

Maffooch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kiblik commented Oct 17, 2022 •

edited

Loading

kiblik commented Jan 12, 2023 •

edited

Loading