Store severity scores #290

sbs2001 · 2020-11-29T10:42:36Z

Implementation of #157 (comment)

Fixes #157

Add data structures to store severity scores

pombredanne

Thanks!
See my comments in line for your consideration.

vulnerabilities/data_source.py

vulnerabilities/importers/nvd.py

vulnerabilities/importers/redhat.py

vulnerabilities/models.py

vulnerabilities/tests/test_nvd.py

vulnerabilities/.vscode/settings.json

vulnerabilities/data_source.py

pombredanne

See my suggested changes for the models.

vulnerabilities/models.py

pombredanne · 2020-12-17T19:33:58Z

@sbs2001 you wrote:

I thought about #290 (comment) , and I think storing the ScoringSystems in the db would be a better approach

It can but then how scoring systems will we have? At most a couple per data sources, and in reality only a handful. And this is purely static data, and not really usable as data for some query except for the identifier so I am not sure this needs the added ceremony and complexity when stored in the DB. (e.g. you then need fixtures to bootstrap etc.). I see this instead as an enhanced but still very small list of value choices for a single field.

pombredanne

Thanks!
See my comments inline

vulnerabilities/importers/redhat.py

vulnerabilities/migrations/0002_vulnerabilityseverityscore.py

vulnerabilities/models.py

vulnerabilities/severity_systems.py

vulnerabilities/tests/test_nvd.py

vulnerabilities/tests/test_redhat_importer.py

* Correct typo in severity_systems.py * Use typo for scoring fields instead of list in models.py Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

* Correct typo in severity_systems.py * Use typo for scoring fields instead of list in models.py * Handle absence of bugzilla and RHSA better in redhat.py Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

vulnerabilities/import_runner.py

pombredanne

Almost there ! see my comments inline for your consideration.

vulnerabilities/importers/nvd.py

vulnerabilities/tests/test_redhat_importer.py

vulnerabilities/severity_systems.py

vulnerabilities/importers/redhat.py

vulnerabilities/models.py

vulnerabilities/tests/test_nvd.py

vulnerabilities/tests/test_redhat_importer.py

A VulnerabilitySeverity models is added in models.py to store severity of vulnerability. Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

A dataclass `VulnerabilitySeverity` is added to enable to transport of severity scores. The logic in importer_runner.py is modified to store, update severity scores and link it to reference and vulnerability Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

scoring_system_identifier is changed to scoring_system Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

* Correct typo in severity_systems.py * Use typo for scoring fields instead of list in models.py * Handle absence of bugzilla and RHSA better in redhat.py Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

pombredanne · 2021-02-09T12:41:26Z

vulnerabilities/importers/redhat.py

+    bugzilla = advisory_data.get("bugzilla")
+    if bugzilla:
+        url = "https://bugzilla.redhat.com/show_bug.cgi?id={}".format(bugzilla)
+        bugzilla_data = requests.get(f"https://bugzilla.redhat.com/rest/bug/{bugzilla}").json()


Just as a side note, this is the kind of JSON we would likely need to store forever as back auditable evidence when we will do this later ... which likely calls for a central place where we fetch things from

pombredanne · 2021-02-09T12:43:29Z

vulnerabilities/models.py

+        max_length=50,
+        choices=scoring_system_choices,
+        help_text="Identifier for the scoring system used. Available choices are: {} ".format(
+                  ", ".join(


For readability, an intermediate variable would be better for this

pombredanne · 2021-02-09T12:45:11Z

vulnerabilities/tests/test_redhat_importer.py

                        namespace="redhat",
                        name="bash",
                        version="4.1.2-48.el6",
-                        qualifiers=OrderedDict(),


Feel free to remove OrderedDict from the whole codebase BTW since the dicts are always ordered now.

pombredanne

LGTM! let's merge.

* Correct typo in severity_systems.py * Use typo for scoring fields instead of list in models.py * Handle absence of bugzilla and RHSA better in redhat.py Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

sbs2001 requested a review from pombredanne November 29, 2020 10:42

sbs2001 changed the title ~~Store severity scores~~ [WIP]Store severity scores Nov 29, 2020

pombredanne requested changes Dec 1, 2020

View reviewed changes

pombredanne reviewed Dec 2, 2020

View reviewed changes

vulnerabilities/.vscode/settings.json Outdated Show resolved Hide resolved

pombredanne reviewed Dec 11, 2020

View reviewed changes

vulnerabilities/data_source.py Outdated Show resolved Hide resolved

vulnerabilities/data_source.py Show resolved Hide resolved

pombredanne reviewed Dec 11, 2020

View reviewed changes

vulnerabilities/models.py Outdated Show resolved Hide resolved

sbs2001 force-pushed the store_severity branch from efcf6df to f6371c6 Compare December 18, 2020 09:23

sbs2001 mentioned this pull request Dec 19, 2020

REST API: Bulk requests for packages and vulnerabilities #284

Closed

sbs2001 force-pushed the store_severity branch 3 times, most recently from 0118d9e to b24c934 Compare December 24, 2020 09:50

sbs2001 changed the title ~~[WIP]Store severity scores~~ Store severity scores Dec 24, 2020

sbs2001 force-pushed the store_severity branch 3 times, most recently from d6b2bee to 9a72e52 Compare January 24, 2021 13:03

sbs2001 requested a review from pombredanne January 26, 2021 04:20

pombredanne requested changes Jan 27, 2021

View reviewed changes

sbs2001 mentioned this pull request Jan 29, 2021

Make RedHat CVE import more robust #319

Merged

sbs2001 force-pushed the store_severity branch from 9a72e52 to b0aeef5 Compare January 30, 2021 11:14

sbs2001 force-pushed the store_severity branch from 2656e76 to 783292b Compare January 31, 2021 11:09

pombredanne reviewed Feb 2, 2021

View reviewed changes

vulnerabilities/import_runner.py Show resolved Hide resolved

pombredanne requested changes Feb 2, 2021

View reviewed changes

sbs2001 added 5 commits February 3, 2021 14:19

Add db models to store severity scores of vulnerabilities

4fc018f

A VulnerabilitySeverity models is added in models.py to store severity of vulnerability. Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Store severity scores given by NVD.

221bc12

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Store severity scores given by RedHat

52724a5

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Remove VulnerabilityReferenceInserter class

98e4082

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

sbs2001 added 16 commits February 3, 2021 14:19

Make code review changes

9fc7ec8

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Avoid unnnecessary init of default args in test_nvd.py

b0ae6a8

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Delete vscode config and add it to gitignore

5d22856

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Rename scores to severities in the advisory dataclass

190440b

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Improve order of fields in Reference dataclass

ee7e509

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Store severity scores given by NVD.

9ca5558

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Store severity scores given by RedHat

0887e83

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Misc Data Structure changes

86f2e4f

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Refactor tests and importers to use new data structure field names

e8e6166

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Rebase and resolve conflicts

c9df2f9

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Create and use ScoringSystem objects for handling severity

0312cfb

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Cast score value to string while inserting it into db

641b457

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Change field naming in VulnerabilitySeverity model

be24df0

scoring_system_identifier is changed to scoring_system Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Make PR review changes for PR aboutcode-org#290

5428b37

* Correct typo in severity_systems.py * Use typo for scoring fields instead of list in models.py * Handle absence of bugzilla and RHSA better in redhat.py Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Add scoring system for vectors

d78cd0f

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

Fix tests

8d19463

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

sbs2001 force-pushed the store_severity branch from c737b5a to 8d19463 Compare February 3, 2021 08:50

sbs2001 requested a review from pombredanne February 3, 2021 11:29

pombredanne reviewed Feb 9, 2021

View reviewed changes

pombredanne approved these changes Feb 9, 2021

View reviewed changes

Merge branch 'main' into store_severity

bea7f16

sbs2001 merged commit 1d7b64e into aboutcode-org:main Feb 9, 2021

sbs2001 mentioned this pull request Feb 14, 2021

Change API to adapt to new models. #351

Closed

sbs2001 mentioned this pull request Apr 26, 2021

Essential modifications required in Models. #22

Closed

Uh oh!

Store severity scores #290

Store severity scores #290

Uh oh!

Conversation

sbs2001 commented Nov 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pombredanne commented Dec 17, 2020

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pombredanne Feb 9, 2021

Choose a reason for hiding this comment

Uh oh!

pombredanne Feb 9, 2021

Choose a reason for hiding this comment

Uh oh!

pombredanne Feb 9, 2021

Choose a reason for hiding this comment

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sbs2001 commented Nov 29, 2020 •

edited

Loading