implement filtering by packages through the config #944

josieang · 2024-04-26T05:47:01Z

I'd like feedback on the config yaml schema, the filter message and it's behaviour if the version is empty (it filters any version of that package).

This is in response to #814

pkg/config/config.go

pkg/osvscanner/osvscanner.go

josieang · 2024-04-26T08:15:08Z

I think I need a better way to filter based on semver parts. I know think deps.dev/util/semver does this, I will come back to it once I'm back from leave.

another-rex

Thanks! Main comment is that I think we can consolidate the two overrides into one, so they can share the package matching logic.

another-rex · 2024-05-03T00:40:00Z

pkg/config/config.go

-	LoadPath          string        `toml:"LoadPath"`
-	GoVersionOverride string        `toml:"GoVersionOverride"`
+	IgnoredVulns                   []IgnoreEntry                        `toml:"IgnoredVulns"`
+	IgnoredPackageVersions         []IgnorePackageVersionEntry          `toml:"IgnoredPackageVersions"`


I feel like ignore package versions and override package versions both could have the same "package matching" logic.

Can we consolidate them into the same field, and just have a bool ignore which if true ignores the package entirely, otherwise an override field where someone can override the license.

E.g.

[[Package]] name = "pkg-name" exactVersion = "1.0.0" ecosystem = "Go" ignore = false # if true ignores the package entirely licenseOverride = ["MIT", "0BSD"] # In the future, if someone requests we can also add a versionOverride field here, or other overrides reason = "abc" # major ... # minor ... # ...

+1.

version (Which I think we should exactVersion to) is optional right?

Additionally, would it make sense to make license a structured field of some sort instead?

I.e.

[[Package]] [[Package.license]] override = ["MIT"]

This makes sense to me! done.

pkg/config/config.go

pkg/osvscanner/osvscanner.go

another-rex · 2024-05-03T00:47:53Z

pkg/osvscanner/vulnerability_result_internal_test.go

+					Allowlist: []models.License{models.License("MIT"), models.License("0BSD")},
+				},
+			},
+			Results: []models.PackageSource{


For maintainability, can we just use the snapshot library here to store/match the results?

should be done but check my code just in case I'm not understanding the snapshot library correctly

pkg/config/config.go

oliverchang · 2024-05-06T03:30:59Z

pkg/config/config.go

-	LoadPath          string        `toml:"LoadPath"`
-	GoVersionOverride string        `toml:"GoVersionOverride"`
+	IgnoredVulns                   []IgnoreEntry                        `toml:"IgnoredVulns"`
+	IgnoredPackageVersions         []IgnorePackageVersionEntry          `toml:"IgnoredPackageVersions"`


+1.

version (Which I think we should exactVersion to) is optional right?

Additionally, would it make sense to make license a structured field of some sort instead?

I.e.

[[Package]] [[Package.license]] override = ["MIT"]

pkg/config/config.go

codecov-commenter · 2024-05-07T01:15:35Z

Codecov Report

Attention: Patch coverage is 81.96721% with 11 lines in your changes missing coverage. Please review.

Project coverage is 65.18%. Comparing base (8fd553a) to head (78fe74d).
Report is 1 commits behind head on main.

Files	Patch %	Lines
pkg/osvscanner/osvscanner.go	8.33%	10 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #944      +/-   ##
==========================================
+ Coverage   65.14%   65.18%   +0.04%     
==========================================
  Files         149      149              
  Lines       12338    12384      +46     
==========================================
+ Hits         8037     8072      +35     
- Misses       3849     3859      +10     
- Partials      452      453       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

josieang · 2024-05-07T01:21:00Z

Was I supposed to provide a token?

https://github.com/google/osv-scanner/actions/runs/8978083643/job/24657957118

Error: Codecov token not found. Please provide Codecov token with -t flag.
Error: Codecov: Failed to properly create commit: The process '/Users/runner/work/_actions/codecov/codecov-action/5ecb98a3c6b[74](https://github.com/google/osv-scanner/actions/runs/8978083643/job/24657957118#step:5:77)7ed38dc09f787459979aebb39be/dist/codecov' failed with exit code 1

another-rex · 2024-05-07T01:29:26Z

We seem to be getting these errors every now and then, current workaround is just rerun the tests

another-rex

Minor variable naming nit, otherwise LGTM, thanks!

another-rex · 2024-05-27T04:22:00Z

pkg/config/config.go

-	LoadPath                       string                               `toml:"LoadPath"`
-	GoVersionOverride              string                               `toml:"GoVersionOverride"`
+	IgnoredVulns      []IgnoreEntry         `toml:"IgnoredVulns"`
+	PackageVersions   []PackageVersionEntry `toml:"PackageVersions"`


Rename this to PackageOverrides and PackageOverrideEntry, since version is optional, and we are specifying packages for the scanner to override the interpretation of

implement filtering by packages using config files

cdea96c

josieang requested a review from another-rex April 26, 2024 05:47

josieang commented Apr 26, 2024

View reviewed changes

pkg/config/config.go Outdated Show resolved Hide resolved

pkg/config/config.go Outdated Show resolved Hide resolved

pkg/osvscanner/osvscanner.go Show resolved Hide resolved

shahar-h mentioned this pull request Apr 30, 2024

Handle OSS licenses and attribution envoyproxy/gateway#2917

Closed

another-rex reviewed May 3, 2024

View reviewed changes

oliverchang reviewed May 6, 2024

View reviewed changes

josieang requested review from another-rex, G-Rath and oliverchang May 7, 2024 01:12

another-rex approved these changes May 27, 2024

View reviewed changes

josieang enabled auto-merge (squash) June 6, 2024 03:55

josieang disabled auto-merge June 6, 2024 03:55

josieang force-pushed the package-config branch 2 times, most recently from 162fa59 to b7e6db1 Compare June 6, 2024 04:18

josieang added 5 commits June 6, 2024 14:28

implement overriding licenses through the config

e378f02

respond to comments

acd5c23

linter

ce63db6

snapshot

95efa12

rename PackageOverrides

cf09e19

josieang force-pushed the package-config branch from bc5e4b4 to cf09e19 Compare June 6, 2024 04:29

josieang added 2 commits June 6, 2024 14:30

Merge branch 'main' into package-config

d929daa

revert snapshot file

78fe74d

josieang merged commit 6d0e29e into google:main Jun 6, 2024
13 checks passed

BrewTestBot mentioned this pull request Jun 21, 2024

osv-scanner 1.8.1 Homebrew/homebrew-core#175301

Merged

another-rex mentioned this pull request Jun 27, 2024

Setting/overwriting license for a dependency #814

Closed

another-rex mentioned this pull request Aug 5, 2024

Add package level ignores to config format #670

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement filtering by packages through the config #944

implement filtering by packages through the config #944

josieang commented Apr 26, 2024 •

edited

Loading

josieang commented Apr 26, 2024

another-rex left a comment

another-rex May 3, 2024

oliverchang May 6, 2024

josieang May 7, 2024

another-rex May 3, 2024

josieang May 7, 2024

oliverchang May 6, 2024

codecov-commenter commented May 7, 2024 •

edited

Loading

josieang commented May 7, 2024 •

edited

Loading

another-rex commented May 7, 2024

another-rex left a comment

another-rex May 27, 2024

implement filtering by packages through the config #944

implement filtering by packages through the config #944

Conversation

josieang commented Apr 26, 2024 • edited Loading

josieang commented Apr 26, 2024

another-rex left a comment

Choose a reason for hiding this comment

another-rex May 3, 2024

Choose a reason for hiding this comment

oliverchang May 6, 2024

Choose a reason for hiding this comment

josieang May 7, 2024

Choose a reason for hiding this comment

another-rex May 3, 2024

Choose a reason for hiding this comment

josieang May 7, 2024

Choose a reason for hiding this comment

oliverchang May 6, 2024

Choose a reason for hiding this comment

codecov-commenter commented May 7, 2024 • edited Loading

Codecov Report

josieang commented May 7, 2024 • edited Loading

another-rex commented May 7, 2024

another-rex left a comment

Choose a reason for hiding this comment

another-rex May 27, 2024

Choose a reason for hiding this comment

josieang commented Apr 26, 2024 •

edited

Loading

codecov-commenter commented May 7, 2024 •

edited

Loading

josieang commented May 7, 2024 •

edited

Loading