Support local DBs

[oliverchang]

Currently the scanner works by using the OSV.dev API, which ensures the matching against latest live DB with little (targeted <15 minute latency from the upstream source)

We should support a local mode, which supports taking in a local OSV DB.

One of the key prerequisites here is:

• Implementing version comparison rules for all our supported ecosystems. This is necessary for precise vulnerability matching based on the OSV version matching algorithm (@G-Rath has something for this that does this for the many of our ecosystems)

This will have some limitations:

• Commit based matching will not work -- the API indexes all commit hashes ingested, and it's not feasible to replicate this index locally.
• Potential performance issues?

[G-Rath]

As you know this is (imo) pretty extensively supported by osv-detector - I'm happy for any code from that to be brought into the scanner, and think a good first step would be to look at that to see what people think is missing (or plain wrong).

Implementing version comparison rules for all our supported ecosystems.

I'm hoping to get around to PRing that over to here this week 😅 done 🎉

Potential performance issues?

Since the detector has been using a local database since day 1, I've spent a lot of time looking at the performance in a number of cases especially when:

• I added support for deduplicating aliases (which added like 3 extra full loops), and extra loops that came from misc fixes like handling Python packages with unnormalized names
• support for the osv.dev API was landed, to see if it was faster than using the local database
• I overhauled semantic to use ecosystem-specific version comparisons, since that greatly changed the work being done by the detector

In all cases the detector performed extremely fast, with the slowest part always being downloading the database (which even then would typically only take <5 seconds, for all the dbs needed for a project) - in contrast with the API it would take 15+ seconds 😅

(not saying there couldn't be performances issues, just sharing my experience so far)

[oliverchang]

@G-Rath any chance you'd have time to try tackle this one within the next few weeks? Otherwise we might be interested in taking this on as we've been seeing requests for this.

[G-Rath]

@oliverchang yup probably - do you want to start with just an --offline flag for now and go from there?

[oliverchang]

@oliverchang yup probably - do you want to start with just an --offline flag for now and go from there?

SGTM! Learning from our mistake with --json / --format though, perhaps something like --db may be more extensible. e.g.

--db='https://api.osv.dev' # default
--db=offline 

This lets us make things a bit more extensible in the future.

We can start this out as an experimental flag (subject to change) to get the ball rolling and change it later if we need to.

CC @another-rex

[G-Rath]

Hmm ok not really a fan of that but cant think of better approach beyond what's in the detector (whichd be a sizable change), so happy to go with it if you think it's the way to go.

Though what if we followed Nodes pattern of sticking --experimental- in front of these flags to let us really play around with them

[oliverchang]

--experimental- sgtm!

[spencerschrock]

which supports taking in a local OSV DB

This DB in question is the one hosted at gs://osv-vulnerabilities?

This will have some limitations:

• Commit based matching will not work -- the API indexes all commit hashes ingested, and it's not feasible to replicate this index locally.

This is for scenarios where a lockfile specifies a dependency by commit hash and the local DB wouldn't know how to map it to a version for matching against vulns?

[spencerschrock]

Friendly ping on these questions.

[another-rex]

Sorry missed this message!

which supports taking in a local OSV DB

This DB in question is the one hosted at gs://osv-vulnerabilities?

Yes, specifically the all.zip's in each ecosystem

This will have some limitations:

• Commit based matching will not work -- the API indexes all commit hashes ingested, and it's not feasible to replicate this index locally.

This is for scenarios where a lockfile specifies a dependency by commit hash and the local DB wouldn't know how to map it to a version for matching against vulns?

Yes, and also for scanning the commit of any git repos that's being scanned.