Offline scan for custom lockfile (C/C++ project from git project) skipped

[cybops35]

Hello, osv-scanner have awesome functionalities

• Scan custom lockfile: to scan C/C++ dependencies from github projects
• A downloadable copy of the OSV database (stored in a GCS bucket maintained by OSV)

But i seems, it is not possible to scan custom lockfile using offline mirror of OSV database...

reproduce

• my commands to mirror the database:

❯ export OSV_SCANNER_LOCAL_DB_CACHE_DIRECTORY="$HOME/.cache"
❯ for eco in $(gsutil cat gs://osv-vulnerabilities/ecosystems.txt); do
    mkdir -p "$OSV_SCANNER_LOCAL_DB_CACHE_DIRECTORY/osv-scanner/$eco"
    gsutil cp gs://osv-vulnerabilities/$eco/all.zip "$OSV_SCANNER_LOCAL_DB_CACHE_DIRECTORY/osv-scanner/$eco/"
done

Copying gs://osv-vulnerabilities/AlmaLinux/all.zip...
- [1 files][  3.7 MiB/  3.7 MiB]                                                
Operation completed over 1 objects/3.7 MiB.                                      
Copying gs://osv-vulnerabilities/Alpine/all.zip...
- [1 files][ 11.8 MiB/ 11.8 MiB]                                                
Operation completed over 1 objects/11.8 MiB.                                     
Copying gs://osv-vulnerabilities/Android/all.zip...
- [1 files][  4.8 MiB/  4.8 MiB]                                                
Operation completed over 1 objects/4.8 MiB.                                      
Copying gs://osv-vulnerabilities/Bitnami/all.zip...
- [1 files][  5.4 MiB/  5.4 MiB]                                                
Operation completed over 1 objects/5.4 MiB.                                      
Copying gs://osv-vulnerabilities/CRAN/all.zip...
/ [1 files][  9.2 KiB/  9.2 KiB]                                                
Operation completed over 1 objects/9.2 KiB.                                      
Copying gs://osv-vulnerabilities/Chainguard/all.zip...
- [1 files][ 14.5 MiB/ 14.5 MiB]                                                
Operation completed over 1 objects/14.5 MiB.                                     
Copying gs://osv-vulnerabilities/Debian/all.zip...
| [1 files][ 64.2 MiB/ 64.2 MiB]                                                
Operation completed over 1 objects/64.2 MiB.                                     
Copying gs://osv-vulnerabilities/GHC/all.zip...
/ [1 files][  2.5 KiB/  2.5 KiB]                                                
Operation completed over 1 objects/2.5 KiB.                                      
Copying gs://osv-vulnerabilities/GIT/all.zip...
\ [1 files][ 34.9 MiB/ 34.9 MiB]                                                
Operation completed over 1 objects/34.9 MiB.                                     
Copying gs://osv-vulnerabilities/GSD/all.zip...
/ [1 files][  5.5 KiB/  5.5 KiB]                                                
Operation completed over 1 objects/5.5 KiB.                                      
CommandException: No URLs matched: gs://osv-vulnerabilities/GitHub/all.zip
CommandException: No URLs matched: gs://osv-vulnerabilities/Actions/all.zip
Copying gs://osv-vulnerabilities/Go/all.zip...
- [1 files][  4.7 MiB/  4.7 MiB]                                                
Operation completed over 1 objects/4.7 MiB.                                      
Copying gs://osv-vulnerabilities/Hackage/all.zip...
/ [1 files][ 32.4 KiB/ 32.4 KiB]                                                
Operation completed over 1 objects/32.4 KiB.                                     
Copying gs://osv-vulnerabilities/Hex/all.zip...
/ [1 files][ 43.3 KiB/ 43.3 KiB]                                                
Operation completed over 1 objects/43.3 KiB.                                     
Copying gs://osv-vulnerabilities/Linux/all.zip...
\ [1 files][ 28.3 MiB/ 28.3 MiB]                                                
Operation completed over 1 objects/28.3 MiB.                                     
Copying gs://osv-vulnerabilities/Mageia/all.zip...
- [1 files][  5.7 MiB/  5.7 MiB]                                                
Operation completed over 1 objects/5.7 MiB.                                      
Copying gs://osv-vulnerabilities/Maven/all.zip...
- [1 files][  7.6 MiB/  7.6 MiB]                                                
Operation completed over 1 objects/7.6 MiB.                                      
Copying gs://osv-vulnerabilities/MinimOS/all.zip...
- [1 files][989.7 KiB/989.7 KiB]                                                
Operation completed over 1 objects/989.7 KiB.                                    
Copying gs://osv-vulnerabilities/NuGet/all.zip...
- [1 files][  1.6 MiB/  1.6 MiB]                                                
Operation completed over 1 objects/1.6 MiB.                                      
Copying gs://osv-vulnerabilities/OSS-Fuzz/all.zip...
- [1 files][  2.8 MiB/  2.8 MiB]                                                
Operation completed over 1 objects/2.8 MiB.                                      
Copying gs://osv-vulnerabilities/Packagist/all.zip...
- [1 files][  6.1 MiB/  6.1 MiB]                                                
Operation completed over 1 objects/6.1 MiB.                                      
Copying gs://osv-vulnerabilities/Pub/all.zip...
/ [1 files][ 18.3 KiB/ 18.3 KiB]                                                
Operation completed over 1 objects/18.3 KiB.                                     
Copying gs://osv-vulnerabilities/PyPI/all.zip...
- [1 files][ 17.2 MiB/ 17.2 MiB]                                                
Operation completed over 1 objects/17.2 MiB.                                     
CommandException: No URLs matched: gs://osv-vulnerabilities/Red/all.zip
CommandException: No URLs matched: gs://osv-vulnerabilities/Hat/all.zip
CommandException: No URLs matched: gs://osv-vulnerabilities/Rocky/all.zip
Copying gs://osv-vulnerabilities/Linux/all.zip...
\ [1 files][ 28.3 MiB/ 28.3 MiB]                                                
Operation completed over 1 objects/28.3 MiB.                                     
Copying gs://osv-vulnerabilities/RubyGems/all.zip...
- [1 files][  1.9 MiB/  1.9 MiB]                                                
Operation completed over 1 objects/1.9 MiB.                                      
Copying gs://osv-vulnerabilities/SUSE/all.zip...
\ [1 files][ 32.9 MiB/ 32.9 MiB]                                                
Operation completed over 1 objects/32.9 MiB.                                     
Copying gs://osv-vulnerabilities/SwiftURL/all.zip...
/ [1 files][ 62.8 KiB/ 62.8 KiB]                                                
Operation completed over 1 objects/62.8 KiB.                                     
Copying gs://osv-vulnerabilities/UVI/all.zip...
/ [1 files][  1.1 KiB/  1.1 KiB]                                                
Operation completed over 1 objects/1.1 KiB.                                      
Copying gs://osv-vulnerabilities/Ubuntu/all.zip...
| [1 files][332.5 MiB/332.5 MiB]                                                
Operation completed over 1 objects/332.5 MiB.                                    
Copying gs://osv-vulnerabilities/Wolfi/all.zip...
- [1 files][  8.1 MiB/  8.1 MiB]                                                
Operation completed over 1 objects/8.1 MiB.                                      
CommandException: Destination (/home/user/.cache/osv-scanner/[EMPTY]/) must match exactly 1 URL
Copying gs://osv-vulnerabilities/crates.io/all.zip...
- [1 files][  2.0 MiB/  2.0 MiB]                                                
Operation completed over 1 objects/2.0 MiB.                                      
Copying gs://osv-vulnerabilities/npm/all.zip...
\ [1 files][ 27.4 MiB/ 27.4 MiB]                                                
Operation completed over 1 objects/27.4 MiB.                                     
Copying gs://osv-vulnerabilities/openSUSE/all.zip...
- [1 files][ 16.0 MiB/ 16.0 MiB]                                                
Operation completed over 1 objects/16.0 MiB.

• an example of a custom lockfile osv-scanner.json:

{
  "results": [
    {
      "packages": [
        {
          "package": {
            "name": "github.com/openssl/openssl",
            "commit": "45fda76bc1b9fd74d10e85e0ce9b65a12dcc58b0"
          }
        }
      ]
    }
  ]
}

• the scan online works:

❯ osv-scanner --lockfile osv-scanner:osv-scanner.json --format markdown                                                       

Scanned /home/user/Documents/foo/osv-scanner.json file as a osv-scanner and found 1 package
| OSV URL | CVSS | Ecosystem | Package | Version | Source |
| --- | --- | --- | --- | --- | --- |
| https://osv.dev/CVE-2024-12797 |  | GIT | github.com/openssl/openssl@45fda76b | github.com/openssl/openssl@45fda76b | osv-scanner.json |
| https://osv.dev/CVE-2024-13176 |  | GIT | github.com/openssl/openssl@45fda76b | github.com/openssl/openssl@45fda76b | osv-scanner.json |
| https://osv.dev/CVE-2024-9143 |  | GIT | github.com/openssl/openssl@45fda76b | github.com/openssl/openssl@45fda76b | osv-scanner.json |

• the scan offline is skipped, even with mirrored database:

❯ osv-scanner --lockfile osv-scanner:osv-scanner.json --format markdown --offline

Scanned /home/user/Documents/foo/osv-scanner.json file as a osv-scanner and found 1 package
Skipping commit scanning for: 45fda76bc1b9fd74d10e85e0ce9b65a12dcc58b0
No issues found

❯ osv-scanner --lockfile osv-scanner:osv-scanner.json --format markdown --offline-vulnerabilities --download-offline-databases

Scanned /home/user/Documents/foo/osv-scanner.json file as a osv-scanner and found 1 package
Skipping commit scanning for: 45fda76bc1b9fd74d10e85e0ce9b65a12dcc58b0
No issues found

Can you help me to scan a custom lockfile on offline network please ?
If it is not possible, is there a way to have an offline version of apo.osv.dev ?

[another-rex]

The issue is doing commit based scanning in offline mode, which is not possible right now. (https://google.github.io/osv-scanner/usage/offline-mode/#limitations).

Because commits are unordered, we have to index every single commit that is vulnerable when we get a record, which as you can imagine is a lot of data and not really feasible to download and run locally.

[cybops35]

Ok my bad, but hope it would be possible some day :)
Have a good day

[cybops35]

Hi @another-rex ! Would it be more feasible to have Github "tag" based scanning in offline mode ? Maybe the dataset is significantly smaller ?
Or, is there a way to build our own offline tag/commit based scanning database from a subset of chosen repositories ?

[another-rex]

Good point, when we designed offline scanning we didn't have tag information then. tag based scanning is likely something we can do, I'll look into it.

[another-rex]

@G-Rath This is the url for the git ecosystem zip. https://storage.googleapis.com/osv-test-vulnerabilities/GIT/all.zip

[bamiaux]

We've found that in our case, the only way to scan third-party c/c++ dependencies is to use custom lockfiles.
We map specific versions (openssl or any required c/c++ dependency) to their specific commits and use it in the lockfile. We don't really use the full power of custom lockfiles, as we only checkout tagged versions.

Having a stripped database of commits containing only tags, would help us doing scans offline.

It is our undestanding that there is no other way to specify in a project that "we depend on OpenSSL 3.2.3"