Failed to upload database for java: HttpError: Not Found

[remkop]

Describe the bug
I enabled CodeQL but did not see SAST being recognized when running scorecard in docker:

----------|-----------------------------------------------------------------------------------------------------
| 0 / 10  | SAST                   | SAST tool is not run on all    | Warn: 1 commits out of 30 are       
|         |                        | commits -- score normalized to | checked with a SAST tool Warn:    
|         |                        | 0                              | CodeQL tool not detected    
           https://github.com/ossf/scorecard/blob/a69e1d97d44ebba908ad4cf574d51c0f2e0f761e/docs/checks.md#sast  

@laurentsimon noticed

... that CodeQl is defined in your workflow, but seems to fail uploading the results: https://github.com/remkop/picocli/actions/runs/1794898507 which may be why scorecard is not detecting it.

It appears that the upload fails at the end of the "Perform CodeQL Analysis" job in the .github/workflows/codeql-analysis.yml in my repo:

      - name: Perform CodeQL Analysis
        uses: github/codeql-action/analyze@8b37404d562d866ad6a65d0ecb4fa5131e047ca4 # v1

I initially thought this was a ossf scorecard issue and reported it as such in ossf/scorecard#1605, but the underlying issue may be in the codeql-action.

Relevant section from the log follows below:

Run github/codeql-action/analyze@8b37404d56[2](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:2)d866ad6a65d0ecb4fa51[3](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:3)1e0[4](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:4)7ca4
  with:
    output: ../results
    upload: true
    cleanup-level: brutal
    add-snippets: false
    skip-queries: false
    checkout_path: /home/runner/work/picocli/picocli
    upload-database: true
    wait-for-processing: false
    token: ***
    matrix: {
    "language": "java"
  }
  env:
    CODEQL_ACTION_RUN_MODE: Action
    CODEQL_ACTION_VERSION: 1.0.30
    CODEQL_ACTION_FEATURE_SARIF_COMBINE: true
    CODEQL_ACTION_FEATURE_WILL_UPLOAD: true
    CODEQL_ACTION_ANALYSIS_KEY: .github/workflows/codeql-analysis.yml:analyze
    CODEQL_WORKFLOW_STARTED_AT: 2022-02-04T12:2[5](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:5):12.314Z
    CODEQL_ACTION_FEATURE_MULTI_LANGUAGE: false
    CODEQL_ACTION_FEATURE_SANDWICH: false
    CODEQL_RAM: 5923
    CODEQL_THREADS: 2
    ODASA_TRACER_CONFIGURATION: /home/runner/work/_temp/codeql_databases/working/tracing/compiler-tracing52835[6](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:6)9625230668[7](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:7)17.spec
    SEMMLE_JAVA_TOOL_OPTIONS: '-javaagent:/opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/java/tools/codeql-java-agent.jar=ignore-project,java' '-Xbootclasspath/a:/opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/java/tools/codeql-java-agent.jar'
    SEMMLE_PRELOAD_libtrace: /opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/tools/linux64/${LIB}_${PLATFORM}_trace.so
    SEMMLE_PRELOAD_libtrace32: /opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/tools/linux64/lib32trace.so
    SEMMLE_PRELOAD_libtrace64: /opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/tools/linux64/lib64trace.so
    CODEQL_SCRATCH_DIR: /home/runner/work/_temp/codeql_databases/working
    CODEQL_DIST: /opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql
    CODEQL_PLATFORM: linux64
    CODEQL_PLATFORM_DLL_EXTENSION: .so
    CODEQL_JAVA_HOME: /opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/tools/linux64/java
    CODEQL_EXTRACTOR_JAVA_ROOT: /opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/java
    CODEQL_EXTRACTOR_JAVA_WIP_DATABASE: /home/runner/work/_temp/codeql_databases/java
    CODEQL_EXTRACTOR_JAVA_LOG_DIR: /home/runner/work/_temp/codeql_databases/java/log
    CODEQL_EXTRACTOR_JAVA_SCRATCH_DIR: /home/runner/work/_temp/codeql_databases/java/working
    CODEQL_EXTRACTOR_JAVA_TRAP_DIR: /home/runner/work/_temp/codeql_databases/java/trap/java
    CODEQL_EXTRACTOR_JAVA_SOURCE_ARCHIVE_DIR: /home/runner/work/_temp/codeql_databases/java/src
    CODEQL_EXTRACTOR_JAVA_THREADS: 2
    CODEQL_EXTRACTOR_JAVA_RAM: 5923
    LD_PRELOAD: /opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/tools/linux64/${LIB}_${PLATFORM}_trace.so
    CODEQL_RUNNER: /opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/tools/linux64/runner
/opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/codeql version --format=terse
2.7.6
Finalizing java
Running queries for java
Interpreting results for java
Analysis produced the following diagnostic data:

|             Diagnostic             |                      Summary                       |
+------------------------------------+----------------------------------------------------+
| Extraction errors                  | 1 result (1 error)                                 |
| Diagnostics for framework coverage | 132 results (101 unknowns, 10 errors, 21 warnings) |
| Successfully extracted files       | 2[8](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:8)[9](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:9) results                                        |
| Extraction warnings                | 0 results                                          |
Analysis produced the following metric data:

|               Metric                | Value |
+-------------------------------------+-------+
| Total lines of code in the database | 65807 |

/opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/codeql database print-baseline /home/runner/work/_temp/codeql_databases/java
Counted a baseline of 67865 lines of code for java.
Counted a baseline of 67865 lines of code for java.

Cleaning up databases
  /opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/codeql database cleanup /home/runner/work/_temp/codeql_databases/java --mode=brutal
  Cleaning up existing TRAP files after import...
  TRAP files cleaned up (3ms).
  Cleaning up scratch directory...
  Scratch directory cleaned up (0ms).

Uploading results
  Processing sarif files: ["/home/runner/work/picocli/results/java.sarif"]
  Uploading results
  Successfully uploaded results
/opt/hostedtoolcache/CodeQL/0.0.0-20220120/x64/codeql/codeql database bundle /home/runner/work/_temp/codeql_databases/java --output=/home/runner/work/_temp/codeql_databases/java.zip --name=java
Creating bundle metadata for /home/runner/work/_temp/codeql_databases/java...
Creating zip file at /home/runner/work/_temp/codeql_databases/java.zip.
RequestError [HttpError]: Not Found
    at /home/runner/work/_actions/github/codeql-action/8b37404d562d866ad6a65d0ecb4fa5131e047ca4/node_modules/@octokit/request/dist-node/index.js:66:23
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Job.doExecute (/home/runner/work/_actions/github/codeql-action/8b37404d562d866ad6a65d0ecb4fa5131e047ca4/node_modules/bottleneck/light.js:405:18) {
  name: 'HttpError',
  status: 404,
  headers: {
    'access-control-allow-origin': '*',
    'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
    connection: 'close',
    'content-encoding': 'gzip',
    'content-security-policy': "default-src 'none'",
    'content-type': 'application/json; charset=utf-8',
    date: 'Fri, 04 Feb 2022 12:30:45 GMT',
    'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
    server: 'GitHub.com',
    'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
    'transfer-encoding': 'chunked',
    vary: 'Accept-Encoding, Accept, X-Requested-With',
    'x-content-type-options': 'nosniff',
    'x-frame-options': 'deny',
    'x-github-media-type': 'github.v3; format=json',
    'x-github-request-id': '0780:7AD8:CB9C09:1E2FF0E:61FD1C75',
    'x-ratelimit-limit': '[10](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:10)00',
    'x-ratelimit-remaining': '987',
    'x-ratelimit-reset': '164398[11](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:11)[12](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:12)',
    'x-ratelimit-resource': 'core',
    'x-ratelimit-used': '[13](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:13)',
    'x-xss-protection': '0'
  },
  request: {
    method: 'PUT',
    url: 'https://api.github.com/repos/remkop/picocli/code-scanning/codeql/databases/java',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'CodeQL-Action/1.0.30 octokit-core.js/3.1.2 Node.js/12.13.1 (linux; x64)',
      authorization: 'token [REDACTED]',
      'content-type': 'application/json; charset=utf-8'
    },
    body: <Buffer 50 4b 03 04 [14](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:14) 00 08 08 08 00 d4 63 44 54 00 00 00 00 00 00 00 00 00 00 00 00 0c 00 00 00 6a 61 76 61 2f 2e 64 62 69 6e 66 6f 6d 52 cb 4e c3 30 10 bc ... 1404[17](https://github.com/remkop/picocli/runs/5066373526?check_suite_focus=true#step:7:17)05 more bytes>,
    request: { agent: [Agent], hook: [Function: bound bound register] }
  },
  documentation_url: 'https://docs.github.com/rest'
}
Warning: Failed to upload database for java: HttpError: Not Found

[aibaars]

Hi @remkop, thanks for the report, and also for picocli, we're happy users of the library for the CodeQL CLI ;-)

Looking at the logs, it seems like the analysis results (in SARIF format) were successfully uploaded:

Uploading results
  Processing sarif files: ["/home/runner/work/picocli/results/java.sarif"]
  Uploading results
  Successfully uploaded results

The error message is from the codeql-action's attempt to store a copy of the CodeQL database. If I'm not mistaken this step is optional. I think it is for caching a copy of the intermediate CodeQL database for a new feature that allows running custom CodeQL queries directly on a repository. As far as I know this feature is only enabled for a limited set of repositories.

The codeql-action has a debug flag. You could turn that on to see more detailed logging. In addition the codeql-action should upload some additional files as artifacts (logs, sarif file, database). The sarif file should contain the alerts that CodeQL found. It's a json file that can be inspected manually, or be loaded in a SARIF viewer such as VS Code.

[chrisgavin]

Hi @remkop!

We've done a bit of investigation and think we know what's happening here. It looks like the analysis is succeeding, but an optional step of uploading the database is failing. This shouldn't affect the visibility of the overall results, but if you want to remove this warning I'm happy to report it should be fixed in the latest version of the CodeQL Action, so if you update your pinned commit to 1a927e9307bc11970b2c679922ebc4d03a5bd980 this warning should go away.

I'm not that familiar with the ossf/scorecard project, but it looks like there are two things it's currently warning about. "1 commits out of 30 are checked with a SAST tool" seems to be because it's looking at each recent commit and seeing if it has a Code Scanning check run on it. Because the pull request that added the CodeQL workflow is the most recent one to be merged it's only seeing the check on the most recent commit. Hopefully as more pull requests are merged, this warning will eventually disappear.

Secondly there's the "CodeQL tool not detected" warning. This looks like it's implemented by doing this code search https://github.com/remkop/picocli/search?q=github%2Fcodeql-action%2Fanalyze+path%3A%2F.github%2Fworkflows which does seem to return no results (at the time of writing this). I'm not sure I know enough about our search implementation to say why it does not find the workflow, but possibly this check could be implemented in a different way, for example fetching all workflows and then checking if they use the expected action.

[laurentsimon]

fyi, for the search implementation and scorecard: we're going to move away from the sear API and parse the workflow, so ignore this problem for now. It'll be fixed in next update.