"dependency not pinned by hash detected -- score normalized to 5" does not distinguish dev-time dependencies

[jab]

My project has no build-time or run-time dependencies, it only has development-time dependencies (test runners, linters, etc.).

(I pin the dev-time dependencies to exact versions. But until pip-tools adds better support for hashes, pinning to hashes incurs prohibitively expensive maintenance costs. In return, pinning to hashes provides little to no benefit to users of my project, for whom none of these dependencies actually get pulled in transitively.)

Nonetheless, it looks like my project's score is being unfairly penalized with "dependency not pinned by hash detected -- score normalized to 5", which is as large a penalty as if these were runtime dependencies that did get pulled in for users, rather than the dev-time dependencies that they actually are.

Can the scoring system please be improved to take this into account, so projects like mine aren't penalized unfairly?

[laurentsimon]

Thanks for the report.

Can you walk me thru your installation process? Typically we expect users to use requirements.txt, but I don't see one in your repo.

Can you tell me the alert reported by scorecard? if you ran the CLI tool, please add --show-details

[jab]

Thanks for taking a look at this.

Typically we expect users to use requirements.txt, but I don't see one in your repo.

You can find these here: https://github.com/jab/bidict/tree/main/dev-deps
(Note these are deliberately named things like lint.txt and test.txt as opposed to requirements.txt to more clearly indicate that these are lint- and test-time dependencies, not runtime dependencies.)

As such, these dependencies are only installed when linting or testing (etc.). See the lines like deps = -r dev-deps/test.txt and deps = -r dev-deps/lint.txt in the tox.ini here:
https://github.com/jab/bidict/blob/main/tox.ini

Can you tell me the alert reported by scorecard?

When I view https://deps.dev/pypi/bidict and expand "Pinned Dependencies", I get the alert I quoted in the issue description, "dependency not pinned by hash detected -- score normalized to 5". Here is a screenshot:

if you ran the CLI tool, please add --show-details

I had not run the CLI tool, but I just did and confirmed that the same alert and resulting score appears in that output:

❯ docker run --env GITHUB_AUTH_TOKEN gcr.io/openssf/scorecard --repo=https://github.com/jab/bidict --show-details
Starting [CI-Tests]
Starting [SAST]
Starting [License]
Starting [Maintained]
Starting [Contributors]
Starting [Pinned-Dependencies]
Starting [Dangerous-Workflow]
Starting [CII-Best-Practices]
Starting [Vulnerabilities]
Starting [Binary-Artifacts]
Starting [Branch-Protection]
Starting [Fuzzing]
Starting [Signed-Releases]
Starting [Token-Permissions]
Starting [Dependency-Update-Tool]
Starting [Packaging]
Starting [Security-Policy]
Starting [Code-Review]
Finished [Dangerous-Workflow]
Finished [CII-Best-Practices]
Finished [Fuzzing]
Finished [Signed-Releases]
Finished [Token-Permissions]
Finished [Vulnerabilities]
Finished [Binary-Artifacts]
Finished [Branch-Protection]
Finished [Packaging]
Finished [Security-Policy]
Finished [Code-Review]
Finished [Dependency-Update-Tool]
Finished [Maintained]
Finished [Contributors]
Finished [Pinned-Dependencies]
Finished [CI-Tests]
Finished [SAST]
Finished [License]

RESULTS
-------
Aggregate score: 6.9 / 10

Check scores:
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
|  SCORE  |          NAME          |             REASON             |                                                        DETAILS                                                        |                             DOCUMENTATION/REMEDIATION                             |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Binary-Artifacts       | no binaries found in the repo  |                                                                                                                       | https://github.com/ossf/scorecard/blob/main/docs/checks.md#binary-artifacts       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 2 / 10  | Branch-Protection      | branch protection is not       | Info: 'force pushes' disabled                                                                                         | https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection      |
|         |                        | maximal on development and all | on branch 'main' Info: 'allow                                                                                         |                                                                                   |
|         |                        | release branches               | deletion' disabled on branch                                                                                          |                                                                                   |
|         |                        |                                | 'main' Warn: settings do not                                                                                          |                                                                                   |
|         |                        |                                | apply to administrators on                                                                                            |                                                                                   |
|         |                        |                                | branch 'main' Info: status                                                                                            |                                                                                   |
|         |                        |                                | checks require up-to-date                                                                                             |                                                                                   |
|         |                        |                                | branches for 'main' Warn: no                                                                                          |                                                                                   |
|         |                        |                                | status checks found to merge                                                                                          |                                                                                   |
|         |                        |                                | onto branch 'main' Warn:                                                                                              |                                                                                   |
|         |                        |                                | number of required reviewers                                                                                          |                                                                                   |
|         |                        |                                | is 0 on branch 'main' Warn:                                                                                           |                                                                                   |
|         |                        |                                | Stale review dismissal                                                                                                |                                                                                   |
|         |                        |                                | disabled on branch 'main'                                                                                             |                                                                                   |
|         |                        |                                | Warn: codeowner review is not                                                                                         |                                                                                   |
|         |                        |                                | required on branch 'main'                                                                                             |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | CI-Tests               | 1 out of 1 merged PRs          |                                                                                                                       | https://github.com/ossf/scorecard/blob/main/docs/checks.md#ci-tests               |
|         |                        | checked by a CI test -- score  |                                                                                                                       |                                                                                   |
|         |                        | normalized to 10               |                                                                                                                       |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 5 / 10  | CII-Best-Practices     | badge detected: passing        |                                                                                                                       | https://github.com/ossf/scorecard/blob/main/docs/checks.md#cii-best-practices     |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 0 / 10  | Code-Review            | 0 out of last 30 changesets    |                                                                                                                       | https://github.com/ossf/scorecard/blob/main/docs/checks.md#code-review            |
|         |                        | reviewed before merge -- score |                                                                                                                       |                                                                                   |
|         |                        | normalized to 0                |                                                                                                                       |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Contributors           | 3 different organizations      | Info: contributors work for                                                                                           | https://github.com/ossf/scorecard/blob/main/docs/checks.md#contributors           |
|         |                        | found -- score normalized to   | pallets,recursecenter,two                                                                                             |                                                                                   |
|         |                        | 10                             | sigma                                                                                                                 |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Dangerous-Workflow     | no dangerous workflow patterns |                                                                                                                       | https://github.com/ossf/scorecard/blob/main/docs/checks.md#dangerous-workflow     |
|         |                        | detected                       |                                                                                                                       |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Dependency-Update-Tool | update tool detected           | Info: Dependabot detected:                                                                                            | https://github.com/ossf/scorecard/blob/main/docs/checks.md#dependency-update-tool |
|         |                        |                                | .github/dependabot.yml:1                                                                                              |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 0 / 10  | Fuzzing                | project is not fuzzed          |                                                                                                                       | https://github.com/ossf/scorecard/blob/main/docs/checks.md#fuzzing                |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | License                | license file detected          | Info: License file found in                                                                                           | https://github.com/ossf/scorecard/blob/main/docs/checks.md#license                |
|         |                        |                                | expected location: LICENSE:1                                                                                          |                                                                                   |
|         |                        |                                | Info: FSF or OSI recognized                                                                                           |                                                                                   |
|         |                        |                                | license: LICENSE:1                                                                                                    |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Maintained             | 19 commit(s) out of 30 and 3   |                                                                                                                       | https://github.com/ossf/scorecard/blob/main/docs/checks.md#maintained             |
|         |                        | issue activity out of 30 found |                                                                                                                       |                                                                                   |
|         |                        | in the last 90 days -- score   |                                                                                                                       |                                                                                   |
|         |                        | normalized to 10               |                                                                                                                       |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Packaging              | publishing workflow detected   | Info: GitHub publishing workflow used in run                                                                          | https://github.com/ossf/scorecard/blob/main/docs/checks.md#packaging              |
|         |                        |                                | https://api.github.com/repos/jab/bidict/actions/runs/2031076361:                                                      |                                                                                   |
|         |                        |                                | .github/workflows/release-to-pypi.yml:9                                                                               |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 5 / 10  | Pinned-Dependencies    | dependency not pinned by hash  | Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/lint.yml:17: update your                        | https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies    |
|         |                        | detected -- score normalized   | workflow using https://app.stepsecurity.io/secureworkflow/jab/bidict/lint.yml/main?enable=pin                         |                                                                                   |
|         |                        | to 5                           | Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/lint.yml:18: update your                        |                                                                                   |
|         |                        |                                | workflow using https://app.stepsecurity.io/secureworkflow/jab/bidict/lint.yml/main?enable=pin Warn:                   |                                                                                   |
|         |                        |                                | third-party GitHubAction not pinned by hash: .github/workflows/lint.yml:24: update your workflow using                |                                                                                   |
|         |                        |                                | https://app.stepsecurity.io/secureworkflow/jab/bidict/lint.yml/main?enable=pin Warn: GitHub-owned                     |                                                                                   |
|         |                        |                                | GitHubAction not pinned by hash: .github/workflows/release-to-pypi.yml:12: update your workflow using                 |                                                                                   |
|         |                        |                                | https://app.stepsecurity.io/secureworkflow/jab/bidict/release-to-pypi.yml/main?enable=pin Warn: GitHub-owned          |                                                                                   |
|         |                        |                                | GitHubAction not pinned by hash: .github/workflows/release-to-pypi.yml:13: update your workflow using                 |                                                                                   |
|         |                        |                                | https://app.stepsecurity.io/secureworkflow/jab/bidict/release-to-pypi.yml/main?enable=pin Warn: third-party           |                                                                                   |
|         |                        |                                | GitHubAction not pinned by hash: .github/workflows/release-to-pypi.yml:19: update your workflow using                 |                                                                                   |
|         |                        |                                | https://app.stepsecurity.io/secureworkflow/jab/bidict/release-to-pypi.yml/main?enable=pin Warn: GitHub-owned          |                                                                                   |
|         |                        |                                | GitHubAction not pinned by hash: .github/workflows/release-to-test-pypi.yml:21: update your workflow                  |                                                                                   |
|         |                        |                                | using https://app.stepsecurity.io/secureworkflow/jab/bidict/release-to-test-pypi.yml/main?enable=pin Warn:            |                                                                                   |
|         |                        |                                | GitHub-owned GitHubAction not pinned by hash: .github/workflows/release-to-test-pypi.yml:22: update your              |                                                                                   |
|         |                        |                                | workflow using https://app.stepsecurity.io/secureworkflow/jab/bidict/release-to-test-pypi.yml/main?enable=pin         |                                                                                   |
|         |                        |                                | Warn: third-party GitHubAction not pinned by hash: .github/workflows/release-to-test-pypi.yml:28: update your         |                                                                                   |
|         |                        |                                | workflow using https://app.stepsecurity.io/secureworkflow/jab/bidict/release-to-test-pypi.yml/main?enable=pin         |                                                                                   |
|         |                        |                                | Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:36: update your                        |                                                                                   |
|         |                        |                                | workflow using https://app.stepsecurity.io/secureworkflow/jab/bidict/test.yml/main?enable=pin                         |                                                                                   |
|         |                        |                                | Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:37: update your                        |                                                                                   |
|         |                        |                                | workflow using https://app.stepsecurity.io/secureworkflow/jab/bidict/test.yml/main?enable=pin                         |                                                                                   |
|         |                        |                                | Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:44: update your                        |                                                                                   |
|         |                        |                                | workflow using https://app.stepsecurity.io/secureworkflow/jab/bidict/test.yml/main?enable=pin Warn:                   |                                                                                   |
|         |                        |                                | third-party GitHubAction not pinned by hash: .github/workflows/test.yml:55: update your workflow using                |                                                                                   |
|         |                        |                                | https://app.stepsecurity.io/secureworkflow/jab/bidict/test.yml/main?enable=pin Warn: GitHub-owned                     |                                                                                   |
|         |                        |                                | GitHubAction not pinned by hash: .github/workflows/update_actions.yml:14: update your workflow                        |                                                                                   |
|         |                        |                                | using https://app.stepsecurity.io/secureworkflow/jab/bidict/update_actions.yml/main?enable=pin Warn:                  |                                                                                   |
|         |                        |                                | third-party GitHubAction not pinned by hash: .github/workflows/update_actions.yml:17: update your                     |                                                                                   |
|         |                        |                                | workflow using https://app.stepsecurity.io/secureworkflow/jab/bidict/update_actions.yml/main?enable=pin               |                                                                                   |
|         |                        |                                | Warn: containerImage not pinned by hash: .devcontainer/Dockerfile:6: pin your                                         |                                                                                   |
|         |                        |                                | Docker image by updating mcr.microsoft.com/vscode/devcontainers/python to                                             |                                                                                   |
|         |                        |                                | mcr.microsoft.com/vscode/devcontainers/python@sha256:03509a740d6d40c156998da7217babb756a5f1cff67d43ce50d296480b6d071c |                                                                                   |
|         |                        |                                | Warn: pipCommand not pinned by hash: upgrade-dev-dependencies.sh:37 Warn: pipCommand not pinned by hash:              |                                                                                   |
|         |                        |                                | .github/workflows/lint.yml:22 Warn: pipCommand not pinned by hash: .github/workflows/lint.yml:24 Warn:                |                                                                                   |
|         |                        |                                | pipCommand not pinned by hash: .github/workflows/release-to-pypi.yml:17 Warn: pipCommand not pinned by hash:          |                                                                                   |
|         |                        |                                | .github/workflows/release-to-test-pypi.yml:26 Warn: pipCommand not pinned by hash: .github/workflows/test.yml:43      |                                                                                   |
|         |                        |                                | Info: no insecure (not pinned by hash) dependency downloads found in Dockerfiles Info: no insecure (not pinned by     |                                                                                   |
|         |                        |                                | hash) dependency downloads found in shell scripts                                                                     |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | SAST                   | SAST tool is run on all        | Info: all commits (1) are                                                                                             | https://github.com/ossf/scorecard/blob/main/docs/checks.md#sast                   |
|         |                        | commits                        | checked with a SAST tool Warn:                                                                                        |                                                                                   |
|         |                        |                                | CodeQL tool not detected                                                                                              |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 9 / 10  | Security-Policy        | security policy file detected  | Info: Found linked content in                                                                                         | https://github.com/ossf/scorecard/blob/main/docs/checks.md#security-policy        |
|         |                        |                                | security policy: SECURITY.rst                                                                                         |                                                                                   |
|         |                        |                                | Info: Found text in security                                                                                          |                                                                                   |
|         |                        |                                | policy: SECURITY.rst Warn:                                                                                            |                                                                                   |
|         |                        |                                | One or no descriptive hints                                                                                           |                                                                                   |
|         |                        |                                | of disclosure, vulnerability,                                                                                         |                                                                                   |
|         |                        |                                | and/or timelines in security                                                                                          |                                                                                   |
|         |                        |                                | policy: SECURITY.rst Info:                                                                                            |                                                                                   |
|         |                        |                                | security policy detected in                                                                                           |                                                                                   |
|         |                        |                                | current repo: SECURITY.rst                                                                                            |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| ?       | Signed-Releases        | no releases found              | Warn: no GitHub releases found                                                                                        | https://github.com/ossf/scorecard/blob/main/docs/checks.md#signed-releases        |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 0 / 10  | Token-Permissions      | non read-only tokens detected  | Warn: no topLevel permission defined: .github/workflows/lint.yml:1: update your workflow using                        | https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions      |
|         |                        | in GitHub workflows            | https://app.stepsecurity.io/secureworkflow/jab/bidict/lint.yml/main?enable=permissions Warn: no                       |                                                                                   |
|         |                        |                                | topLevel permission defined: .github/workflows/release-to-pypi.yml:1: update your workflow using                      |                                                                                   |
|         |                        |                                | https://app.stepsecurity.io/secureworkflow/jab/bidict/release-to-pypi.yml/main?enable=permissions                     |                                                                                   |
|         |                        |                                | Warn: no topLevel permission defined:                                                                                 |                                                                                   |
|         |                        |                                | .github/workflows/release-to-test-pypi.yml:1: update your workflow using                                              |                                                                                   |
|         |                        |                                | https://app.stepsecurity.io/secureworkflow/jab/bidict/release-to-test-pypi.yml/main?enable=permissions                |                                                                                   |
|         |                        |                                | Warn: no topLevel permission defined: .github/workflows/test.yml:1: update your workflow using                        |                                                                                   |
|         |                        |                                | https://app.stepsecurity.io/secureworkflow/jab/bidict/test.yml/main?enable=permissions Warn: no                       |                                                                                   |
|         |                        |                                | topLevel permission defined: .github/workflows/update_actions.yml:1: update your workflow using                       |                                                                                   |
|         |                        |                                | https://app.stepsecurity.io/secureworkflow/jab/bidict/update_actions.yml/main?enable=permissions                      |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Vulnerabilities        | no vulnerabilities detected    |                                                                                                                       | https://github.com/ossf/scorecard/blob/main/docs/checks.md#vulnerabilities        |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|

In this case, it looks like it's actually the lack of pinning-by-hash in files like .github/workflows/test.yml (as opposed to pip requirements .txt files) that is causing this, but the same problem still applies: These are still all only things like test-time dependencies, and yet the penalty for not pinning-by-hash is as harsh as if they were runtime dependencies.

[raghavkaul]

I think this is a good fit for Annotations, a previously proposed feature that could in this case tell scorecard to skip certain files for the Pinned-Dependencies check.

[jab]

Maybe I'm misunderstanding, but rather than hard-coding a list of files to skip, ideally the tool would recognize that this is a Python package, and as such, look for its dependencies in standard locations, see e.g. https://packaging.python.org/en/latest/specifications/declaring-project-metadata/.

For my project, the tool should see dependencies = [] in my pyproject.toml (i.e. no runtime dependencies), and immediately know not to penalize my project as if it were depending on unpinned runtime dependencies.

[heitorlessa]

Had the same issue and trying to understand whether the scanner expects the use of --required-hashes argument in pip - is that right?

I'm using two variations in different dev deps and both got flagged:

• Dockerfile installing an immutable version: RUN python3 -m pip install git+https://github.com/pre-commit/pre-commit@5da4258b17dea7bd4601358de200e185699f9997
• Pip install from a list of dependencies and their hashes (pip-tools+pip): pip install -r /tmp/requirements.txt

[laurentsimon]

Had the same issue and trying to understand whether the scanner expects the use of --required-hashes argument in pip - is that right?

I'm using two variations in different dev deps and both got flagged:

• Dockerfile installing an immutable version: RUN python3 -m pip install git+https://github.com/pre-commit/pre-commit@5da4258b17dea7bd4601358de200e185699f9997

This one may be a false positive. iiuc, the @<hash> means it's effectively pinned by hash, correct? (One limitation is that dependabot and renovatebot won't understand it and won't send version bumps.)

• Pip install from a list of dependencies and their hashes (pip-tools+pip): pip install -r /tmp/requirements.txt

this one expects a --require-hashes

[heitorlessa]

I moved everything to --require-hashes to get the job done and just reached 9.1 ;)

Thanks a lot for confirming @laurentsimon!

---

Only two unrelated items left for us to get to 10 - separate to this regardless:

1. SAST - it's enabled for years but scorecard recognises 0 commits
2. Fuzzing - my fault as I'm lost in how to get it working for Python (remediation isn't helpful and Google docs are focused on C only tbh)

[laurentsimon]

@gabibguti fyi