[Bug]: Time-to-Remediate not detecting threshold breach

[cbulleninf]

What happened?

There seems to have been a bug introduced somewhere between v2.9.1 and v2.11.

I have the below policy to flag critical alerts if they are not remediated after 3 days. I have a test repo with 2 critical dependabot alerts (and 8 others, 10 in total) which have been open for 2 months. If I run PaC on v2.9.1 it correctly errors/flags the threshold breach. If I run the exact same config/repo on v2.11 it passes the evaluation.

If I remove the "remediate" component from the config and just have the levels, then it does error. So the problem seems to be with the remediation feature.

If I look in the logs, it correctly identifies the 10 total dependabot issues, so it's not an issue about not being able to see the dependabot issues.

Policy:

general:
  remediate:
    critical: 3
  
codescanning:
  level: critical

dependabot:
  level: critical

licensing:

secretscanning:
  level: critical

Version

v2 (current major version)

Where are you experiencing the issue?

GitHub Actions

Relevant log output

Run advanced-security/policy-as-code@4914793a39812ef4674b9eda188d1dd480465f8e
Run echo "Running Policy as Code..."
Running Policy as Code...
/home/runner/work/_actions/advanced-security/policy-as-code/4914793a39812ef4674b9eda188d1dd480465f8e/ghascompliance/__version__.py:35: SyntaxWarning: invalid escape sequence '\|'
  """
INFO:root:GitHub Repository :: xxxxx
INFO:root:GitHub Instance :: https://github.com
INFO:root:GitHub Reference (branch/pr) :: master
INFO:root:Loading Policy as Code from Repository - xxxxxx
INFO:root:Cloning policy repo - xxxxx@test
INFO:root:Loading policy file - /tmp/repo/policies/test.yaml
INFO:root:Enabling Time to Remediate (global) :: codescanning
INFO:root:Enabling Time to Remediate (global) :: dependabot
INFO:root:Enabling Time to Remediate (global) :: licensing
INFO:root:Enabling Time to Remediate (global) :: secretscanning
INFO:root:Policy loaded successfully
INFO:root:Saving Policy...
INFO:root:Policy saved
INFO:root:Finished loading policy
INFO:root:Policy as Code Summary:
INFO:root:
INFO:root:general: 'error'
INFO:root:codescanning: 'critical'
INFO:root:dependabot: 'critical'
INFO:root:licensing: 'disabled'
INFO:root:secretscanning: 'critical'
INFO:root:dependencies: 'error'
INFO:root:
INFO:root:Total Code Scanning Alerts :: 1
INFO:root:Code Scanning violations :: 0
INFO:root:Total Dependabot Alerts :: 10
INFO:root:Dependabot violations :: 0
INFO:root:Total Dependency Graph :: 14
INFO:root:Dependency Graph violations :: 0
INFO:root:Total Secret Scanning Alerts :: 0
INFO:root:Secret Scanning violations :: 0
INFO:root:Total unacceptable alerts :: 0
INFO:root:Acceptable risk and no threshold reached.
 _____  _   _   ___   _____ _____                       _ _                      
|  __ \| | | | / _ \ /  ___/  __ \                     | (_)                     
| |  \/| |_| |/ /_\ \\ `--.| /  \/ ___  _ __ ___  _ __ | |_  __ _ _ __   ___ ___ 
| | __ |  _  ||  _  | `--. \ |    / _ \| '_ ` _ \| '_ \| | |/ _` | '_ \ / __/ _ \
| |_\ \| | | || | | |/\__/ / \__/\ (_) | | | | | | |_) | | | (_| | | | | (_|  __/
 \____/\_| |_/\_| |_/\____/ \____/\___/|_| |_| |_| .__/|_|_|\__,_|_| |_|\___\___|
                                                 | |                             
                                                 |_|         v2.11.0
    GitHub Advanced Security Policy as Code by GeekMasher

GitHub Repository :: xxxxx
GitHub Instance :: https://github.com
GitHub Reference (branch/pr) :: master
Policy as Code
Code Scanning Results
Dependabot Results
  Total Dependabot Alerts :: 10
  Dependabot violations :: 0
Dependency Graph
Dependency Graph Results - Licensing
Secret Scanning Results

Summary
  Total unacceptable alerts :: 0
  Acceptable risk and no threshold reached.

Code of Conduct

• I agree to follow this project's Code of Conduct

[cbulleninf]

I've tried to run both versions with debugging on. v2.9.1 doesn't really seem to provide any additional information. v2.11 however does seem to print out all the dependabot issues (all 10, including the criticals).

I've put an example of one of the critical lines below, interestingly there seems to be no creation date of the issue as part of this object - perhaps this is the issue? I took a quick look at the code and it seems like if no date is provided it assumes the issue is created at the time of the scan (which would comply with the 3 day requirement)

DEBUG:ghastoolkit.octokit.dependabot:Advisory :: Advisory(ghsa_id='ghsa-v778-237x-gjrc', severity='critical', aliases=[], summary='Misuse of ServerConfig.PublicKeyCallback may cause authorization bypass in golang.org/x/crypto', description='Applications and libraries which misuse the ServerConfig.PublicKeyCallback callback may be susceptible to an authorization bypass.\n\nThe documentation for ServerConfig.PublicKeyCallback says that "A call to this function does not guarantee that the key offered is in fact used to authenticate." Specifically, the SSH protocol allows clients to inquire about whether a public key is acceptable before proving control of the corresponding private key. PublicKeyCallback may be called with multiple keys, and the order in which the keys were provided cannot be used to infer which key the client successfully authenticated with, if any. Some applications, which store the key(s) passed to PublicKeyCallback (or derived information) and make security relevant determinations based on it once the connection is established, may make incorrect assumptions.\n\nFor example, an attacker may send public keys A and B, and then authenticate with A. PublicKeyCallback would be called only twice, first with A and then with B. A vulnerable application may then make authorization decisions based on key B for which the attacker does not actually control the private key.\n\nSince this API is widely misused, as a partial mitigation golang.org/x/crypto@v0.31.0 enforces the property that, when successfully authenticating via public key, the last key passed to ServerConfig.PublicKeyCallback will be the key used to authenticate the connection. PublicKeyCallback will now be called multiple times with the same key, if necessary. Note that the client may still not control the last key passed to PublicKeyCallback if the connection is then authenticated with a different method, such as PasswordCallback, KeyboardInteractiveCallback, or NoClientAuth.\n\nUsers should be using the Extensions field of the Permissions return value from the various authentication callbacks to record data associated with the authentication attempt instead of referencing external state. Once the connection is established the state corresponding to the successful authentication attempt can be retrieved via the ServerConn.Permissions field. Note that some third-party libraries misuse the Permissions type by sharing it across authentication attempts; users of third-party libraries should refer to the relevant projects for guidance.', url=None, cve_id='CVE-2024-45337', cwes=['CWE-285'], cvss={'vector_string': 'CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N', 'score': 9.1}, cvss_severities={'cvss_v3': {'vector_string': 'CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N', 'score': 9.1}, 'cvss_v4': {'vector_string': None, 'score': 0.0}}, epss={'percentage': 0.36067, 'percentile': 0.96904}, identifiers=[{'value': 'GHSA-v778-237x-gjrc', 'type': 'GHSA'}, {'value': 'CVE-2024-45337', 'type': 'CVE'}], references=[{'url': 'https://github.com/golang/crypto/commit/b4f1988a35dee11ec3e05d6bf3e90b695fbd8909'}, {'url': 'https://go.dev/cl/635315'}, {'url': 'https://go.dev/issue/70779'}, {'url': 'https://groups.google.com/g/golang-announce/c/-nPEi39gI4Q/m/cGVPJCqdAQAJ'}, {'url': 'https://pkg.go.dev/vuln/GO-2024-3321'}, {'url': 'https://nvd.nist.gov/vuln/detail/CVE-2024-45337'}, {'url': 'http://www.openwall.com/lists/oss-security/2024/12/11/2'}, {'url': 'https://security.netapp.com/advisory/ntap-20250131-0007'}, {'url': 'https://github.com/advisories/GHSA-v778-237x-gjrc'}], published_at='2024-12-11T22:03:04Z', updated_at='2025-01-31T15:30:44Z', withdrawn_at=None, affected=[{'package': {'ecosystem': 'go', 'name': 'golang.org/x/crypto'}, 'severity': 'critical', 'vulnerable_version_range': '< 0.31.0', 'first_patched_version': {'identifier': '0.31.0'}}])

[GeekMasher]

@cbulleninf Thanks for reporting this bug, I have found the issue but if you can please test the PR changes before I approve them.

#175

Usage:

- uses: advanced-security/policy-as-code@fix-time-to-remediate

[cbulleninf]

@GeekMasher - Thanks - with the new version, it now correctly picks up the threshold breach.

[GeekMasher]

v2.11.1 has the fix
https://github.com/advanced-security/policy-as-code/releases/tag/v2.11.1