Skip to content

fix(security): redact secrets in npu_client log, bound ReDoS regexes - #17104

Closed
mrveiss wants to merge 2 commits into
mainfrom
fix-codeql-clear-text-log-and-redos
Closed

mrveiss wants to merge 2 commits into
mainfrom
fix-codeql-clear-text-log-and-redos

Conversation

@mrveiss

@mrveiss mrveiss commented Sep 19, 2026

Copy link
Copy Markdown
Owner

Thinking Path

Fetched origin/main at the merge commit for #17077 (928573e3bb), then pulled the actual CodeQL SARIF for that PR's Python analysis (refs/pull/17077/merge, analysis id 1803753251, 3 results) rather than trusting only the alert summary — the org-wide /code-scanning/alerts endpoint hadn't indexed these yet (main's CodeQL scan for this exact commit hasn't run), so the SARIF was the only way to get the precise CodeQL data-flow trace and confirm these are real, not fabricated line/rule mappings.

The SARIF flow for Finding 1 showed generate_embedding_with_fallback's text parameter is reachable, at npu_client.py:471, both from a redacted fact-content path and (confirmed separately by grepping callers) from several unredacted callers (work_intent_similarity.py, semantic_search.py, vector_search_engine.py, code_intelligence/*) — so this isn't CodeQL missing an existing sanitizer, it's a genuine gap for callers that don't pre-redact.

For Findings 2/3, read both flagged regexes (_PEM_BLOCK_RE, _DATA_URI_RE) and reasoned through the backtracking: both have an unbounded quantifier ([\s\S]*?, [^,\s]+) that gets retried from every occurrence of the pattern's own anchor literal in adversarial input with no closing marker — O(n^2). Fixed by bounding both quantifiers to generous-but-finite limits, then verified empirically (see Verification) rather than assuming correctness.

What Changed

  • autobot-backend/services/npu_client.py: generate_embedding_with_fallback's failure-path error log now redacts text via autobot_shared.secret_redaction.redact_content() before truncating to 80 chars, instead of logging the raw caller-supplied prefix (CodeQL py/clear-text-logging-sensitive-data). Reused the existing security(secrets): credential redaction is name-keyed and cannot see credentials in free text — add content scanning before mail is indexed #13708 redaction utility rather than writing new masking logic, per repo convention.
  • autobot_shared/secret_redaction.py: bounded the two regex quantifiers CodeQL flagged as polynomial-ReDoS-vulnerable (py/polynomial-redos):
    • _PEM_BLOCK_RE: header class capped at 40 chars (real PEM key-type headers are well under 12), body capped at 16384 chars (~5x a real 4096-bit RSA key body, ~2.5x an 8192-bit one).
    • _DATA_URI_RE: media-type span capped at 255 chars (real MIME types, even long ones, are well under that).

Verification

  • Reconstructed both regexes standalone (not importing repo code) and ran every _TRUE_POSITIVES/_FALSE_POSITIVES fixture from secret_redaction_test.py through old vs. new: all 16 cases match (old == new match presence for every fixture, for both regexes) — no behavior change for real credentials or ordinary text.
  • Benchmarked the new bounded PEM regex against adversarial input ("-----BEGIN PRIVATE KEY-----" * n, no closing marker) at n=2500/5000/10000/20000: time scales linearly with input size (roughly doubles when input doubles), confirming the fix changes the complexity class from O(n^2) to O(n) rather than just shrinking the constant.
  • Did not run the repo's own test suite locally per policy; instead let the pre-push hook run it — pytest on autobot_shared/secret_redaction_test.py passed on push, independently confirming the equivalence analysis above.
  • Confirmed via gh api that generate_embedding_with_fallback has unredacted callers beyond the fact-content path, so Finding 1 is a real gap, not a CodeQL false positive against an already-sanitized value.

Residual note (not fixed here, out of scope for this PR): the direct callers of redact_content/scan_content_for_credentials via knowledge_graph_routes.py → pipeline/runner.py have no upstream content-length cap (unlike the bulk.py fact-content path, which caps at 100000 chars) — so a very large adversarial document on that path would still take non-trivial (linear, not quadratic) time. Flagging for awareness; happy to file a follow-up issue if wanted.

Single-issue rationale

Both fixes are the direct, minimal remediation for the 3 CodeQL alerts introduced by the same merge (#17077) and land together because they're the same security gate blocking main — no unrelated changes bundled in.

Model Used

Claude Sonnet 5

…le regexes

CodeQL flagged 3 high-severity alerts introduced by the #17077 merge:

- py/clear-text-logging-sensitive-data (npu_client.py:471): the error log
  on embedding-generation failure logged a raw prefix of caller-supplied
  text. generate_embedding_with_fallback() is a generic low-level client
  called from several places (work_intent_similarity, semantic_search,
  vector_search_engine, code_intelligence) that don't all redact before
  embedding, so a failed call could log real secret material. Now redacts
  via autobot_shared.secret_redaction.redact_content() before truncating.

- py/polynomial-redos x2 (secret_redaction.py:325, :347): the PEM-block and
  data-URI matchers used unbounded quantifiers ([\s\S]*? and [^,\s]+) whose
  lazy/greedy scan restarts at every occurrence of the anchor literal in the
  input, giving O(n^2) worst case on adversarial content (many "-----BEGIN
  ... PRIVATE KEY-----" or "data:" occurrences with no closing marker) --
  ironic since this module itself scans arbitrary KB/fact content. Bounded
  both quantifiers to generous but finite limits that comfortably cover real
  PEM key bodies (up to ~8192-bit RSA) and real MIME types, verified linear
  (not quadratic) scaling and full behavioral equivalence against every
  true-positive/false-positive fixture in secret_redaction_test.py.
@coderabbitai

coderabbitai Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 42 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: mrveiss/AutoBot-AI/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 26f628b3-b60d-49b9-aa5c-a6c7f99e5de4

📥 Commits

Reviewing files that changed from the base of the PR and between 928573e and b5d9c38.

📒 Files selected for processing (2)
  • autobot-backend/services/npu_client.py
  • autobot_shared/secret_redaction.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…og path (#17104 review)

redact_content(text)[:80] ran every redaction regex over the full
caller-supplied text (up to ~100 KB) on an unconditional logger.error
in the embedding-failure path. Slice to 512 chars before redacting --
well over the final 80-char prefix, so a credential crossing that
boundary is still masked, but the regex cost is now bounded.
@mrveiss mrveiss added the land-next Landing set: CI capacity goes to these PRs first (owner direction 2026-09-18, #15397) label Sep 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Notice: 29 open PRs — past the runaway threshold (25)

There is no PR queue limit, and this is not a request to defer this PR. Work proceeds one issue at a time without a cap on open PRs; review capacity is the constraint.

This notice only means the count is high enough to be worth a glance for a runaway — something opening PRs in a loop, or a merge pipeline that has stalled so nothing is draining.

Currently open:

If the queue is draining normally, ignore this. Otherwise:

  1. Check whether CI is dispatching at all — see the ci-dispatch-watchdog status on these PRs
  2. Merge the ones whose CI has finished and review has passed: gh pr merge <number> --squash --delete-branch
  3. Look for a loop opening near-identical PRs

Warn-only runaway detector — .github/workflows/pr-queue-gate.yml. It never blocks a merge.

model_name,
last_reason,
text[:80],
redact_content(text[:512])[:80],
@mrveiss

mrveiss commented Sep 19, 2026

Copy link
Copy Markdown
Owner Author

Carried by #17116 (cb492ba0e314fb0744d47c709e8cdfbae255374c). Closing as superseded, branch kept.

@mrveiss mrveiss closed this Sep 19, 2026
@mrveiss mrveiss removed the land-next Landing set: CI capacity goes to these PRs first (owner direction 2026-09-18, #15397) label Sep 19, 2026
mrveiss added a commit that referenced this pull request Sep 19, 2026
mrveiss added a commit that referenced this pull request Sep 19, 2026
…ason

- role_registry.py: 727 -> 715 lines. Collapses the slm-backend #16889
  comment from 9 lines to 3 and merges it with the adjacent pre-existing
  #14275 comment, and tightens the post_sync_cmd string layout -- no
  behavior change, back under the grandfathered 715-line ceiling (#14236).
- secrets_baseline_reasons.py: the Basic Auth Credentials entry for
  autobot_shared/secret_redaction.py cited a bare "line 242", which drifts
  whenever a regex rewrite upstream (like this vehicle's own #17104 member)
  moves the comment -- reworded to cite the comment above
  `_BASIC_AUTH_URL_RE` by name instead.
- glob_declared_reads_15900_test.py: the main merge below brought in
  #15317's chromadb_bind_not_hardcoded_15317_test.py, whose *.service,
  *.service.j2 and docker/*.yml glob declarations were undetected by that
  guard's own coverage sweep -- recorded all three (plus adding the guard
  to the pre-existing *.sh entry's set) so the pre-push hook's
  glob-declared-reads check passes.

Also merges origin/main (143 commits) to bring this code-touching PR to
exactly 0-behind, per the owner's merge-safety rule.
mrveiss added a commit that referenced this pull request Sep 20, 2026
chore(vehicle): land the security batch — ReDoS/log redaction (#17104) and unicode injection matching (#16354)
mrveiss added a commit that referenced this pull request Sep 22, 2026
…dependabot uv) (#17116)

* fix(ci): give the SLM backend's tests their own venv in CI (#16394)

python-shard shared one venv between autobot-backend and
autobot-slm-backend's pytest invocations, built from requirements-ci.txt --
autobot-backend/requirements.txt's langgraph-sdk caps websockets<16, while
autobot-slm-backend/requirements.txt declares websockets>=17.1,<18 (SLM's
real production version). SLM's tests always ran against 15.0.1 as a result,
and #16391 carried a named exemption in check_dependency_floors.py's
KNOWN_CROSS_VENV_EXEMPTIONS pending a real fix.

setup-python-suite/action.yml now builds a second, parallel venv from
autobot-slm-backend/requirements.txt + requirements-ci-test.txt, exposed via
a new `slm-venv` output (not activated onto $GITHUB_PATH, reached explicitly
so it coexists with the backend venv). ci.yml points the slm-backend pytest
invocation at that venv's own interpreter, and splits the strict
dependency-floor check into two --roots-scoped calls (one per venv, mirroring
scripts/setup-ci-parity-env.sh's existing pattern) so each service's floors
are judged against what its own venv actually installs. The
KNOWN_CROSS_VENV_EXEMPTIONS entry is removed and pinned empty by a test.

* fix(ci): python-paths covers the shell wrappers python-suite tests (#16249)

`pipeline-scripts/check-pre-commit-hook-pr.sh` is run by two python-suite
tests, and no pattern in `.github/filters/python-paths.yml` matched it. A change
confined to the wrapper computed `python != 'true'`, the required-context shim
reported `python-suite` green, and the tests that exercise the change never ran
— a guard bypassable by touching the one file it exists to watch.

Six paths, not the one reported: four wrappers with sibling tests
(check-pre-commit-hook-pr.sh, check_baseline_no_growth.sh,
detect-hardcoded-values.sh, pr-queue-open-list.sh) and the two libraries they
source (scripts/lib/git-scope.sh, scripts/lib/hardcoded-value-rules.sh).

Both extra findings came from the new guard failing against my own enumeration,
before review rather than after:

  * I paired scripts to tests by NAME, which crosses a hyphen/underscore
    boundary silently — pr-queue-open-list.sh is tested by
    pr_queue_open_list_test.py. The guard pairs them by CONTENT and named it.
  * my source-line pattern matched the quoted argument as `[^"]+`, which stops
    at the first INNER quote. The portable spelling nests them —
    `source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../x.sh"` — so it
    yielded `$(cd `, ended in no `.sh`, and was dropped as if the line sourced
    nothing. Only the simple `"${REPO_ROOT}/..."` spelling parsed, which is how
    hardcoded-value-rules.sh — sourced by two of the four — surfaced at all,
    and how git-scope.sh then read as absent. The extractor now matches the
    `.sh` argument itself, last one on the line.

Both variables are assigned `$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)`,
checked in each script rather than inferred from the name. The transitive
closure was checked too, per the issue's "not verified" note: neither library
sources anything, though both look like they might — git-scope.sh contains the
word `source` four times and hardcoded-value-rules.sh six, and every occurrence
is prose in a comment. An anchored grep returned zero for the first; counting
the unanchored word is what made that zero trustworthy.

Why the existing coverage guard cannot catch this, which the issue did not
record. `python_filter_covers_its_guards_test.py` exists for exactly this class
and two independent things stop it: it sweeps `repo_tests/` only, and
`pipeline-scripts`, `scripts` and `tools` sit in its `_NOT_A_READ` set — whose
stated premise, "trees whose contents no guard reads directly", is false, since
check-pre-commit-hook-pr_test.py reads its own `.sh`. Sweeping those directories
would still discard the finding.

Widening it was measured, not guessed. The sweep was reimplemented independently
and validated against the recorded baseline (38 uncovered reads, equal to
MAX_UNCOVERED_READS and to the 38 recorded entries) before any option was
judged:

  today                                         38
  also sweep pipeline-scripts/                  58
  ...and drop pipeline-scripts from _NOT_A_READ 63
  ...and drop scripts too                       77

Every option raises a count whose own rule is that it may only fall, and the 25
newly surfaced reads are mostly workflow and config files — a different
population from this defect. So the narrow invariant is asserted directly in
`repo_tests/python_filter_covers_tested_shell_wrappers_test.py`: a wrapper that
python-suite tests must itself be able to trigger python-suite. Exact, and it
costs the ratchet nothing — re-measured after the filter change, still 38, so
the equality rule holds in both directions.

That guard imports the matcher from the existing one rather than restating it.
Two matchers drifting apart would disagree silently and certify coverage the
real gate does not grant; a rename breaks the import loudly instead.

Four of its six tests are controls: the sweep must find at least four tested
wrappers and name the reported one; the extractor is pinned to exact sourced
sets for all four real wrappers, including the empty one, which is what would
catch an extractor that started matching prose; an argument that does not
resolve to a repository file is REPORTED rather than skipped, so an unreadable
source line cannot look like a script that sources nothing; and the imported
matcher is asserted to say no to an absent path, without which both findings
tests pass vacuously.

Closes #16249

* test(guards): record the new wrapper guard's `*.sh` glob (#16249)

`python-suite shard 8/12` failed on
`glob_declared_reads_15900_test.py::test_the_record_names_exactly_the_guards_that_declare_each_glob`:

  declaring-but-not-recorded:
  ['repo_tests/python_filter_covers_tested_shell_wrappers_test.py']

The guard added by this PR sweeps `pipeline-scripts/*.sh`, and #15900's record
names every guard that declares each tracked glob so a sweep cannot quietly
appear or vanish. A new declarer has to be added in the same change — which is
the record doing its job, not an obstacle.

Only `*.sh` needed recording. The guard also calls `glob("*_test.py")`, and that
pattern is not tracked: the record carries `*_test.sh` but no `*_test.py`.

Checked the two other guards this session added for the same trap —
`command_position_scan_test.py` and `hook_declarations_are_reachable_test.py` —
and neither globs at all; they read a module's `dir()` and the router's exported
routes respectively. So this is the only entry needed.

* fix(ci): declare numpy/jinja2/aiosqlite the SLM venv now needs for real (#16394)

Giving autobot-slm-backend's tests their own venv (this same PR) surfaced 3
ModuleNotFoundErrors across every shard's SLM test run -- numpy, jinja2, and
aiosqlite were never declared in autobot-slm-backend/requirements.txt, only
ever present because CI's OLD shared venv installed them transitively for the
backend's own needs (aiosqlite>=0.22.1 is autobot-backend/requirements.txt's
own pin; numpy/jinja2 arrived from some other backend dependency's own
closure, never declared directly there either).

tests/test_bi_dashboard.py needs real numpy (documented in its own comment,
no stub) and falls back to a jinja2 stub only when the real package is
genuinely absent; migrations/migrate_system_secrets_to_vault_test.py and
user_management/services/sso_e2e_test.py need the real `sqlite+aiosqlite://`
SQLAlchemy driver for in-memory async DB tests.

numpy pinned via the existing `-c ../constraints/shared.txt` convention
(autobot-backend/requirements.txt:9's same mechanism), not a local version --
this dev box's Python 3.10.12 cannot resolve constraints/shared.txt's
numpy>=2.5.3 floor (needs 3.11+), so that one is unverifiable locally; CI
runs the declared 3.14.7. jinja2 and aiosqlite verified installable in an
isolated venv on this box (jinja2 3.1.6, aiosqlite 0.22.1).

* fix(ci): mirror SLM's jinja2 into CI and fix its Docker constraints copy (#16394)

* fix(tests): use the canonical repo_root() helper, not a hand-derived one (#16249)

* chore(deps): bump the uv group across 1 directory with 2 updates

Bumps the uv group with 2 updates in the /autobot-infrastructure/shared/mcp/tools/knowledge-base-mcp directory: [anyio](https://github.com/agronholm/anyio) and [httpx2](https://github.com/pydantic/httpx2).


Updates `anyio` from 4.12.1 to 4.14.2
- [Release notes](https://github.com/agronholm/anyio/releases)
- [Commits](agronholm/anyio@4.12.1...4.14.2)

Updates `httpx2` from 2.10.0 to 2.12.0
- [Release notes](https://github.com/pydantic/httpx2/releases)
- [Changelog](https://github.com/pydantic/httpx2/blob/main/src/httpx2/CHANGELOG.md)
- [Commits](pydantic/httpx2@v2.10.0...v2.12.0)

---
updated-dependencies:
- dependency-name: anyio
  dependency-version: 4.14.2
  dependency-type: indirect
  dependency-group: uv
- dependency-name: httpx2
  dependency-version: 2.12.0
  dependency-type: indirect
  dependency-group: uv
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix(tests): import Path in python_filter_covers_tested_shell_wrappers_test.py (#16249)

pyflakes F821: Path is used in 3 type annotations (dict[Path, list[str]],
def _sourced_paths(script: Path)) but never imported. from __future__ import
annotations defers evaluation at runtime, but pyflakes still resolves
annotation strings statically. Pure import addition, no logic change.

* fix(security): redact secrets in npu_client log, bound ReDoS-vulnerable regexes

CodeQL flagged 3 high-severity alerts introduced by the #17077 merge:

- py/clear-text-logging-sensitive-data (npu_client.py:471): the error log
  on embedding-generation failure logged a raw prefix of caller-supplied
  text. generate_embedding_with_fallback() is a generic low-level client
  called from several places (work_intent_similarity, semantic_search,
  vector_search_engine, code_intelligence) that don't all redact before
  embedding, so a failed call could log real secret material. Now redacts
  via autobot_shared.secret_redaction.redact_content() before truncating.

- py/polynomial-redos x2 (secret_redaction.py:325, :347): the PEM-block and
  data-URI matchers used unbounded quantifiers ([\s\S]*? and [^,\s]+) whose
  lazy/greedy scan restarts at every occurrence of the anchor literal in the
  input, giving O(n^2) worst case on adversarial content (many "-----BEGIN
  ... PRIVATE KEY-----" or "data:" occurrences with no closing marker) --
  ironic since this module itself scans arbitrary KB/fact content. Bounded
  both quantifiers to generous but finite limits that comfortably cover real
  PEM key bodies (up to ~8192-bit RSA) and real MIME types, verified linear
  (not quadratic) scaling and full behavioral equivalence against every
  true-positive/false-positive fixture in secret_redaction_test.py.

* fix(perf): bound redact_content's input on npu_client's hot failure-log path (#17104 review)

redact_content(text)[:80] ran every redaction regex over the full
caller-supplied text (up to ~100 KB) on an unconditional logger.error
in the embedding-failure path. Slice to 512 chars before redacting --
well over the final 80-char prefix, so a credential crossing that
boundary is still masked, but the regex cost is now bounded.

* fix(security): stop logging redacted-but-content-derived text on the embedding-failure path

CodeQL py/clear-text-logging-sensitive-data does not model redact_content()
as a sanitizer, so a redacted excerpt still trips the query. Log no
input-derived content at all: length plus a truncated sha256 hash is
enough to correlate failures without ever reproducing what was sent.
Drops the now-unused redact_content import.

* fix(ci): route slm-backend's post_sync_cmd through build-filtered-requirements.sh (#16889 review)

#16394 gave autobot-slm-backend/requirements.txt its own sibling-relative
-c ../constraints/shared.txt include, which the deployed directory can't
resolve. A bare pip install -r aborted the sync, and the && silently
skipped alembic upgrade head (#11069/#14272) -- the same failure mode
already fixed for the backend and ai-stack roles. Routes through the
same canonical script so all three deploy paths share one implementation.

* fix(ci): give slm-backend's filtered-requirements temp file its own name (#16889 review)

The previous fix copied the backend role's post_sync_cmd pattern
verbatim, including its /tmp/requirements-filtered-slm.txt output path
-- already used by the backend role itself. code_sync runs sync jobs
as concurrent tasks, so on a node hosting both roles the two jobs can
interleave and one filter's output gets pip-installed into the other
role's venv: a silent wrong-deps install, not a visible failure.

Adds a regression test asserting every role's filtered-output /tmp
path is unique, verified to catch the exact collision by reverting
locally and confirming it fails, then restoring.

* fix(vehicle): trim role_registry.py to ceiling, reword line-number reason

- role_registry.py: 727 -> 715 lines. Collapses the slm-backend #16889
  comment from 9 lines to 3 and merges it with the adjacent pre-existing
  #14275 comment, and tightens the post_sync_cmd string layout -- no
  behavior change, back under the grandfathered 715-line ceiling (#14236).
- secrets_baseline_reasons.py: the Basic Auth Credentials entry for
  autobot_shared/secret_redaction.py cited a bare "line 242", which drifts
  whenever a regex rewrite upstream (like this vehicle's own #17104 member)
  moves the comment -- reworded to cite the comment above
  `_BASIC_AUTH_URL_RE` by name instead.
- glob_declared_reads_15900_test.py: the main merge below brought in
  #15317's chromadb_bind_not_hardcoded_15317_test.py, whose *.service,
  *.service.j2 and docker/*.yml glob declarations were undetected by that
  guard's own coverage sweep -- recorded all three (plus adding the guard
  to the pre-existing *.sh entry's set) so the pre-push hook's
  glob-declared-reads check passes.

Also merges origin/main (143 commits) to bring this code-touching PR to
exactly 0-behind, per the owner's merge-safety rule.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants