Skip to content

security: untrack two committed key files and make secret scanning see the whole tree (#16275) - #16300

Merged
mrveiss merged 4 commits into
Dev_new_guifrom
issue-16275-secrets
Sep 11, 2026
Merged

mrveiss merged 4 commits into
Dev_new_guifrom
issue-16275-secrets

Conversation

@mrveiss

@mrveiss mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner

Closes #16275
Refs #16299

Two real secrets were tracked in this public repository, and every scanner missed both, for the same reasons. One scope, one PR: stop tracking committed key material, and make the scanning able to see it.

Thinking Path

  • .infra_encryption_key is a Fernet key generated by the infrastructure manager that was removed in January. It has been tracked since 2025-10-18. The owner confirmed it never protected real credentials, so it is untracked, with no rotation and no history rewrite (security: an auto-generated Fernet credential key has been tracked in this public repo since Oct 2025 #16275).
  • autobot-infrastructure/shared/config/service-keys/service-keys-20251005-220914.yaml holds six generated per-service keys. It slipped past .gitignore:420, whose root-level config/service-keys/ only matches the old path. The owner ruled untrack and rotate.
  • Why nothing caught either one:
    • security.yml's "Secret Detection" grepped three shapes (AKIA, PEM headers, password = '…' in *.py) in three directories.
    • Semgrep and Bandit are SAST tools scoped to the same three directories.
    • detect-secrets had a baseline, .secrets.baseline dated 2025-08-15 with zero results, but nothing ever ran it.
    • Measured with detect-secrets 1.5.0: even running, it reports 0 for the Fernet file, because its high-entropy plugins only read quoted strings. So a dependency-free shape detector is needed alongside it.

What Changed

  • Untracked: both files, with the local copies kept. .gitignore gains the real export path, autobot-infrastructure/shared/config/service-keys/.
  • detect-secrets wired for real:
    • A pre-commit hook, Yelp/detect-secrets pinned v1.5.0 with --baseline .secrets.baseline, covers all files.
    • A CI secret-detection job runs on every event, ungated, pinned ==1.5.0. Step 1 is the shape detector over git ls-files. Step 2 is a baseline comparison that fails on any finding not audited is_secret: false.
    • It prints only path, line and type, never a value, and captures status without a bare rc=$?.
  • .secrets.baseline regenerated and audited: 1,331 entries in 284 files, every one is_secret: false with a reason. They break down as 655 UI and i18n strings, 292 test fixtures, 252 doc examples and placeholders, 82 templates and prose, 26 checked individually, 21 identifiers, 2 documented template defaults and 1 parsing artefact. The six real entries left with their file.
  • pipeline-scripts/tracked_key_material.py and repo_tests/no_tracked_key_material_test.py:
    • No tracked file, extensionless dotfiles included, may contain Fernet-shaped key material or a PEM private-key header, beyond 4 named, reasoned exemptions.
    • .infra_encryption_key must stay untracked and ignored.
    • Planted positives and look-alike negatives are generated at test time.
    • The hook rev, the CI pin and the baseline version are pinned together.
  • service_auth role: deploy-keys.yml looked for exports under {{ playbook_dir }}/../../../infrastructure/.... That path no longer exists, since the directory was renamed, so it couldn't have been deploying from the committed file. It now reads service_auth_keys_source_dir (default {{ autobot.base_dir }}/config/service-keys, the directory rotate-service-keys.yml writes), and fails with a clear message when no export exists, instead of indexing an empty list.
  • docker/generate-secrets.sh now generates AUTOBOT_DB_PASSWORD and GRAFANA_ADMIN_PASSWORD. The tracked template defaults remain as they were, for the reasons in security: guessable default credentials ship in templates, ansible defaults and the compose file #16299 (a project hook blocks .env* edits, and the compose file can't be touched without its pre-existing port literals failing a strict hook).
  • A doc fix: docs/security/SERVICE_AUTH_ENFORCEMENT_ROLLOUT_PLAN.md carried a real-looking bypass-token value in a copyable command. It is now a placeholder.

Verification

  • AC1: both files are gone from the tree at this head. main follows at the next sync.
  • AC2: the guard's planted-key test covers an extensionless dotfile, plus an untracked-and-ignored assertion for .infra_encryption_key. The CI shape step is ungated. These have not run yet: CI is their first run, and no repo code was run locally.
  • AC3: the owner's answer is recorded on security: an auto-generated Fernet credential key has been tracked in this public repo since Oct 2025 #16275.
  • AC4: fixed (an ungated job and a wired hook) and explained (the shape detector covers what detect-secrets can't read).
  • Owner step after merge: rotate the six service keys with playbooks/rotate-service-keys.yml through the SLM. It writes to the now-ignored directory the deploy role reads.
  • Hooks: every pre-commit hook ran and passed on all three commits, the new detect-secrets hook included.

Model Used

Claude Opus 5 (claude-opus-5): coordinator session plus one implementation subagent.

…over the whole tree (#16275)

.infra_encryption_key, an auto-generated Fernet key from the removed
infrastructure host manager, had been tracked at the repository root since
2f60bc7 although .gitignore lists it. The owner confirmed nothing live was
ever encrypted with it, so it is untracked with no rotation and no history
rewrite. The .gitignore entry stays.

Why nothing caught it, and what catches it now:
- security.yml's "Secret Detection" step grepped three shapes inside three
  directories, so the root was never read. An ungated secret-detection job
  replaces it and scans every tracked file on every event.
- .secrets.baseline was orphaned: no hook or workflow ran detect-secrets. The
  detect-secrets hook (v1.5.0) now runs in pre-commit over every staged file.
  CI runs the same pinned version over the whole tree and fails on any finding
  the baseline does not hold with an audited not-a-secret verdict.
- Even when wired, detect-secrets 1.5.0 reports nothing for a bare key file,
  because its high-entropy plugins read quoted strings.
  pipeline-scripts/tracked_key_material.py checks that shape (Fernet keys, PEM
  private-key headers) in any tracked file. repo_tests/
  no_tracked_key_material_test.py runs it, plants a key in an extensionless
  dotfile, and pins the hook, the baseline version and the CI wiring.

Baseline: regenerated over every tracked file and audited entry by entry.
1331 entries are marked not-a-secret: UI labels, test fixtures, placeholders,
identifiers, doc examples, commit SHAs and checksums. Six entries, the
per-service keys in the tracked service-keys backup under
autobot-infrastructure/shared/config/, are marked real. They fail the gate on
purpose until the owner decides how that file is handled.

Weak defaults, partly done: docker/generate-secrets.sh now also generates
AUTOBOT_DB_PASSWORD and GRAFANA_ADMIN_PASSWORD, which override the template
values when docker/.env.secrets is passed after docker/.env.docker. The
shipped defaults stay for now. The project hook blocks edits to .env files,
and any edit to docker-compose.yml surfaces nine pre-existing port literals
that the hardcoded-values baseline cannot absorb for a touched file.

A rollout doc's example service-auth override token is replaced with a
placeholder, so it cannot be copied into a deployment.
…m the rotation directory (#16275)

The audit found a second real secret tracked in this public repository:
autobot-infrastructure/shared/config/service-keys/service-keys-20251005-220914.yaml,
six per-service keys. It slipped past .gitignore's root-level
config/service-keys/ rule, which only matches the old path. The owner ruled:
untrack and rotate.

- Untracked (the local file is kept), with the real export path added to .gitignore.
- The six is_secret: true baseline entries are removed with it.
- service_auth's deploy-keys.yml read keys from a repo-relative
  ../../../infrastructure/... path that no longer exists (the directory was
  renamed), so it could not have been finding this file. It now reads
  service_auth_keys_source_dir, the directory rotate-service-keys.yml writes,
  and fails with a clear message when no export exists instead of indexing an
  empty list.

Rotation is the owner's step, through rotate-service-keys.yml. The other
guessable defaults the audit found are #16299.
…s install root (#16275)

project_root is only a default inside three other roles, and none of the five
playbooks that run service_auth define it, so the previous commit's default could be
undefined. autobot.base_dir is set in inventory/group_vars/all.yml and already used
36 times across the roles.
@coderabbitai

coderabbitai Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 95579518-3063-41c5-a5b4-883215dcad4d


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Security review at e80f3aa6a2: changes requested. There's one in-scope guard gap and one tracking gap. The rest is sound.

# Severity Where Finding Fix
1 Medium security.yml, the detect-secrets step An empty rescan passes. If detect-secrets scan --baseline .secrets.baseline runs successfully but reads nothing, new is empty and the step exits 0. That could come from an exclude or filter in the baseline's settings widened to match everything, or from a plugin set that matches nothing. total is written to the step summary but never asserted. That's the #16275 failure mode itself: a scanner that saw nothing and reported clean. The shape detector already guards against it (no file was read -- a broken enumeration, not a clean tree, exit 2), but this step doesn't. Fail when the rescan's total is 0 while the committed baseline is non-empty. Better, fail when it drops below a floor declared next to the baseline (the committed count is 1,331), so a filter change that silently narrows the scan is a red, not a smaller number in the summary.
2 Medium Post-merge action Rotating the six service keys isn't tracked. Untracking service-keys-20251005-220914.yaml doesn't un-expose it: the keys stay in public git history. The owner's decision is "untrack and rotate after merge via rotate-service-keys.yml", but no open issue or acceptance criterion carries the rotation, and #16275's ACs cover only .infra_encryption_key. A host action with no issue is the kind that doesn't happen. File it before merge, with host evidence as its AC: every service authenticating with a newly generated key, and the old keys refused. If those keys are live today, the rotation doesn't need to wait for this PR.

Verified, and it holds:

  • Both files are untracked, and the export path is gitignored. test_the_committed_key_stays_untracked_and_ignored.
  • The shape detector reads the whole tree (git ls-files -z) on every event, deliberately without a path gate, since an extensionless root dotfile is what slipped through. A Fernet candidate must actually decode, so non-decoding lookalikes aren't flagged. PEM private-key headers are matched, and only paths and line numbers are printed. Tests: test_a_planted_key_in_an_extensionless_dotfile_is_flagged (security: an auto-generated Fernet credential key has been tracked in this public repo since Oct 2025 #16275 AC2), test_a_planted_pem_private_key_header_is_flagged, test_non_decoding_lookalikes_are_not_flagged.
  • Exemptions are exact counts with reasons. The four files hold format placeholders or truncated fixtures. A new match fails, a stale entry fails, and entries may only be removed.
  • The jq comparison fails closed on errors. Under set -euo pipefail, a malformed committed baseline or a failing scan aborts the step. Any baseline entry not marked is_secret: false fails, and a new finding is matched on detect-secrets' own identity (file, type, hashed secret), so a moved line stays known.
  • The baseline: 1,331 entries, 0 marked true, 0 unaudited, none on the removed key files. It's version 1.5.0, pinned equal to the hook and to the CI install by test_the_scanner_is_one_version_everywhere. The entries carry the verdict only; the detect-secrets format has no per-entry reason field.
  • service_auth reads only {{ autobot.base_dir }}/config/service-keys, never the repository, and fails with an explicit message when no export exists.
  • security: an auto-generated Fernet credential key has been tracked in this public repo since Oct 2025 #16275 AC4, the blind spot, is explained. The old step grepped three shapes in three directories and never read the root, and detect-secrets' entropy plugins read only quoted strings.

#16299 correctly takes the guessable template defaults that generate-secrets.sh now overrides.

…n the committed baseline holds (#16275)

The secret-detection job failed only on findings the committed baseline did
not hold. A rescan that read nothing -- a filter in the baseline's settings
widened to match every file, a plugin set that matches nothing, an empty file
list -- produced no new finding and exited 0, with its total only printed.
That is #16275's own failure mode: a scanner reporting clean on what it never
read.

pipeline-scripts/secrets_rescan_floor.py now asserts the rescan's count. The
step fails when the rescan reads zero findings against a non-empty committed
baseline, when it falls below FLOOR, or when the committed baseline itself
falls below FLOOR. FLOOR is 1,300: the committed baseline held 1,331 findings
on 2026-09-11. Only counts are printed, and the step captures the result with
`|| status=1`.

repo_tests/no_tracked_key_material_test.py loads the gate by path and checks:
- FLOOR sits at or under the committed count, within a 100-finding headroom;
- planted counts and planted baselines, including a malformed one that must
  raise rather than read as zero;
- the CI step invokes the gate against the rescanned and committed baselines.
@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Delta re-review since my changes-requested review at e80f3aa6a2: approved at 800aaccb35. It merges on a settled green at that SHA, and this head's run is the first execution of both the rescan and its floor. Finding 1 is fixed; finding 2 is tracked as #16301.

Finding Evidence at 800aaccb3
1. An empty rescan passed pipeline-scripts/secrets_rescan_floor.py (stdlib only, FLOOR = 1_300, with the measurement recorded: 1,331 findings in 284 files, 2026-09-11, detect-secrets 1.5.0). floor_problems fails in three cases: a rescan that read 0 while the committed baseline holds findings, a rescan below the floor, and a committed baseline below the floor, so removing audited entries is a deliberate floor change in the same diff. count_findings raises on anything that isn't a detect-secrets baseline, so a malformed or unreadable file can't read as zero. It prints counts only. security.yml runs it with both baselines and captures failure with … || status=1 (errexit-safe, no bare rc=$?). The print-only total is gone, and the step summary reports "rescan read N; committed holds M; floor F". Tests: test_the_rescan_floor_sits_just_under_the_committed_baseline (at or under the committed count, and at most 100 below it), test_the_floor_refuses_a_rescan_that_read_too_little (planted zero, narrowed, growth, below-floor and empty cases) and test_findings_are_counted_from_a_planted_baseline.
2. Rotation untracked #16301: compare each node's key by hash against the six public values, rotate via rotate-service-keys.yml through the SLM, show a request with an old key is refused, and keep the new export only in the ignored directory. All host-evidence ACs. Its body says rotation needn't wait for this merge if any node uses those keys.

Everything else from my first review stands. The branch is 0 behind.

@github-actions

Copy link
Copy Markdown
Contributor

✅ SSOT Configuration Compliance: Passing

🎉 No new hardcoded values of either class — ssot and other both block.

Known backlog in pipeline-scripts/hardcoded_values_baseline.txt is suppressed and tracked in #14371.

@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Correction to this PR's body, found while drafting closure evidence. The body says each of the 1,331 baseline entries is is_secret: false "with a reason". The tracked .secrets.baseline has no reason field: each entry carries only filename, hashed_secret, is_secret, is_verified, line_number and type. The reasons exist only as the category breakdown in this body.

That includes the template's weak default passwords, which the owner said must never be allowlisted silently. This body links them to #16299, but the tree doesn't. The fix is a tracked reason record, with those defaults marked as pending #16299, and it's now an acceptance criterion on #16299. This PR is frozen in merge train #16321, so the correction is recorded here rather than pushed.

mrveiss added a commit that referenced this pull request Sep 11, 2026
@mrveiss
mrveiss merged commit a12b4bf into Dev_new_gui Sep 11, 2026
75 of 78 checks passed
@mrveiss
mrveiss deleted the issue-16275-secrets branch September 11, 2026 13:46
mrveiss added a commit that referenced this pull request Sep 11, 2026
…ername fixture (#15758)

The whole-tree Secret Detection check (added by #16300) flagged a Secret Keyword in user_reserved_username_test.py: its fixture carried "password": "Str0ngPassw0rd". UserCreate.password is optional (SSO accounts) and these tests exercise only the username, so the fixture now omits it. No .secrets.baseline change, keeping clear of the #16343/#16349 line churn.
This was referenced Sep 11, 2026
mrveiss added a commit that referenced this pull request Sep 13, 2026
…tay contained (#16310)

Review findings, all fixed:

BLOCKING 1 -- marker collision with the SLM self-update (#12202). The
deletion pass wrote .deployed_commit, the exact file _get_slm_deployed_commit()
reads for the C4 self-update skip-gate -- a deletion pass that ran before
content actually synced made the gate see "already current" and skip the
real sync, deleting files with new code never copied in. Fixed:
- services/sync_deletions.py now owns its own marker, DELETION_MARKER
  (.autobot_sync_deletions_commit), and never touches .deployed_commit.
- _read_previous_commit reads the dedicated marker; if absent, it reads
  (never writes) the legacy .deployed_commit as a one-time bootstrap
  baseline, so the 16 known stale files are still caught on first run.
- The apply point moved out of _run_pull_stage (which ran before any
  content sync) to two call sites that already have a real per-component
  success signal: api/code_sync.py's _run_colocated_role_procedures (after
  run_role_full_procedure reports success -- new apply_role_deletions,
  mapping role name to drift_checker component, deliberately NOT
  role.source_paths/target_path, which serve a different deployed-path
  convention for several roles) and _sync_slm_from_code_source's per-component
  rsync loop (after that component's own rsync succeeds). remove_deleted_paths
  performs no gating itself -- a failed sync simply never calls it, so the
  tree and marker are untouched by construction.

BLOCKING 2 -- untracked-but-kept host state (#16300's pattern: a file
`git rm --cached` then gitignored, kept on disk, still shows as git-deleted).
New services/host_state_filter.py: a candidate matching
deploy_artifacts.HOST_STATE_EXCLUDES (minus HOST_STATE_REINCLUDES) or
reported ignored by `git check-ignore` at the new commit is kept, named, and
reported -- never silently dropped, never deleted.

MEDIUM 3 -- symlink containment. _unlink_one resolves the target and refuses
to unlink anything that resolves outside deployed_dir.

MEDIUM 4 -- drift classification. full_tree_drift.py's
_classify_deployed_only now runs the same host-state/ignored check BEFORE
the git-history check, so a kept-on-disk gitignored file reads as
host_state:gitignored, not removed_from_source (which would have steered an
operator into deleting a live key by hand).

Standards: services/sync_deletions.py and services/full_tree_drift.py split
into more, shorter functions (largest is now 34 lines, none exceed the
65-line hard limit). All new/touched modules use plain
logging.getLogger(__name__), matching
autobot_shared/user_management/password_epoch.py:50-58's precedent --
get_logger() builds a RotatingFileHandler that crashes at creation time
under the config-MagicMock harness tests/api/test_collect_outdated_node_ids.py
uses, since api/code_sync.py imports this module chain.

Remote-node scope: ansible `synchronize` still never deletes on fleet nodes.
Filed as a native GitHub sub-issue of #16310: #16322.

New tests:
- services/sync_deletions_test.py: never writes the legacy marker,
  bootstraps from it once then stops reading it, own marker wins once it
  exists, a git-rm-cached-then-gitignored file survives, a HOST_STATE_EXCLUDES
  match survives without needing .gitignore, a symlinked-parent escape is
  refused, apply_role_deletions' skip/report paths.
- services/full_tree_drift_test.py: a git-rm-cached-then-gitignored file
  reads as host_state:gitignored, not removed_from_source.
- tests/api/test_code_sync_colocated_role_deletion_16310.py: a successful
  role procedure triggers apply_role_deletions; a failed one, a no_playbook
  one, and one that raises never call it.
mrveiss added a commit that referenced this pull request Sep 13, 2026
…file (#16310)

Three failure classes from the first real pre-push run of these tests
(f3768b8 was blocked before any of them had actually executed):

1. repo_tests/sync_deletions_ansible_wiring_16310_test.py couldn't import
   services.deploy_artifacts -- repo_tests is a separate source root and
   cannot import SLM-backend packages. Replaced the import with
   _read_literal_string_collection(), an ast-based reader of ARTIFACT_DIRS/
   ARTIFACT_DIR_SUFFIXES's literal collection, asserting the shape rather
   than silently returning nothing if it ever stops being a literal.
   Fixing that import surfaced a second, previously-masked bug in the same
   test: `re.search(r"find .*?-prune", text, re.DOTALL)` matched from an
   EARLIER "find"/"-prune" mention in the file's own header prose (the
   newline-in-filename residual-limitation comment) instead of the real
   `find {{ sync_deletions_target_dir }} ... -prune` command, silently
   extracting zero -name tokens. Anchored the regex on the real invocation.

2. sys.modules leak guard failures in three files:
   - scripts/sync_deletion_planner_test.py and services/sync_deletions_test.py
     each installed services.git_tracker (a synthetic stub) unconditionally,
     with no restore. Folded into the same _SWAPPED/_prev_modules/finally
     cycle the six real-loaded modules already use.
   - services/full_tree_drift_test.py leaked "services" itself
     (synthetic conftest MagicMock -> genuine module, "exempt-refused:
     multi-source-root" -- "services" is a real, independently importable
     package under both autobot-backend/ and autobot-slm-backend/, and
     pytest's own package-aware collection of a test file living inside a
     real services/__init__.py package is what reaches for it). Captured
     and restored "services" and "services.git_tracker" in the same
     try/finally as the rest, for the same reason.
   No baseline entries added (the guard's baseline only shrinks).

3. A REAL BUG: services/full_tree_drift_test.py::
   test_a_git_rm_cached_then_gitignored_file_reads_as_host_state failed
   (assert None == 1, exclusions={}). Root cause: `_walk_checksums` is a
   raw filesystem walk of source_dir, not a git query, and `git rm --cached`
   never touches the working tree -- a file untracked-then-gitignored
   (#16300's pattern) stays physically present in the CONTROLLER's own
   code_source checkout with unchanged content. When that stale copy's
   checksum still matched the deployed copy, `_classify_all_paths` read the
   pair as identical and `continue`d, never reaching
   `_classify_deployed_only`, the git-history-aware check that would have
   called it host_state:gitignored. This is option 2 from the coordinator's
   list ("the file isn't reaching the deployed-only branch... counted as
   compared... earlier") -- specifically because it's counted as a false
   match, not merely pruned or skipped.

   Fixed with _source_ignored_paths(): one bulk `git ls-files --others
   --ignored --exclude-standard` call per component (not a per-file
   `git check-ignore` -- the round-3 bootstrap-enumeration-cost lesson),
   excluding source-side paths git ignores from src_checksums BEFORE the
   comparison runs, so a stale gitignored file on the controller's disk can
   never again read as "matches source". Verified against a real,
   disposable git repository (subprocess git, not the module under test)
   that this exact fixture now (a) has the pre-fix false-match confirmed,
   (b) the bulk ls-files call correctly names the one ignored path, and
   (c) git check-ignore itself returns 0 for it, matching what
   kept_reason()/is_git_ignored() would report.

   Proved the fix covers a real host layout, not just the fixture:
   services/sync_deletions_test.py::
   test_a_git_rm_cached_then_gitignored_file_is_kept is the same fixture
   shape run through compute_deletion_plan, and was never affected --
   sync_deletions.py's candidate list comes entirely from `git diff`/
   `git log` against repo_root, never from walking source_dir's raw disk,
   so kept_reason's gitignored check was always reachable there. Cross-
   referenced both tests so the distinction (and why one module had this
   bug and the other structurally couldn't) is discoverable from either
   side.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant