Repository navigation
security(service-auth): where should the service keys' master copy live? The plaintext all-keys export is sanctioned by no decision #16366
Description
Activity
Owner direction, 2026-09-11 (still under discussion, not final): leaning towards the system vault as the master copy of service keys, for scalable infrastructure, with a master (root) key rotation option designed in.
The sketch under discussion:
- Rotation and first install write the keys to the vault through the library, holding the root key, so they need no service key. That breaks the bootstrap cycle.
- Per-host
.envfiles are rendered from the vault, which stays within Umbrella/PRD: unify AutoBot secrets — one RBAC-governed store, vaults + one key abstraction (replaces 11 stores, 5 keys, 2 UIs) #10088's file-based bootstrap set. - The all-keys export file goes away.
- Root-key rotation uses versioned keys: every entry records its key version, reads accept both versions during the switch, and rotation re-encrypts everything and then retires the old key.
The open sub-decisions are where the root key lives (a host env file now, with a KMS later via #10088 Task 1.5), the rotation mechanism, the trigger, and the first-step scope. A read-only fact-finding pass on the envelope store's current key-versioning support is running. Interim: #16372 (the safe export) still merges, because it reduces the risk today.
Owner decision, 2026-09-11: the phased design is filed as #16406 (a sub-issue of #10088). Phase 1 is a versioned key ring plus bulk re-wrap, which gives root-key rotation. Phase 2 is a vault write path from the SLM host. Phase 3 is service keys mastered in the vault. KeePass (the owner's suggestion) is evaluated there, as the operator-held escrow and backup for the root key, not as the runtime store. #16372's hardened export is the interim until phase 3.
Closed — the question was answered and the work it produced is tracked elsewhere
This issue asked where the service keys' master copy should live. It is a
decide
issue, and the decision exists:Owner decision, 2026-09-11: the phased design is filed as #16406 (a sub-issue
of #10088).It carries no unticked acceptance criteria — its deliverable was an answer, and the
answer produced a design issue with a parent. That is the correct ending for a decision
issue: the ruling is recorded here, the engineering lives at #16406, and #10088 holds
the parentage so neither is orphaned.Why this stayed open is the general case, not an oversight. A
decideissue has no
code to merge, so nothing auto-closes it and no acceptance checklist goes green — the
mechanisms that close ordinary issues do not apply, and nobody is prompted. It has been
answered for three weeks. Tonight's triage of theneeds-decisionlabel found this shape
repeatedly: issues ruled on and left open because closing them is a separate manual act
nothing asks for.Also removing the
needs-decisionlabel, which is now false and makes the real queue of
unanswered questions look larger than it is.Found by the
autobot-marketing-a5triage of theneeds-decisionbacklog; the decision
quote re-verified here.
Decision needed (owner)
Found while scoping #16348. It asks where the service keys' master copy should live, given the standing rule that required credentials are stored in AutoBot's secrets management system.
How service keys flow today
None of the hops goes through the secrets manager. File:line references are to base.
generate_service_keys.pycallsServiceAuthManager.generate_service_key(autobot-backend/security/service_auth.py:55)service:key:{id}, 90-day expiry, which the verifier trusts (service_auth.py:192)_save_backup(generate_service_keys.py:~107)config/service-keys/service-keys-<ts>.yaml. This is the artifact that leaked (#16301). Rotation never prunes old exports.rotate-service-keys.yml:142-145runs the generator withchdir: {{ autobot_root }}(hard-coded/opt/autobot, :88)service_auth:service_auth_keys_source_dir(defaults/main.yml:25), thendeploy-keys.yml:35-75picks the newest YAML/etc/autobot/service-keys/<id>.env, mode 0600EnvironmentFile=;utils/service_client.py:341-367; the NPU worker reads its.envWhat's already decided: umbrella #10088's bootstrap constraint makes the per-host
.envfiles and the auth-bootstrap keys an irreducible file-based set, because you must authenticate before you can call the secrets service. So hops 4 and 5 are sanctioned.What isn't decided anywhere: the all-keys plaintext export (hop 2). #16300 and #16301 reused it without making a storage choice.
Options
/api/v2/secrets/system, the envelope store, Umbrella/PRD: unify AutoBot secrets — one RBAC-governed store, vaults + one key abstraction (replaces 11 stores, 5 keys, 2 UIs) #10088). The generator writes each key to the vault, and Ansible renders the 0600.envfiles from the vault rather than from a YAML export. That removes the plaintext master copy. But it conflicts with Umbrella/PRD: unify AutoBot secrets — one RBAC-governed store, vaults + one key abstraction (replaces 11 stores, 5 keys, 2 UIs) #10088's classification of service-auth keys as irreducibly file-based, so Umbrella/PRD: unify AutoBot secrets — one RBAC-governed store, vaults + one key abstraction (replaces 11 stores, 5 keys, 2 UIs) #10088's bootstrap section would need amending, e.g. vault for the master copy, files only on the hosts that use them. It also introduces a chicken-and-egg problem on first install, when there is no key to reach the vault yet..envfiles directly, with no file at rest outside/etc/autobot/service-keys/.Recommendation: land 1 now as #16348, since it stops the leak class whatever is decided, and take 3 as the target. It removes the plaintext master copy without the vault bootstrap problem, and it stays within #10088's recorded classification. Choose 2 if you want every credential's master copy in the vault regardless.
Also found (not a decision, but needs an owner)
slm-backend(generate_service_keys.py:43), but the role's service map (roles/service_auth/defaults/main.yml:30-42) has noslm-backendentry. That key is generated and exported but never deployed to a.envfile. Is it dead, or is a consumer missing?docs/developer/THREAT_MODEL.md§4 (secrets) mentions neither the envelope store nor service keys.Refs #10088, #16348, #16301, #16300, #12684