Repository navigation
AUTOBOT_SECRETS_ROOT_KEY is generated by nothing — the canonical secret store is dark on every provisioned deployment #14758
Description
Activity
Fixed — verified on base
Merged via #14784 with 10/10 required checks and 12/12 python-suite shards
green. Verified againstorigin/Dev_new_gui, not the PR branch:Surface AUTOBOT_SECRETS_ROOT_KEYdocker/secrets-init.shpresent (generates) docker/with-secrets.shpresent (exports) docker/generate-secrets.shpresent (pinned override) slm-secrets.env.j2present backend.env.j2present Nested-brace occurrences on base: 0. Guard test present.
What was wrong
The key is mandatory —
load_root_keyraises when it is unset — and nothing
generated or wrote it, so the canonical envelope store was unreachable on every
provisioned deployment. What made it invisible rather than loud is that every
consumer swallows the failure:vault_clientdegrades to legacy-only, lifespan
hydration logs at debug,retrieve_secretreturnsNone. An entirely absent
secret store looked exactly like "that secret does not exist".Two constraints ruled out copying the sibling secrets, both verified before writing
anything:- The value is base64-decoded and must be exactly 32 bytes.
openssl rand -hex 32is 64 characters and decodes to 48 — rejected. 50,000 samples of the
chosen scheme decoded to exactly 32 bytes, with the hex form as a control that
must fail. - It derives every vault KEK, so regenerating it makes every already-sealed secret
undecryptable.secrets-init.shpreviously exited early once both signing
secrets existed, so an existing deployment could never have gained the root
key. It is now per-key, tested across fresh / re-run / upgrade:
(a) fresh -> all three generated (b) re-run -> all "already present — reusing"; file byte-identical (c) existing install -> JWT preserved: YES; root key added: 1The bug my own verification could not have caught
A security review found a nested
{{ }}inside an open{{ }}in the fresh-install
set_fact— aTemplateSyntaxErrorin the same multi-key task that produces
slm_secret_key,slm_encryption_key,slm_admin_password,slm_auth_tokenand
chromadb_auth_token. It would have aborted every greenfieldslm_manager
provision, a regression of working functionality rather than a failure to add the
new key. Upgrades were unaffected — they take thelineinfileback-fill.I had validated those files as YAML, and they were valid YAML. YAML validity
says nothing about Jinja validity, so my check was structurally incapable of
catching it. The guard now compiles the expressions with Ansible's filters stubbed,
and it is not vacuous:BROKEN (as shipped): TemplateSyntaxError -> expected token ':', got '}' FIXED: compiles expressions checked: 7 (asserts >= 3)Not covered
The variable is still not registered in
env_registry.py. The registry checker
scans.pyfiles only and cannot see the indirectos.environ.get(ROOT_KEY_ENV)
read, which is why base was green without it. Registering it requires regenerating
the env docs; that belongs in its own change rather than inside a security fix.The entropy inconsistency noted in review also stands:
openssl rand -base64 32
gives 256 bits while the ansible and busybox paths give ~191. Not a weakness at that
margin, recorded rather than silently accepted.This does not consolidate the secret stores — that is #12684's architecture
decision. It makes the canonical store reachable, which that decision depends on
either way.- The value is base64-decoded and must be exactly 32 bytes.
Summary
AUTOBOT_SECRETS_ROOT_KEYis mandatory for the canonical envelope secret store,and nothing generates or writes it. On any provisioned deployment the store
is therefore unreachable — and every consumer degrades silently.
Found while scoping #12684. Neither #12684 nor #12681 reports it, and it
invalidates the premise of both.
Evidence
The key is required:
Nothing provisions it. Searching the whole tree, the only occurrences outside
.py/.mdare generated artifacts:It is absent from
roles/backend/templates/backend.env.j2(which does provisionSECRET_KEY,AUTOBOT_JWT_SECRET,AUTOBOT_INTERNAL_API_KEYand others),from
install.sh, fromdocker/generate-secrets.sh, and from every playbook.The consequence is a clean 503:
Why it is invisible
Every consumer swallows it:
vault_clientcalls fail, sosystem_secrets_vault.retrieve_secretdegrades to legacy-only and warns only for non-404 errors.
initialization/lifespan.pyhydration catches broadly and logs at debug.retrieve_secretreturnsNoneonVaultClientError.So the canonical store being entirely absent looks identical to "that secret
isn't set".
LLC_SECRET_MASTER_KEYis provisioned nowhere either, so LLCcompany secrets are in the same state.
Why this is P0 rather than one item among many
It inverts the plan in #12684. That issue asks the SLM to adopt the
canonical store. Consolidating onto it today would migrate secrets into a store
that returns 503 — the migration would appear to succeed and the secrets would
be unreadable.
It is the only part of #12684 that #12681 needs. Getting a provider
credential to an adapter subprocess needs a working vault; it does not need the
SLM store migration or the UI unification.
The specific danger to get right
Generate once and persist. A task that regenerates on every deploy silently
destroys every secret already written —
DecryptionErroris raised deep andswallowed at both
lifespan.pyandretrieve_secret. Nothing would surface ituntil an operator noticed a credential had stopped working.
Escrow before first write. Root-key loss is unrecoverable for everything in
the envelope store. It must be in the backup set before anything is written to
that store, not after.
Acceptance criteria
idempotently — running the role twice leaves the value byte-identical,
asserted by a test rather than claimed
secrets, and included in the
/etc/autobotbackup archivedocs/runbooks/EMERGENCY_RECOVERY.mdrecords that loss is unrecoverableGET /api/v2/secrets/systemreturnssomething other than 503
"secret not found", rather than debug-logged (may fold into the
observability work in Umbrella: silent failures and telemetry gaps — 4 incidents where systemctl said active and the service was not working #13852)