Skip to content

security(redis): P1b — SLM-generated Redis password, every consumer authenticates as default while the server stays nopass (#13568) #16627

Description

@mrveiss

Scope

Phase P1b of #13568, which follows P1a.

Acceptance criteria

Activity

  1. mrveiss commented on Sep 13, 2026

    @mrveiss
    OwnerAuthor

    Requirement for this phase, from the #16626 review: roles/backend/templates/backend.env.j2 puts the Redis password, and since #16626 the username, into AUTOBOT_REDIS_URL without URL-encoding.

    Today that is harmless. The password is empty, and AUTOBOT_REDIS_URL is only checked for presence (autobot-backend/startup_validation.py:40); no client connects with it. Once this phase renders the SLM-generated password there, a :, @ or / in the password breaks the URL.

    Do one of the following:

    • generate the password from a URL-safe alphabet, such as hex or secrets.token_urlsafe;
    • encode both fields in the template;
    • drop the composite URL, if its presence check can take the separate fields instead.

    ssot_config.redis_url_with_auth already encodes both credentials as of #16626.

  2. mrveiss commented on Sep 13, 2026

    @mrveiss
    OwnerAuthor

    Owner decisions for P1b, 2026-09-13:

    • The secret is read on the SLM host. The shared read task reads the SLM-generated password from the SLM's own secrets file on the controller, once per run. The SLM runs every playbook, so every node gets the same value, remote nodes included. This departs on purpose from the security(chromadb): no CHROMA_SERVER_AUTHN configured — any internal-network container can reach chroma unauthenticated #12513 chroma pattern, which reads on each target and so only reaches hosts that carry the file. When a run has no secrets file, nothing is adopted and nothing changes.
    • Windows NPU worker delivery is automated in P1b. The worker reads its Redis credentials from a local bootstrap config, and nothing delivered them until now. This PR adds that delivery path, so the worker also authenticates as default with the password.
  3. mrveiss commented on Sep 13, 2026

    @mrveiss
    OwnerAuthor

    Owner decision, 2026-09-13, which refines the Windows NPU answer above: P1b strips the Redis password from the unauthenticated POST /api/npu/workers/bootstrap response (#16657). Windows workers get default plus the password through the admin-only pair and repair push. An authenticated bootstrap now blocks P2 (#16628).

  4. mrveiss commented on Sep 13, 2026

    @mrveiss
    OwnerAuthor

    Owner decision, 2026-09-13, which refines the Windows NPU answer again: the worker ignores a Redis config pushed at pairing, and wiring it up while /pair is unauthenticated (#16658) would let anyone who can reach a worker redirect its Redis. So P1b adds the default username to the admin pair/repair payloads and changes no worker code. Windows workers keep connecting without Redis auth, which is fine while the server stays nopass. Delivery moves to #16658, which now blocks #16628.

  5. mrveiss commented on Sep 14, 2026

    @mrveiss
    OwnerAuthor

    AC evidence against origin/main after #16660 (07815fd) and #16679 (a494d4c). Two of four ticked.

    AC status evidence / gap
    The SLM generates the password once and never regenerates or blanks it. Roles read it and never generate it. ticked autobot-slm-backend/ansible/roles/slm_manager/tasks/main.yml:429-432 keeps a non-empty slm_redis_password and otherwise calls lookup('password', '/dev/null length=48 chars=ascii_letters,digits'). slm-secrets.env is templated only when: not slm_secrets_stat.stat.exists (:452, :463). For an existing file, the key is back-filled only when grep -q '^AUTOBOT_REDIS_PASSWORD=.\+' (:540) fails (when: … _slm_redis_pw_check.rc != 0, :581-583). Test: tests/test_redis_password_provisioned_16627.py:152 test_no_consumer_generates_a_redis_password.
    Every consumer in the #13568 inventory authenticates as default with the password not ticked Not every consumer is verified. The deploy playbooks (inventory 10–11) are the open follow-up #16678. The NPU worker's credential path (inventory 4) is #16657, which is open. I could not determine the redis-replication role's password source (inventory 8).
    SLM replication and backup run redis-cli --user with the canonical password ticked services/replication.py:649 and :772 auth = redis_cli_auth(redis_password), with the password from :488 ssot_config.redis.password. services/backup.py:528 auth = redis_cli_auth(). services/redis_cli_auth.py:46 falls back to ssot_config.redis.password, and :49 sets username = ssot_config.redis.username or "default". The source is canonical: slm-secrets.env.j2:38 AUTOBOT_REDIS_PASSWORD={{ slm_redis_password }} is loaded by autobot-slm-backend.service.j2:48 EnvironmentFile= (default slm-secrets.env, defaults/main.yml:20-21) and read at autobot_shared/ssot_config.py:762 alias="AUTOBOT_REDIS_PASSWORD". Test: tests/test_redis_cli_auth_16627.py:51 asserts redis-cli --user default INFO replication (:58). Out of scope: backup's verify command at backup.py:327 (redis-cli PING && redis-cli DBSIZE) sends no auth. It is one of the six no-auth sites that #13568 assigns to #16628.
    Host evidence: every consumer authenticates against the still-nopass server not ticked Needs host evidence. Code cannot show it.

    Found during this audit: #16686. deploy-hybrid.sh:151 and deploy-native.sh:175 write a literal Redis password with no username.

  6. added this to the v0.11.0 milestone on Sep 14, 2026
  7. mrveiss commented on Sep 20, 2026

    @mrveiss
    OwnerAuthor

    Checked the 4 local branches assigned against this issue (pr16660latest3, pr16660new, pr16660review, pr16660latest2) — all four are stale local snapshots of PR #16660, which already merged 2026-09-14. Ran the file-level supersession test: every core file they introduce (services/redis_cli_auth.py, ansible/_shared/tasks/read_redis_password.yml, services/backup.py, services/replication.py, tests/test_redis_cli_auth_16627.py, the changelog entry) is byte-identical between the branch tips and origin/main. The ansible role files that differ do so only from ordinary later drift — spot-checked roles/slm_manager/tasks/main.yml (the largest diff) and confirmed the slm_redis_password logic is still there, just reformatted/extended since. Nothing to consolidate; none of the four add anything main doesn't already have. Not reviving any of them.

    On the two open ACs from the 2026-09-14 audit comment:

    AC2 (every #13568-inventory consumer authenticates as default) — resolved the one item that comment left as "could not determine": inventory item 8, the redis-replication role, has no password source because it is not wired into anything. Grepped every playbook's roles: list and every include_role/import_role/string reference across autobot-slm-backend/ansible/ for redis-replication/redis_replication — it appears nowhere outside its own role directory. provision_host.yml/recover_host.yml both use role: redis (the regular role, already fixed here), never redis-replication. So this isn't an undetermined password source, it's an unwired role that never runs. Filed #17176 rather than guessing at whether it should be wired in or retired — that's an owner call. Items 10-11 (deploy playbooks, #16678) and item 4 (NPU worker, #16657) are still tracked as their own open issues, unchanged by this pass.

    AC4 (host evidence) — still needs the owner or a live host; nothing I can add by reading source.

    Not closing — two ACs remain genuinely open, tracked precisely by #16678, #16657, and now #17176.

  8. mrveiss commented on Sep 25, 2026

    @mrveiss
    OwnerAuthor

    Blocked by #17434 — this issue's mechanism does not reach the server.

    Host evidence from a role-provisioned node: the running Redis Stack reports config_file:/opt/redis-stack/etc/redis-stack.conf (23 bytes, port 6379 + daemonize yes, no include), while roles/redis/tasks/main.yml:162 renders its 2933-byte config to /etc/redis-stack.conf. The unit's ExecStart passes no config argument, so the rendered file is never read.

    redis-cli CONFIG GET requirepass  ->  ""
    

    So requirepass is currently empty on a node where the role ran, and rendering it into /etc/redis-stack.conf — which is what this issue's approach does — would not change that. The path question in #17434 has to be settled first, or a green render here will read as done while the server stays unauthenticated.

    Bounded: one node checked, not the database-role node. #17434 AC1 is that check.

  9. mrveiss commented on Sep 25, 2026

    @mrveiss
    OwnerAuthor

    Correction to my earlier comment — the empty requirepass has TWO independent causes and #17434 fixes only one. Credit to e5, who found this while answering #17434's AC1; I verified it before posting.

    I wrote that rendering requirepass into /etc/redis-stack.conf would not take effect because nothing reads that file. True, but incomplete: requirepass is not in the render at all.

    roles/redis/templates/redis-stack.conf.j2:72-74:

    {% if redis_password is defined and redis_password %}
    requirepass {{ redis_password }}
    {% endif %}
    

    roles/redis/defaults/main.yml:10 sets redis_password: "" — falsy, so the block is skipped. The one inventory that overrides it, group_vars/database.yml:37, sets password: "{{ vault_redis_password }}", and vault_redis_password is referenced in four files and defined in none:

    grep -rln vault_redis_password  ->  roles/redis/vars/main.yml
                                        inventory/group_vars/database.yml
                                        roles/ai-stack/defaults/main.yml
                                        roles/redis/tasks/code_only.yml
    grep -rn  '^vault_redis_password'  ->  (no definition)
    

    So the two causes are independent and both must clear:

    cause owner state
    the rendered config is never read #17434 direction settled — canonical is /etc/redis-stack.conf; e5 has a branch
    there is no password value to render this issue open

    #17434 does not set a password. It removes the reason a password could not take effect — which is what this issue needs, and is why it is a blocker rather than a duplicate.

    The practical consequence for closing this issue: a green render is not evidence. The acceptance criterion has to be CONFIG GET requirepass returning a non-empty value from a provisioned node, because a template whose conditional is false produces a perfectly valid file with no authentication in it and nothing anywhere reports a problem.

  10. added 2 commits that reference this issue on Sep 26, 2026
  11. added
    blocked: observation-gatedCode remains, but cannot be written until one observation answers a question
    on Oct 3, 2026
  12. mrveiss commented on Oct 3, 2026

    @mrveiss
    OwnerAuthor

    Host evidence for the final AC — the server is still nopass as the design assumes, but consumers are not authenticating; they are landing on default anonymously

    Read-only from a live database-role node. This settles half of the last criterion and falsifies the other half, so I am not ticking it.

    The server half — confirmed, exactly as the design assumes

    $ redis-cli PING          # no AUTH sent
    PONG
    $ redis-cli ACL LIST
    user default on nopass sanitize-payload ~* &* +@all
    

    Still nopass. The rendered config carries no requirepass directive at all — the template gates it on {% if redis_password is defined and redis_password %} and that branch was false, leaving a # Authentication header with nothing under it. So redis_password is empty on this node and the SLM-generated password of AC1 has not reached it.

    The consumer half — not met, and the gap is quantified

    Every connected client is the default user:

    $ redis-cli CLIENT LIST | grep -oE 'user=[^ ]+' | sort | uniq -c
         24 user=default
    

    26 of them identify as redis-py. And there are no connection failures:

    Metric Value
    connected_clients 24
    total_connections_received 42,528 (over uptime_in_days:1, 145,212 s)
    rejected_connections 0
    total_error_replies 141

    But "every consumer is default" is not "every consumer authenticates as default", and this AC asks for the second. Against a nopass server, a client that sends AUTH default <password> is refused. Probed directly:

    $ redis-cli AUTH ""
    ERR AUTH <password> called without any password configured for the default user.
    Are you sure your configuration is correct?
    

    That is an ERR reply, so it lands in errorstat_ERR. If consumers were sending AUTH on connect, the ERR count would track the connection count. It does not:

    errorstat_ERR:count=125        vs    total_connections_received:42528
    

    125 against 42,528. At most ~0.3% of connections could have carried an AUTH, and that is an upper bound — ERR also counts unrelated failures, so the true figure is lower still. The conclusion the numbers support is that consumers connect anonymously and are assigned default because that is what an unauthenticated connection gets, not because they authenticate as it.

    Why this matters to this issue specifically

    AC1 is ticked: the SLM generates the password and roles never blank it. The remaining criteria are about consumers being ready for that password. The evidence says they are not — the user=default reading looks like readiness and is not. The moment requirepass is rendered, these 24 clients break, because nothing in the current traffic is sending credentials.

    That is precisely the failure #16628 exists to prevent, and it is the reason its ordering matters: consumers must authenticate before the server stops being nopass, and the user=default metric cannot tell you whether they do.

    Criteria status

    • AC2 — every consumer in the Database role: Redis config + exporter unit still have no update path (credential has no source) #13568 inventory authenticates as default with the password — not met. The live traffic shows no AUTH at a rate consistent with any consumer sending one. Which specific inventory members do and do not send AUTH I could not determine from server-side stats: errorstat_ERR aggregates by error code, not by message or client, so I cannot attribute the 125. That needs a client-side audit of the inventory, not another host read.
    • AC4 — host evidence: every consumer authenticates as default against the still-nopass server, with no connection errors — not met. The "still-nopass" premise and the "no connection errors" clause are both confirmed (rejected_connections:0); the "authenticates" clause is contradicted by the ERR-to-connection ratio.

    Leaving both unticked and the issue open.

    Context

    Read-only: PING, ACL LIST, CLIENT LIST, INFO, and one AUTH "" probe that sent no credential. No value was read or printed; ACL LIST output was redacted of any hash before quoting. Nothing was written and no service was touched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions