Repository navigation
bug(slm): the backend cannot resolve a Redis host for 'main' and reports itself healthy anyway #14299
Description
Observed on the live test deployment while diagnosing #14293.
Problem
The SLM backend cannot resolve a Redis host, continuously:
autobot_shared.redis_management.connection_manager - WARNING - Async pool creation for 'main'
failed: Redis host for database 'main' is empty — configuration error (set REDIS_HOST / vm.redis);
refusing to retry
autobot_shared.redis_management.connection_manager - ERROR - Circuit breaker opened for database
'main' after 5 failures
This repeats every ~2 seconds, opening the circuit breaker roughly once a minute, indefinitely.
The message at autobot_shared/redis_management/connection_manager.py:807 is correct and
actionable — the configuration genuinely resolves to empty — so the defect is not the detection.
Why it matters beyond Redis
Two things follow from it:
The service does not report itself unhealthy. The node hosting this is online and
/api/code-sync/status answers normally, so a component that cannot reach its primary datastore is
invisible to every roll-up. That is umbrella #13852's shape.
It drowns the logs. During a code-sync failure this pair of lines is emitted often enough that
the ansible output around the failure is buried — diagnosing #14293 meant filtering these out
first. A "refusing to retry" path that nonetheless re-attempts every two seconds and re-logs at
WARNING each time is a retry loop wearing a no-retry label.
What is needed
- Establish why
REDIS_HOST/vm.redisresolves to empty on a deployed SLM backend — the SSOT
path is what should be supplying it. - A config error that "refuses to retry" should log once and then stay quiet until the config
changes, rather than re-deriving and re-logging on every call. - The condition should reach a health surface, so a backend with an open circuit breaker on its
main database is not reported as healthy.
Activity
fix(redis): resolve the 'main' host via SSOT config and stop the conf…
github-actions commentedon Aug 17, 2026
PR #14401 (merged to
Dev_new_gui) references this issue with a close keyword.If this issue is fully resolved, close it manually. If work remains, no action is needed.
mrveiss commentedon Aug 17, 2026
Closed — landed in
Dev_new_gui(PR #14401). 10/10 required checks green, all 12 python-suite shards SUCCESS, 68 checks, 0 failures — includingverify-generated-types-slm.Root cause.
RedisConfig.host's dataclass field default was frozen at import time fromNetworkConstants.REDIS_VM_IP, which silently resolves to""outsideautobot-backend— itsconfig.registry.ConfigRegistrydependency is not importable from the SLM backend. Compounding it, the SLM backend's own systemd unit never setREDIS_HOST/AUTOBOT_REDIS_HOST, unlike every sibling role.So
'main'had no host to resolve to, and the failure was reported in a way that could not be acted on.What changed: an
autobot_shared.ssot_config-based fallback tier, with'main'resolving through that chain rather than the frozen default; a config error now opens the circuit breaker permanently and logs once instead of every ~2 seconds; Redis health is wired into/healthwith a message that distinguishes a config error from a generic failure; and the Ansible deployment gap is closed so the unit actually sets the variable.That last distinction was part of the defect, not cosmetic: "cannot resolve" and "resolved to nothing configured" are different operator actions, and the old message conflated them.
This PR was blocked on infrastructure, and fixing that unblocked it automatically.
verify-generated-types-slmwent red because wiring Redis health into/healthchanged the SLM OpenAPI schema — andauto-fix-generated-types.ymlhad no SLM coverage, so a backend schema change self-healed while an SLM one required regenerating against a locally-running SLM app, which this repo does not do locally.That gap was already filed as #14368. Rather than hand-patching around it, it was fixed: within minutes of #14403 merging,
github-actions[bot]pushedchore(types): regenerate generated API types from OpenAPI schema(s)to this branch, touching exactlyautobot-slm-frontend/openapi.jsonandsrc/types/generated/api.ts. This PR went from blocked to green with no manual step, and every future SLM schema change now behaves the same way.