config-hygiene(backend): 194 drifted SSOT keys with '0 .env keys', and 6 settings exported as empty strings that silently defeat their defaults (same pattern as #12778)

Symptom

A live node reports 194 drifted SSOT keys and silently falls back to defaults for at least six tuning knobs, every boot:

initialization.lifespan - WARNING - env drift detected in /opt/autobot/autobot-backend —
    env drift: 194 drifted, 0 unknown, (194 SSOT keys, 0 .env keys)

The 0 .env keys is the notable part: the backend .env contributes nothing, so all 194 SSOT keys are unmatched.

Six settings are being exported as empty strings, which is worse than being absent — an empty value looks "set" to anything doing a presence check, and only fails at parse time:

chat_history.cache - WARNING - AUTOBOT_CHAT_SESSION_CACHE_TTL='' is not an integer; falling back to 86400s (24h)
chat_history.cache - WARNING - AUTOBOT_CHAT_RECENT_MAX_ENTRIES='' is not an integer; falling back to 1000
services.llm_key_rotation_scheduler - WARNING - Invalid AUTOBOT_LLM_KEY_ROTATION_INTERVAL_MINUTES=''; defaulting to 60m
api.codebase_analytics.chromadb_storage - WARNING - Invalid CODEBASE_INDEX_PARALLEL_BATCHES, using default 1
api.codebase_analytics.file_analyzer   - WARNING - Invalid CODEBASE_INDEX_PARALLEL_FILES, using default 50
api.codebase_analytics.scanner         - WARNING - Invalid CODEBASE_SCAN_PARALLEL_FILES, using default 50

Also emitted on the same boot:

utils.error_catalog - WARNING - Error catalog YAML not found; using built-in fallback (42 errors)
event_manager       - WARNING - Config file not found at /opt/autobot/autobot-backend/infrastructure/shared/config/config.yaml. Using default debug_mode=False.

Why this matters beyond noise

The empty-string pattern is the same class of defect that produced #12778, where REDIS_HOST was set by the unit but the loader read an empty fallback constant and refused to connect. dict.get("key", default) returns "" — not the default — when the key exists and is blank, so an empty export silently defeats every default-argument fallback in the codebase.

Concretely here: the three CODEBASE_*_PARALLEL_* knobs all collapse to their defaults, which is directly relevant to #12779 (duplicate-detection scans saturating CPU) — the parallelism intended to bound that work is not actually configurable on this node.

AUTOBOT_LLM_KEY_ROTATION_INTERVAL_MINUTES='' silently means key rotation runs on a 60 m default rather than whatever was intended, which is a security-relevant default to be landing on by accident.

Proposed fix

Find what writes these as empty strings — most likely a template rendering an undefined variable to "" rather than omitting the line. An unset variable should be absent, never exported blank.
Treat blank as absent at the read sites: prefer os.environ.get(k) or default over os.environ.get(k, default), or centralise on a helper that coerces "" → None before parsing. This is the same fix shape bug(slm-agent): crash-loop state-change events silently dropped — _load_redis_config ignores REDIS_HOST env and NetworkConstants.REDIS_VM_IP resolves empty outside the backend interpreter #12778 needs, so the two should be considered together.
Investigate the 0 .env keys result — a backend .env matching none of the 194 SSOT keys suggests the drift detector and the deployed .env disagree on key namespace, in which case the "194 drifted" figure may itself be misleading and the detector needs to be trusted before it can be acted on.
Ship the missing error_catalog YAML and infrastructure/shared/config/config.yaml, or stop looking for them if the built-in fallbacks are now canonical.

bug(slm-agent): crash-loop state-change events silently dropped — _load_redis_config ignores REDIS_HOST env and NetworkConstants.REDIS_VM_IP resolves empty outside the backend interpreter #12778 — blank Redis host defeating a default fallback (same root pattern, different key)
bug(codebase-analytics): duplicate-detection timeouts abandon uncancellable executor threads — full-tree globs pile up, saturate CPU, balloon RSS 2.8→7GB and make GUI report 'Backend API Unreachable' #12779 — duplicate-detection CPU saturation; the CODEBASE_*_PARALLEL_* knobs above are the ones meant to bound it

Verification

python -m autobot_shared.env_drift_detector
grep -oE "AUTOBOT_[A-Z_]+='' is not an integer|Invalid [A-Z_]+, using default" /var/log/autobot/backend-error.log | sort -u

Investigated for batching into PR #12821 and deliberately not fixing it there — the obvious fix would mask the defect rather than resolve it.

Finding on proposed item 2 ("treat blank as absent at the read sites"). The read sites already degrade safely. e.g. chat_history/cache.py:36-56:

raw = config.misc.chat_session_cache_ttl
if raw is None:
    return TTL_24_HOURS
try:
    value = int(raw)
except ValueError:
    logger.warning("AUTOBOT_CHAT_SESSION_CACHE_TTL=%r is not an integer; falling back ...")
    return TTL_24_HOURS

An empty string fails int(""), is caught, and the default is applied. Same shape in chromadb_storage.py:46-51 and the other four. So the values are not silently wrong — they fall back correctly and say so.

That warning is currently the only signal that the deployed .env is broken. Coercing "" → None at the read sites would remove the warning without changing the outcome, i.e. it would make a real deployment defect invisible. That is the opposite of what is wanted.

Item 1 is not fixable in this repo. grep across all *.j2/*.yml finds nothing that templates AUTOBOT_CHAT_SESSION_CACHE_TTL, CODEBASE_INDEX_PARALLEL_BATCHES or the other four — whatever writes them as empty strings is outside the codebase (the deployed .env, which is not repo state and must not be hand-edited).

So the actionable order is the reverse of the proposal: find and fix the producer (item 1) first; item 2 should only follow if the producer legitimately emits blanks that must be tolerated. Item 3 (the 0 .env keys / 194-drift figure) still needs live investigation before the number can be trusted at all.

Leaving this open and unclaimed rather than shipping a change that reduces observability.

Related: #12778 is cited as the same pattern, but it differs in the way that matters — there the empty value was consumed without a warning and produced a blank Redis host, so the fallback genuinely was silent. Here it is not.

Partial progress: PR #12859 lands this issue's fix #2 (read-side defence), tracked discretely as #12858 — autobot_shared/env_utils.py now treats a blank or whitespace-only env var as absent across env_raw/env_str/env_int/env_float/env_int_clamped/env_flag, so a blank export can no longer defeat a default.

Worth flagging one case that was not in this issue's list: env_flag used truthy(raw), and truthy("") is False — so a blank export silently flipped a default-True flag OFF. That is now covered by a test.

Still open here (both need the live node):

Fix Docs improvements #1 — finding what writes the blanks. An unset variable should be absent, never exported blank; the codebase is now immune either way, but the template is still producing them.
Fix Fix frontend errors and remove unused code #3 — the 0 .env keys discrepancy. A backend .env matching none of the 194 SSOT keys suggests the drift detector and the deployed .env disagree on key namespace, so the "194 drifted" figure needs to be trusted before it can be acted on.
Fix Dev new gui #4 — the missing error_catalog YAML and infrastructure/shared/config/config.yaml, or a decision that the built-in fallbacks are now canonical.

fix(config): treat a blank env var as absent so defaults are not sile…

fix(config): treat a blank ssot_config knob as absent so six settings…

PR #12899 (merged to Dev_new_gui) references this issue with a close keyword.

fix(config): treat a blank ssot_config knob as absent so six settings stop warning every boot (#12782)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

Still reproducing after tonight's update — and the "0 .env keys" line means something different from what this issue assumes.

Live on the current boot (backend restarted 11:49, deployed tree at tonight's tip):

2026-07-29 11:49:58 initialization.lifespan - WARNING - env drift detected —
  (env drift: 194 drifted, 0 unknown, (194 SSOT keys, 0 .env keys))

This issue reads that as "the backend .env contributes nothing", i.e. the file is effectively empty. It is not:

-rw------- autobot:autobot  /opt/autobot/autobot-backend/.env   size=7075
uppercase keys in it: 93

The file holds 93 keys and the detector counts 0. So the defect is not that .env was generated empty — it is that the drift detector is not reading the file it reports on. Every one of the 194 "MISSING" findings is therefore suspect: with a 0-key side, every SSOT key is trivially unmatched, which is exactly the shape of the output.

That changes where to look: autobot_shared.env_drift_detector's .env resolution/parsing, not the Ansible task that writes the file. Candidates worth checking first — the path it resolves relative to (the warning names /opt/autobot/autobot-backend, where the file does exist), whether it can read a 0600 file as the user it runs as, and whether a parse failure is being swallowed into an empty dict rather than raised.

Second observation: the six empty-string settings this issue lists no longer warn. Zero occurrences of is not an integer; falling back, Invalid AUTOBOT_LLM_KEY_ROTATION, or Invalid CODEBASE_ since the 11:49 restart. Either they are now set correctly or those modules did not initialise this boot — worth confirming which before treating that half as fixed, since "no warning" and "correct value" are not the same evidence.

I have not changed anything. Flagging because a fix aimed at .env generation would not touch the actual failure, and the 194-key figure should not be trusted as a count of real drift until the detector is reading the file.

fix(config): env drift detector read the base DIRECTORY, not the .env…

PR #12964 merged as 38e0c9bc5. Leaving this issue open — the fix removes the false measurement; the real drift figure is still unknown.

_resolve_env_path's fallback returned the base directory because the / ".env" join sat inside a trailing comment. _parse_env_file raised IsADirectoryError, its except OSError swallowed it into an empty dict, and every SSOT key then compared against nothing.

So 194 drifted, (194 SSOT keys, 0 .env keys) was an artefact of the empty side — on a host whose .env holds 93 keys. The confirming line was already in the log and had gone unread:

env_drift_detector: cannot read /opt/autobot/autobot-backend: [Errno 21] Is a directory

Verification: 5 new tests, 1125 passed across autobot_shared, flake8 clean. The tests pin the invariant — the resolved path is always a .env file, never a directory — so this class is caught regardless of which branch produces it, and a directory still parsing to {} is pinned deliberately so the swallow stays visible instead of becoming load-bearing.

Why this stays open:

The real drift count is unmeasured. Once a host runs the corrected detector, the number will be whatever it actually is — possibly near zero, possibly not. Nothing should be concluded from 194.
The six empty-string settings are a separate symptom and their warnings do not fire on the current boot. That is not the same as being fixed — the modules may simply not have initialised — and it wants confirming before that half is closed.

autobot_shared reaches hosts via the code-sync path rather than an Ansible role, so unlike the role-gated fixes on #12959 this should arrive on the next update normally.

fix(hot-reload): watch a module path that actually exists (#12975) (#…

fix(config): point the startup drift check at this backend's own .env (…

PR #13166 (merged to Dev_new_gui) references this issue with a close keyword.

fix(config): point the startup drift check at this backend's own .env (#12782)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

Delivered by PR #13166, squash-merged to Dev_new_gui as d3edbb8e8.

Verification: all required checks green on the PR head before merge; remote branch tip verified against the PR head as a separate step prior to merging.

Closing manually with evidence — Closes #N only fires on the default branch, and this PR targeted Dev_new_gui.

fix(gateway): guard GatewayConfig.from_env() against blank ssot strin…

fix(gateway): inbound ingest governance — bot-self filter, dedup, rec…

fix(slm): read the manifest cache TTL through env_int, and test the b…

fix(slm): the manifest cache TTL was hardcoded and its bypass was unr…

Navigation Menu

Repository navigation

config-hygiene(backend): 194 drifted SSOT keys with '0 .env keys', and 6 settings exported as empty strings that silently defeat their defaults (same pattern as #12778) #12782

Symptom

Why this matters beyond noise

Proposed fix

Related

Verification

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

config-hygiene(backend): 194 drifted SSOT keys with '0 .env keys', and 6 settings exported as empty strings that silently defeat their defaults (same pattern as #12778) #12782

Description

Symptom

Why this matters beyond noise

Proposed fix

Related

Verification

Activity

mrveiss commented on Jul 27, 2026

mrveiss commented on Jul 28, 2026

github-actions commented on Jul 28, 2026

mrveiss commented on Jul 29, 2026

mrveiss commented on Jul 29, 2026

github-actions commented on Aug 1, 2026

mrveiss commented on Aug 1, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions