security: sweep the 56 call sites that interpolate a credential into an Authorization header (#15204 follow-up) #16809

Activity

1. mrveiss

   added

   bugSomething isn't workingSomething isn't working

   security

   on Sep 16, 2026
2. github-actions

   added

   backend

   on Sep 16, 2026
3. mrveiss commented on Sep 16, 2026

   

   mrveiss

   on Sep 16, 2026

   OwnerAuthor

   More actions

   Classification complete — 4 of 5 HTTP libraries echo, and the remediation set is 2 files, not 56 sites

   Answered empirically rather than by reading error paths: each library was driven with an Authorization header carrying a distinctive canary and an illegal value, and the exception was searched for the canary in its message, its __cause__/__context__ chain, and the formatted traceback.

   Negative control first, per this issue's own instruction. http.client — the confirmed #15204 leak — was probed by the same harness before anything else, and it lit up. A run where it had come back clean would have meant the harness was broken and every other answer worthless.

   Verdicts

   Library	Verdict	The exception it actually produces
   http.client (stdlib) [control]	echoes	ValueError: Invalid header value b'Bearer <canary>\nX-Injected: 1'
   urllib3 2.7.0	echoes	same ValueError — it hands the header to http.client
   requests 2.34.2	echoes	InvalidHeader: Invalid leading whitespace, reserved character(s), or return character(s) in header value: 'Bearer <canary>…'
   httpx 0.28.1 (via h11 0.16.0)	echoes	LocalProtocolError: Illegal header value b'Bearer <canary>…'
   aiohttp 3.14.3	does not echo	ValueError: Forbidden control character detected in headers. Potential header injection attack.
   PyJWT 2.13.0 decode(token)	does not echo	DecodeError: Not enough segments
   PyJWT 2.13.0 encode(key)	does not echo	InvalidKeyError: Could not parse the provided public key.

   Nothing landed in could not determine — but two nearly did, and both were false cleans caught before they were written down:

   • aiohttp first came back "does not echo" because the probe died on connection-refused before header validation ran. That is did not look, not nothing found. Re-probed at the writer, which is where it would actually be sent.
   • httpx first came back "no exception at all". It accepts the illegal value at Request construction and stores the raw CRLF in headers.raw; the rejection happens later, at send, inside h11. Probing only construction would have scored it safe.

   The httpx detail is worth carrying forward: because validation is deferred to send, a call site that wraps only request construction in a try block will not see this error at all, and the leak surfaces from a different frame than the one a reader would suspect.

   aiohttp's message is the model for what the others should do — it names the fault class and refuses to quote the value.

   What this means for the count

   The original "56" counted line hits. Those live in 33 non-test files, and the library is what determines exposure:

   Client	Files	Exposure
   aiohttp	23	none — does not echo
   httpx	2	echoes — the remediation set
   urllib/http.client	1	already fixed by #15204
   no client imported	7	undetermined

   So the real remediation set is two files:

   • autobot-slm-backend/api/llm_config.py
   • autobot-slm-backend/services/hf_token_validator.py

   Undetermined, recorded rather than rounded down

   Seven files build an Authorization header and import no HTTP client themselves, so the library is whichever the eventual caller uses, and settling each one means tracing where the dict goes:

   • autobot-backend/integrations/whatsapp_integration.py
   • autobot-backend/integrations/slack_integration.py
   • autobot-backend/voice_processing/providers/cloud/deepgram_provider.py
   • autobot-backend/services/mcp_server_credentials.py
   • autobot-backend/services/command_extraction_service.py
   • autobot-backend/services/execution/claude_code_backend.py
   • autobot-backend/skills/builtin/community_growth.py

   A file in this list is not a file that is safe. It is a file nobody has checked. Given 23 of the 25 determined files use aiohttp, the base rate suggests most of these are aiohttp too — but that is an expectation, not a measurement, and it should not be recorded as one.

   Caveats

   • Verdicts are for the installed versions named above. A library that does not echo today can start; aiohttp's and PyJWT's good behaviour is not a contract, and the only durable form of this answer is a test per library that fails if the message starts carrying the value.
   • The probe tests one fault class: a control character in the header value. A library could be quiet about that and loud about something else.

   Revised acceptance criteria

   • Determine, per library, whether an invalid credential value reaches the exception message — done, table above
   • Route the two httpx call sites through a check equivalent to pipeline-scripts/header_safe_secret.py's require_header_safe. That module currently sits in pipeline-scripts/, which autobot-slm-backend/ cannot import — it needs to move somewhere shared first, and that move is the larger half of this work
   • Pin each verdict in a test, including the negative ones, so a version bump that changes the behaviour is caught rather than assumed
   • Resolve the seven undetermined files by tracing where their header dict is sent
4. mrveiss commented on Sep 16, 2026

   

   mrveiss

   on Sep 16, 2026

   OwnerAuthor

   More actions

   The per-library test is the deliverable, not a caveat — revising the order

   Reordering the acceptance criteria above, because I filed the version-pinning point as a caveat and it belongs first.

   aiohttp and PyJWT are safe in the versions installed today. Nobody upstream owes us that. Without a test, a routine dependency bump silently converts two non-echoing libraries into two echoing ones, with no signal at all — and the classification above becomes a document asserting something that stopped being true, which is worse than never having measured it, because it will be believed.

   So the ordering is:

   1. A test per library that fails the day its message starts carrying the value — including the ones that currently pass. A negative verdict with no test is an expectation with a date on it.
   2. Route the two httpx call sites (autobot-slm-backend/api/llm_config.py, autobot-slm-backend/services/hf_token_validator.py) through a header-safety check.
   3. Resolve the seven undetermined files by tracing where their header dict is sent.

   That test also costs less than the remediation and covers more: seven libraries, one file, no call-site surgery.

   Two scope notes for whoever picks this up

   Where the try has to go. httpx accepts an illegal header value at Request construction and defers rejection to h11 at send. A call site that wraps only construction will never see the error, and the leak surfaces from a frame the reader is not looking at. Guarding construction alone would look like a fix and change nothing.

   header_safe_secret.py cannot be imported by the code that needs it. It sits in pipeline-scripts/, and both remediation targets are in autobot-slm-backend/. Moving it somewhere both trees can import is the larger half of item 2 — the call-site edit is two lines, the move is the actual work. Worth knowing before anyone estimates this as small.