Repository navigation
bug(chat): LangGraph chat stream dies mid-response — root cause unknown, needs backend log investigation #3491
Description
Problem
Chat session 4f825e59-9ab4-479b-8ed3-ddf7daaf68ec (2026-04-05) shows the LangGraph stream consistently dying:
- Message 1 ('Hello Autobot testing.'): immediate error before any LLM output
- Message 2 ('Sound testing.'): thought + planning chunks streamed successfully via
qwen3.5:9b, then stream died before the final response
The stream death triggers _run_graph_task's except Exception handler, which puts the error in the queue and deletes the Redis checkpoint for the session.
What's Known
- Backend VM (
172.16.168.20) was unreachable via SSH during investigation — backend logs not accessible - The
_run_graph_taskalready logs withexc_info=Trueat ERROR level — the exception details are in the journal - After fix in issue bug(chat): empty Python exception message causes 'Unknown error' instead of diagnostic info #3488, future failures will include the exception class name in the displayed error
- Possible causes: Ollama/vLLM timeout during generation, LangGraph checkpoint Redis failure, malformed model response, network interruption between backend and AI stack (
172.16.168.24:8080)
Steps to Diagnose
- Reproduce the stream failure (send a chat message)
- SSH to backend VM:
ssh autobot@172.16.168.20 journalctl -u autobot-backend --since '10 minutes ago' | grep -A20 'Graph execution error'- Identify the exception type and full traceback
- Fix the root cause
Acceptance Criteria
- Root cause identified from backend logs
- Fix implemented and tested
- Chat streaming works consistently with
qwen3.5:9bmodel
Activity
fix(ansible): ai-stack role missing source deploy, wrong venv, wrong …
fix(agents): asyncio.Lock collision causes health check failure in St…
mrveiss commentedon Apr 5, 2026
Fix Complete
All root causes identified and resolved:
1. AI stack crash-loop — missing source files
fa07aab0f: Added Ansible tasks to deployai_api_server.py,ai_container_main.py,requirements-ai.txtfromshared/docker/ai-stack/; createdsrc/agentsandsrc/autobot_sharedsymlinks; fixed service to use backend venv + PYTHONPATH2. SSE error display — wrong key name
400d5553c: Three error handlers inapi/chat.pyused"message"key; frontend readsdata.content→ undefined → fell back to literal'Stream error'. Fixed to"content"3. Empty exception string — meaningless errors
400d5553c:str(asyncio.TimeoutError()) == ''→ frontend fallback. Fixed withstr(exc) or f'{type(exc).__name__}: unexpected error'4. Error status overwritten —
sentoverrideserror400d5553c: Post-loop cleanup unconditionally setstatus:'sent'. Fixed withif (msg.status !== 'error')guard5.
_ConfigStubmissing methods — re-entrant init crash46eb3c275:_ConfigStub(returned during circular ConfigManager init, Bug: Circular import kills test collection — logging_manager imports config_manager at module level #1862) was missingget_redis_config(),get_llm_config(),is_feature_enabled(), and async variants. AI stack'sErrorBoundaryManagercalledget_redis_config()during init, hitting the missing method. Added all 9 missing stub methods with env-var fallbacks.6. asyncio.Lock / threading.Lock collision — health check always unhealthy
46eb3c275:StandardizedAgent.__init__overrodeBaseAgent._stats_lock(threading.Lock) withasyncio.Lock().BaseAgent.health_check()used syncwith self._stats_lock:which fails for asyncio.Lock. Renamed to_async_stats_lockto avoid collision. All 3 agents (chat, rag, classification) now reporthealthy.7. Agent LLM provider env vars — missing config
46eb3c275: AddedAUTOBOT_CLASSIFICATION_PROVIDER,AUTOBOT_CHAT_PROVIDER,AUTOBOT_RAG_PROVIDERtoai-stack.env.j2withollamadefaults.Verified
/healthendpoint