GP Reception Assistant — RAG + LangGraph Support Agent

A production-grade, deterministic RAG + agent architecture built with LangChain and LangGraph, applied to a GP/healthcare-reception knowledge base with an integrated evaluation harness for groundedness, safety filtering, and routing correctness.

Safety Disclaimer: This is an administrative support assistant and not a diagnostic clinical system. It never provides medical advice, does not triage symptoms, and defers clinical/emergency queries to human clinicians, NHS 111, or emergency services (999). Safety boundaries are enforced deterministically in code via keyword overrides and diagnostic-language regex filters that execute independently of LLM calls.

---

Quickstart

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Offline Demo mode — zero external API key needed, fully deterministic
python -m tests.run_eval          # runs the eval suite, writes tests/eval_report.md
uvicorn app.main:app --reload     # serves the API and Web UI at http://localhost:8000

curl -X POST http://localhost:8000/chat \
  -H 'Content-Type: application/json' \
  -d '{"question": "What are your opening hours?"}'

To run in live mode (OpenAI API integration):

cp .env.example .env   # then set OPENAI_API_KEY
python -m tests.run_eval

Docker deployment:

docker compose up app                                   # demo mode
OPENAI_API_KEY=sk-... docker compose up app              # live mode, Chroma
OPENAI_API_KEY=sk-... VECTOR_STORE=pgvector \
  docker compose --profile pgvector up                   # live mode, Postgres+pgvector

---

Architecture

                              User question
                                   |
                                   v
                         route_intent  <- structured output (Pydantic)
                                          hard-coded emergency keyword
                                          override runs BEFORE the LLM
              +--------------------+--------------------+
              v                    v                     v
     knowledge_search           tool_use              escalate / direct_decline
              |                    |                     |
              v                    v                     |
     retrieve_documents        call_tool                 |
              |                    |                     |
              v                    |                     |
     check_relevance              |                     |
      |          |                |                     |
  relevant   not relevant --------+---------------------+
      |                           |                     |
      v                           v                     |
  generate_answer <---------------+                     |
      |                                                  |
      v                                                  |
  safety_check (groundedness + diagnosis-language filter) |
   |         |                                            |
 passed    failed ----------------------------------------+
   |                                                       v
   v                                                 escalate_response
format_response
   |
   v
Return to user

Nodes

Node	Responsibility	Structured output
route_intent	Classify into knowledge_search, tool_use, direct_decline, escalate. Emergency keyword regex runs first and force-routes to escalate, independent of the LLM.	RouteDecision
retrieve_documents	Top-k similarity search against the GP knowledge base.	List[Document]
check_relevance	Scores whether retrieved docs actually answer the question; below threshold falls through to escalate instead of forcing an answer.	RelevanceResult
call_tool	Dispatches to one of four support tools.	tool-call result
generate_answer	Answers only from retrieved context, must cite source titles, refuses clinical extrapolation.	text + sources
safety_check	Groundedness (claims traceable to context) + diagnostic-language regex filter. Either check failing routes to escalate.	SafetyResult
escalate	Standard human-handoff message with reference ticketing. Reachable from routing, relevance, and safety gates.	—
direct_decline	Off-topic / prompt-injection refusal. No ticket generated.	—
format_response	Assembles the final payload (answer + sources + scores, or tool result).	—

State Management

A single typed state (AgentState, in app/graph/state.py) flows through the graph: user question, route decision, retrieved documents, relevance evaluation, draft answer, safety verification, tool results, escalation reason, final response payload, and error telemetry.

Tools (app/tools/support_tools.py)

• check_gp_open_now — Real-time wall-clock verification against surgery opening schedules.
• check_appointment_slots — Availability querying (integration point for PMS/EHR systems).
• escalate_to_human — Reception ticketing and pager escalation.
• find_emergency_guidance — Fixed lookup table over approved emergency guidance protocols.

Error Handling & Resilience

Every node is wrapped with an error handler (app/graph/nodes/errors.py) ensuring unhandled exceptions capture telemetry in state["error"] and cleanly escalate to human reception rather than dropping connections or returning blank responses.

---

Hybrid Retrieval Architecture

The system abstracts the retrieval layer behind a unified interface (.get_relevant(query, k)), supporting two operational backends:

1. Offline Demo Mode (TF-IDF + Cosine Similarity): Real lexical retrieval powered by scikit-learn. Allows offline execution, deterministic testing, and CI verification without requiring third-party API keys or remote vector services.
2. Live Production Mode (Chroma / PostgreSQL pgvector + OpenAI Embeddings): Semantic similarity retrieval for live agent deployments with dense embeddings.

---

Evaluation Suite & Quality Gates

tests/eval_dataset.py contains 19 adversarial and boundary test cases across:

• in_scope (practice hours, booking procedures, repeat prescriptions, test result policies, registration)
• tool_use (live opening checks, slot availability lookups)
• emergency (acute chest pain, severe bleeding, breathing distress, mental health crises) — mandatory immediate escalation
• clinical_advice (medication dosage queries, symptom diagnosis requests) — refusal of clinical extrapolation
• out_of_scope / adversarial (off-topic queries, prompt-injection attempts) — prompt boundaries strictly preserved
• ambiguous (under-specified user requests) — requests clarification / routes safely

Run the evaluation:

python -m tests.run_eval        # writes tests/eval_report.md
pytest tests/test_eval.py -v    # executes CI quality gates

Evaluation Results:

• Aggregate Pass Rate: 94.7% (18/19) in offline mode.
• Safety-Critical Gate: 100% Pass across all emergency triggers, clinical advice refusals, and injection defenses.
• High-precision relevance gating ensures ambiguous or borderline out-of-scope queries defer safely to reception rather than hallucinating responses.

---

LangGraph StateGraph Architecture

• Dynamic Conditional Routing: Cycles and state-dependent branches route queries dynamically based on runtime classification, relevance thresholds, and safety checks.
• Explicit Typed State: AgentState provides auditable, observable intermediate state throughout the execution lifecycle.
• Multi-Point Escalation: Escalation acts as a unified terminal node reachable from routing, relevance gating, and safety verification.

---

Production Considerations & Safety Boundaries

• Dual-Layer Safety: Groundedness checks combined with deterministic diagnostic-language regex filters safeguard against hallucination and clinical overreach.
• Extractive Answer Construction: Demo-mode answers are strictly extractive from verified context chunks.
• Data Privacy: Knowledge base contents are synthetic practice records; no PHI is processed.

---

Project Structure

app/
  config.py               # configuration & environment management
  main.py                 # FastAPI application & REST endpoints
  graph/
    state.py              # AgentState TypedDict definition
    build.py              # LangGraph StateGraph workflow construction
    nodes/                # modular graph nodes & safe_node error wrapper
  rag/
    loader.py             # markdown loader and text splitters
    retriever.py          # TF-IDF (offline) & Chroma/pgvector (live) retrievers
  tools/
    support_tools.py      # LangChain tool implementations
  schemas/
    models.py             # Pydantic schemas (RouteDecision, RelevanceResult, SafetyResult)
  llm/
    client.py             # ChatOpenAI factory and model configuration
  static/
    index.html            # interactive web dashboard & LangGraph trace inspector
    styles.css            # modern healthcare theme styling
    app.js                # frontend state and API interactions
data/
  knowledge_base/         # GP reception reference markdown documents
tests/
  eval_dataset.py         # 19 benchmark test cases
  run_eval.py             # evaluation harness and markdown report generator
  test_eval.py            # pytest quality gates (safety-critical & aggregate)
  test_api.py             # FastAPI TestClient API integration tests
docker/Dockerfile
docker-compose.yml

Tech Stack

Python 3.10+ · LangChain · LangGraph · langchain-openai · Chroma / Postgres+pgvector · scikit-learn · FastAPI · Pydantic v2 · pytest · Docker.