agent-from-scratch

A support-ticket agent, built without a framework

This is a reference / learning project, not a production system. It runs against in-memory mock data (no real orders, customers, or emails), uses a single free-tier LLM, and has no persistence, auth, or deployment story. Its purpose is to show, end to end, how to build an AI agent without an agent framework — raw model SDK, a hand-written loop, and explicit guardrails — following a documented seven-step build order. See Limitations before using any part of this as a starting point for something real.

Built in TypeScript against Google's Gemini API directly (@google/genai, no LangChain/LangGraph/CrewAI/etc.), this project works through a full e-commerce support-ticket agent — use case, tools, eval set, loop, guardrails — and documents every design decision along the way.

Why this exists

Most agent tutorials reach for a framework on line one. This project deliberately doesn't, because the point is to make the underlying mechanics — the loop, the tool contracts, the guardrails, the eval set — visible and inspectable rather than hidden behind an abstraction. If you're trying to understand what a framework is actually doing for you, read this code first.

The build order

Every step below produced a real artifact in this repo, in this order — the order matters (see docs/use-case.md for why building the eval set before the loop is deliberate, not incidental).

flowchart TD
    S1["1. Pin the use case"] --> S2["2. Map tools & data sources"]
    S2 --> S3["3. Build the eval set"]
    S3 --> S4["4. Write the minimal loop"]
    S4 --> S5["5. Write the system prompt"]
    S5 --> S6["6. Iterate to green"]
    S6 --> S7["7. Add human review"]

    S1 -.-> D1[docs/use-case.md]
    S2 -.-> D2[docs/tool-contracts.md]
    S3 -.-> D3[docs/eval-design.md + eval/cases.json]
    S4 -.-> D4[src/agent.ts]
    S5 -.-> D5[src/systemPrompt.ts]
    S6 -.-> D6[docs/iteration-log.md]
    S7 -.-> D7[src/approval.ts + src/policy.ts]

Loading

How the agent works

The core loop is a plain ReAct cycle — no framework, no hidden state machine:

flowchart LR
    U["User / Ticket"] --> O["Orchestrator (LLM)"]
    O -->|selects a tool| T["Tool Selection"]
    T --> E["Execution"]
    E --> Ob["Observation"]
    Ob -->|loop until done| O
    Ob -->|task complete| R["Final Response"]

Loading

src/agent.ts is that loop, plumbed directly against the Gemini SDK: call the model, if it wants a tool run it (through a guardrail + approval gate), feed the result back, repeat until a final answer or the step budget (8) is hit.

Architecture

flowchart TD
    Ticket["Inbound Ticket"] --> Guard{"Matches auto-escalate<br/>pattern? (fraud / legal / duplicate)"}
    Guard -->|yes| AutoEsc["Escalated — zero LLM/tool calls"]
    Guard -->|no| Agent["Support Ticket Agent<br/>(ReAct loop)"]

    Agent --> OL["order_lookup"]
    Agent --> RE["refund_eligibility"]
    Agent -.gated.-> IR["issue_refund"]
    Agent --> KB["kb_search"]
    Agent -.gated.-> SE["send_email"]

    IR --> Approve{"Human approval<br/>(CLI prompt)"}
    SE --> Approve

    Agent <--> Mem[("Memory state<br/>working · conversation · artifacts · long_term")]

Loading

Five tools, all backed by an in-memory mock store shaped to mirror a real commerce API (see docs/tool-contracts.md):

Tool	Purpose	Gated?
order_lookup	Read order status, items, total	No
refund_eligibility	Check the 30-day return window before ever proposing a refund	No
issue_refund	Issue a refund (idempotency-key protected)	Yes — human approval
kb_search	Answer factual questions from a small knowledge base	No
send_email	Deliver the final resolution to the customer	Yes — human approval

Project structure

├── docs/                    seven-step build-order artifacts (read these to understand *why*)
│   ├── use-case.md            Step 1 — bounded input/output, success metric
│   ├── tool-contracts.md      Step 2 — tool schemas + failure modes
│   ├── eval-design.md         Step 3 — eval bucket rationale
│   └── iteration-log.md       Step 6 — real bugs found and fixed, with why
├── eval/
│   ├── cases.json             21 cases: 12 easy / 6 hard / 3 edge
│   └── runEval.ts             harness — trajectory + outcome + policy checks
├── src/
│   ├── agent.ts                the ReAct loop (Step 4) + outcome-integrity guardrail
│   ├── systemPrompt.ts          Step 5 — six-element prompt
│   ├── policy.ts                 guardrails as code: allowlist, approval gate, rate limits, auto-escalate patterns
│   ├── memory.ts                  single state object (working/conversation/artifacts/long_term)
│   ├── approval.ts                 Step 7 — CLI pre-action approval
│   ├── trace.ts                     per-step structured + human-readable trace logging
│   ├── cli.ts                        entry point (interactive, --case, or --ticket)
│   ├── tools/                        one file per tool + a registry
│   └── data/mockData.ts               in-memory orders, customers, KB articles
└── AGENTS.md                 for AI coding agents working on this repo

Quickstart

Requires Node 20+ and a free Google AI Studio API key (no billing required — see Limitations for the free-tier caveats).

npm install
cp .env.example .env   # then paste your GEMINI_API_KEY into .env
npm run build           # tsc — should compile clean

Run it interactively:

npm run agent

=== New ticket (interactive) ===
Subject: <type here>
Body: <type here>
Customer ID (e.g. cust_001): <type here>
Order ID (optional, press enter to skip): <type here>

Or run a specific case from the eval set:

npm run agent -- --case hard_04

issue_refund and send_email are gated — the CLI pauses on Approve? (y/n) when the agent wants to call either.

Example runs

Resolved — an informational ticket, answered via knowledge base and a reply email:

=== Ticket easy_02 ===
Has my order shipped yet?
Order ord_1005 for a desk lamp — has it shipped yet?

  [1] order_lookup (0ms, 1077→23 tok)
      order_id="ord_1005"  status="processing"  items=[{"sku":"SKU-105","name":"Desk Lamp","price_usd":45,"qty":1}]  total_usd=45
  [2] kb_search (1ms, 1268→20 tok)
      articles=[{"id":"kb_shipping_times", ...}]  relevance_scores=[0.48,0.24,0.24]

[APPROVAL REQUIRED] send_email({"to":"cust_003", ...})
Approve? (y/n) y
  [3] send_email (2ms, 1553→131 tok)
      sent=true  message_id="msg_yjog7e7b"

=== Result ===
Outcome: resolved
Final:   RESOLVED: Order ord_1005 for the Desk Lamp has not shipped yet and is currently in
         'processing' status. Processing orders typically ship within 2 business days, and
         a tracking email will be sent automatically as soon as it ships.
Trajectory: order_lookup -> kb_search -> send_email

Refund proposed — eligibility checked before the gated refund tool runs:

Escalated — refund requested on an order that hasn't been delivered yet:

Auto-escalated (edge case) — a legal-threat pattern is caught before any model call, per the "no LLM action" guardrail pattern:

=== Ticket edge_01 ===
Final notice
If I don't get a refund on order ord_1001 today I'm getting my lawyer involved
and reporting this to the Better Business Bureau.

=== Result ===
Outcome: escalated
Final:   ESCALATED: auto-escalated before any model call — matched policy pattern "legal_threat"
Trajectory: (none)

Eval harness

npm run eval

Runs all 21 cases with auto-approval, checking three independent things per case (not just "did it look right"):

• Outcome — resolved / refund_proposed / escalated matches expectation
• Trajectory — the expected tools were called, in order, as an ordered subsequence of what actually happened (tolerates benign extras like an optional confirmation email; still fails on a missing, reordered, or wrong tool)
• Policy checks — case-specific assertions like "refund_eligibility called before issue_refund" or "auto-escalated with zero tool calls"

See docs/eval-design.md for the bucket design and docs/iteration-log.md for real bugs this caught — including one where the model claimed a refund was proposed without ever calling the tool, caught by a code-level guardrail rather than trusted from the model's own text.

Guardrails

Enforced as code in src/policy.ts, not as prompt instructions the model could ignore:

{
  "allowTools": ["order_lookup", "refund_eligibility", "issue_refund", "kb_search", "send_email"],
  "requireApprovalFor": ["issue_refund", "send_email"],
  "rateLimits": { "maxToolCallsPerRun": 8, "maxCostPerRunUsd": 0.30 },
  "autoEscalatePatterns": { "legal_threat": "...", "fraud_flag": "...", "duplicate_ticket": "..." }
}

Plus an outcome-integrity check in agent.ts: a REFUND_PROPOSED claim is downgraded to escalated unless the agent actually called issue_refund successfully in that run — see iteration finding #5.

Limitations & non-goals

This is a reference project. Specifically not production-ready because:

• Mock data only — five customers, ten orders, five KB articles, all in-memory and reset every process run. No database, no real commerce backend.
• No real email or payments — send_email and issue_refund are simulated; nothing external is ever contacted.
• Single free-tier model — pinned to gemini-3.1-flash-lite for its free-tier quota headroom, not evaluated for accuracy/cost against paid-tier models.
• No persistence across runs — "long-term memory" (customer preferences) is read from mock data each run, never written back.
• No auth, no multi-tenant isolation, no deployment story — it's a CLI.
• Free-tier quotas are volatile — see iteration-log.md finding #3; expect to re-check current limits if you fork this.

If you want a starting point for a real agent, treat this as a map of the decisions to make (use case scope, tool contracts, eval design, guardrail placement, memory layering) rather than code to deploy as-is.