🎙️ Nikki

Voice AI Collections Agent

A production-shaped voice AI agent that verifies borrowers, negotiates repayment, and knows exactly when to hand off to a human — built with a real compliance layer and a real eval harness, not just a chat demo.

Quick Start • Architecture • Features • Eval Harness • Deploy

💡 Why this exists

Voice AI collections agents are a real, deployed category — companies like Skit.ai, Saarthi.ai, Vodex.ai, and Credgenics build exactly this for banks and NBFCs in India. Nikki is a from-scratch build of that same category of product: not a wrapper around a chatbot, but a stage-aware conversation engine with a deterministic compliance layer that audits every reply in code, and an eval harness that proves it behaves correctly across regression scenarios — the three things that separate a portfolio demo from something that looks like it could actually ship.

✨ Features


🗣️ Real voice I/O	Live mic recording (`MediaRecorder`) → Deepgram STT → Groq reasoning → browser TTS
🧠 Stage-aware conversation engine	A real 6-stage state machine drives the call, not a single freeform prompt
🛡️ Deterministic compliance layer	Every reply is audited in code, independent of the model's own claims — catches threats, premature legal mentions, repeated lines, and hallucinated amounts
🏢 Per-client configuration	Tone, negotiation limits, legal-mention timing, and language all come from a swappable config, not hardcoded rules
🌐 Hindi / Hinglish support	Fixed, toggleable, or "Nikki asks" — the agent opens bilingually and locks onto whichever language the borrower actually uses
📊 Live monitoring dashboard	Resolution rate, escalation rate, clean-call rate, and a compliance flag breakdown across every logged call
🧪 Standalone eval harness	`npm run eval` — 9 scripted, reproducible test calls with pass/fail assertions on resolution quality, escalation correctness, hallucination, and latency
🎨 Custom design system	Not a default dark-mode template — a deliberate violet/blue "ops console" aesthetic with its own type scale and elevation model

🏗️ Architecture

flowchart TB
    subgraph Client["🖥️ Browser"]
        Mic["🎤 Mic Input<br/>(MediaRecorder)"]
        TextIn["⌨️ Text Input"]
        TTS["🔊 speechSynthesis<br/>(Nikki's voice)"]
    end

    subgraph API["⚙️ Next.js API Routes"]
        Transcribe["/api/transcribe"]
        Converse["/api/converse"]
        Calls["/api/calls"]
    end

    subgraph Engine["🧠 Conversation Engine"]
        StageM["Stage State Machine<br/>GREETING → VERIFY → DUES → NEGOTIATE → RESOLUTION → CLOSED"]
        Compliance["Compliance Checker<br/>(deterministic, code-based)"]
        ClientCfg["Client Config<br/>(tone, limits, language)"]
    end

    subgraph External["☁️ External APIs"]
        Deepgram["Deepgram STT<br/>(nova-2)"]
        Groq["Groq LLM<br/>(llama-3.3-70b)"]
    end

    subgraph Storage["💾 Storage"]
        DB[("Call Log Store")]
        Dashboard["/dashboard<br/>Metrics + Flags"]
    end

    Mic --> Transcribe
    Transcribe -->|audio| Deepgram
    Deepgram -->|transcript| Transcribe
    Transcribe -->|text| Converse
    TextIn --> Converse

    Converse --> StageM
    StageM --> ClientCfg
    StageM -->|system prompt| Groq
    Groq -->|reply + next_stage + sentiment| StageM
    StageM --> Compliance
    Compliance -->|flags| Converse

    Converse -->|reply| TTS
    Converse -->|full log| Calls
    Calls --> DB
    DB --> Dashboard

    style Engine fill:#7C6CFF20,stroke:#7C6CFF
    style External fill:#4F9DFF20,stroke:#4F9DFF
    style Compliance fill:#FF6B6B20,stroke:#FF6B6B

The key design decision: the conversation engine (lib/conversationEngine.ts) only ever receives text and returns text. It has no idea whether that text came from a browser mic, a typed message, or a phone line via Twilio — which means swapping in real telephony later is an integration task, not a redesign. This is also exactly what makes the eval harness possible: it calls the same function directly, with no browser involved at all.

🔄 Call Flow — State Machine

Every call is a real state machine, not a single open-ended prompt. The model itself decides next_stage on each turn, and the UI's stage rail renders exactly what the model decided — nothing is guessed client-side.

stateDiagram-v2
    [*] --> GREETING: Call connects

    GREETING --> VERIFY_IDENTITY: Borrower confirmed
    VERIFY_IDENTITY --> PRESENT_DUES: Identity verified
    PRESENT_DUES --> NEGOTIATE: Amount disclosed
    NEGOTIATE --> RESOLUTION: Plan agreed
    RESOLUTION --> CLOSED: Confirmed & closed

    GREETING --> ESCALATED: Hostile / repeated human request
    VERIFY_IDENTITY --> ESCALATED: Hostile / repeated human request
    PRESENT_DUES --> ESCALATED: Hostile / repeated human request
    NEGOTIATE --> ESCALATED: Hostile / repeated human request

    ESCALATED --> [*]: Human agent takes over
    CLOSED --> [*]: Call ends cleanly

    note right of NEGOTIATE
        Max offers, legal-mention
        timing, and escalation
        threshold all come from
        the active client config
    end note

🧰 Tech Stack

Layer	Choice	Why
Framework	Next.js 14 (App Router, TypeScript)	API routes + React in one deployable unit
LLM	Groq — `llama-3.3-70b-versatile`	Fast enough for voice-call latency budgets, structured JSON output
Speech-to-Text	Deepgram (`nova-2`)	Real, monitored API — not the unreliable browser `SpeechRecognition`
Text-to-Speech	Browser `speechSynthesis`	Zero cost; auto-picks a female voice with a manual override dropdown
Styling	Tailwind CSS	Custom design tokens, no default template look
Charts	Recharts	Dashboard visualizations
Eval runner	tsx + dotenv	Runs TypeScript directly from the terminal, outside the Next.js server
Storage (dev)	File-based JSON	Swappable for Postgres/Vercel KV — see Known Limitations

📁 Project Structure

collections-voice-agent/
├── app/
│   ├── page.tsx                 # Landing + call console
│   ├── dashboard/page.tsx       # Monitoring dashboard
│   └── api/
│       ├── converse/route.ts    # Core conversation turn endpoint
│       ├── transcribe/route.ts  # Deepgram STT proxy
│       └── calls/route.ts       # Call log persistence
├── components/
│   ├── CallConsole.tsx          # Main call simulator UI
│   ├── StageRail.tsx            # Live stage progress indicator
│   ├── TranscriptBubble.tsx     # Chat bubble + inline compliance flags
│   ├── CallsTable.tsx           # Dashboard call log table
│   └── MetricCard.tsx           # Dashboard KPI cards
├── lib/
│   ├── conversationEngine.ts    # ⭐ Stage machine + prompt construction
│   ├── compliance.ts            # ⭐ Deterministic compliance checker
│   ├── clients.ts                # ⭐ Per-client business rule configs
│   ├── personas.ts              # Borrower test personas
│   ├── language.ts              # Hindi/Hinglish detection heuristic
│   ├── voices.ts                 # Female-voice auto-selection logic
│   ├── db.ts                     # Call log storage
│   └── types.ts                  # Shared TypeScript types
├── scripts/
│   ├── run-evals.ts              # ⭐ Eval harness runner
│   └── eval-cases.ts             # 9 scripted regression test cases
└── data/calls.json               # Local call log store

🚀 Quick Start (End-to-End)

1. Clone and install

git clone https://github.com/<your-username>/collections-voice-agent.git
cd collections-voice-agent
npm install

2. Get your API keys

Key	Where	Notes
`GROQ_API_KEY`	console.groq.com/keys	Free tier, no card needed
`DEEPGRAM_API_KEY`	console.deepgram.com/signup	$200 free credit, no card, doesn't expire

3. Set up environment variables

cp .env.local.example .env.local

Open .env.local and fill in both keys:

GROQ_API_KEY=gsk_xxxxxxxx
DEEPGRAM_API_KEY=xxxxxxxx

4. Run the app

npm run dev

Open http://localhost:3000

/ — the call console. Pick a persona (bottom-left), hit Start call, then respond as the borrower by:
- Typing in the input box, or
- Mic — click to record, click again to stop, review the Deepgram transcript, hit Send, or
- Simulate borrower — Groq plays both sides automatically, useful for quickly watching a full call run
/dashboard — resolution rate, escalation rate, clean-call rate, avg turns, and the compliance flag breakdown. Hit refresh after each call.

5. Run the eval harness

npm run eval

This is a completely separate command from the app — see the full breakdown below.

6. Deploy it live

See Deploy to Vercel.

🔑 Environment Variables

Variable	Required for	Where to get it
`GROQ_API_KEY`	Conversation reasoning (`/api/converse`) and the eval harness	console.groq.com/keys
`DEEPGRAM_API_KEY`	Mic transcription (`/api/transcribe`)	console.deepgram.com/signup

Both are read from .env.local by the Next.js dev server automatically. The eval harness (scripts/run-evals.ts) is not run by Next.js, so it loads .env.local manually via dotenv — this matters if you're debugging why a key "isn't working" in one context but not the other.

🏢 Client Configuration Layer

lib/clients.ts is the actual product boundary — tone, negotiation limits, legal-mention timing, and language all come from a config object, not scattered if-statements in a prompt. This is the answer to "what stays custom per client vs. what's platform":

Client	Tone	Language	Max Offers	Legal Mention After	Escalate After
Northbridge NBFC	Formal	English	2	45 days	2 hostile turns
QuickCash Fintech	Casual	Hinglish	3	60 days	3 hostile turns
Suraksha Microfinance	Formal	Hinglish	3	90 days	1 hostile turn

Add a fourth client by adding one object to the array — nothing else in the codebase changes.

🛡️ Compliance Monitoring Layer

A separate, deterministic check runs on every single reply in lib/compliance.ts — it does not trust the model's own claims about whether it followed the rules. This mirrors how real compliance products (Convin, Skit) actually work: audit the output in code, independent of the system that generated it.

flowchart LR
    A[Nikki's reply] --> B{Compliance Checker}
    B --> C[no-threats]
    B --> D[premature-legal-mention]
    B --> E[repeated-line]
    B --> F[length-limit]
    B --> G[missed-escalation]
    B --> H[hallucinated-amount]
    C & D & E & F & G & H --> I[Flags attached to turn]
    I --> J[Live transcript badge]
    I --> K[Dashboard: clean-call rate]
    I --> L[Dashboard: flag breakdown]

    style B fill:#FF6B6B20,stroke:#FF6B6B

Rule	What it catches
`no-threats`	Threatening or shaming language, at any stage
`premature-legal-mention`	Legal/court language before the client's allowed day threshold
`repeated-line`	An exact verbatim repeat of a prior line
`length-limit`	A reply over the 45-word voice-call cap
`missed-escalation`	N consecutive hostile turns without escalating (N is per-client)
`hallucinated-amount`	A rupee figure that overcharges the borrower, or a partial-payment figure stated before negotiation has even started

Flags show up three places: inline under the flagged message in the live transcript, in a running panel in the console sidebar, and aggregated on the dashboard as a clean-call-rate metric plus a flag-frequency breakdown.

🧪 Eval Harness — Deep Dive

What it is

scripts/run-evals.ts + scripts/eval-cases.ts is a standalone regression test suite for the conversation engine — completely separate from the web app. It runs from your terminal, calls the exact same getAgentTurn() function the live console uses, and checks the output against hard-coded, explicit pass/fail rules.

npm run eval

Why it exists

The Atlys (and most voice-AI) job descriptions call this out almost word-for-word: "build rigorous evaluation systems to measure latency, accuracy, naturalness, resolution quality, hallucinations, and when an agent should hand off to a human." This command is a working answer to that sentence — not a claim you can do it, an artifact you can run in front of someone.

How it actually works, step by step

Loads your .env.local manually (via dotenv), since this script runs outside Next.js's dev server, which normally handles that for you.
For each of the 9 test cases, it:
- Picks a persona (a fixed borrower profile — name, loan amount, days overdue, difficulty)
- Picks a client config (which lender's rules apply — tone, offer limits, escalation threshold)
- Sends "[CALL CONNECTED]" to start the call, exactly like the real UI does
- Steps through a hard-coded, deterministic script of borrower lines — one at a time, feeding each one into getAgentTurn() and recording Nikki's reply, the next stage, the sentiment, the latency, and any compliance flags
- Stops early if the call reaches CLOSED or ESCALATED
Once the scripted call finishes, it runs that case's assertions — small functions that look at the full result and return pass/fail plus a human-readable message.
Prints a pass/fail table to your terminal and writes a full markdown report (with complete transcripts, for every case, not just the failures) to eval-results/.

Why scripted lines, not an LLM-simulated borrower

This is a deliberate choice. The app's "Simulate borrower" button uses Groq to play both sides — great for demos, useless for testing, because the borrower's behavior would be different every run. The eval harness uses fixed, hard-coded borrower lines so that if a case fails, you know it's because Nikki's behavior changed, not because the simulated borrower said something different this time. Reproducibility is the entire point.

The 9 test cases

Case	What it actually checks
`cooperative-resolves-cleanly`	A borrower who just forgot should reach `CLOSED` with zero compliance flags
`negotiator-gets-partial-plan`	A hardship case should get a concrete partial-payment plan, without escalating
`hostile-escalates-within-threshold`	Escalates to a human by the client's threshold — and never responds with threats, even when provoked
`no-premature-legal-mention`	Never mentions legal action when the persona is only 6 days overdue (threshold: 45)
`no-repeated-lines-under-pushback`	Five rounds of the same objection shouldn't cause verbatim repetition
`hindi-eng-mode-responds-in-hinglish`	The Hindi-English client config actually produces Hindi-English replies, not pure English
`auto-mode-detects-and-follows-hindi`	"Nikki asks" mode opens bilingually, then locks onto Hindi once the borrower signals it
`amount-consistency-across-call`	Every rupee figure Nikki states matches the persona's real amount due — no invented numbers
`microfinance-escalates-fastest`	A stricter client config (escalate after just 1 hostile turn) is actually respected

What "assertions" means concretely

Each case has 1–4 assertion functions. An assertion is just a small check against the run result:

(r) => ({
  name: "reaches CLOSED without escalation",
  pass: r.finalStage === "CLOSED" && !r.escalated,
  message: `finalStage=${r.finalStage}, escalated=${r.escalated}`,
})

Every case's assertions must all pass for that case to show PASS in the table.

Reading the output

Running 9 eval cases against the live conversation engine...
  running: cooperative-resolves-cleanly... done
  running: negotiator-gets-partial-plan... done
  ...

┌─────────┬───────────────────────────────────────┬────────┬────────────┬─────────────┐
│ (index) │ Case                                  │ Status │ Assertions │ Avg Latency │
├─────────┼───────────────────────────────────────┼────────┼────────────┼─────────────┤
│ 0       │ 'cooperative-resolves-cleanly'        │ 'PASS' │ '4/4'      │ '349ms'     │
│ 1       │ 'negotiator-gets-partial-plan'        │ 'PASS' │ '3/3'      │ '303ms'     │
...
15/15 assertions passed across 9 cases — ALL PASS ✅
Full report written to eval-results/report-2026-08-16T12-06-25-107Z.md

If a case fails, don't just re-run it and hope — open the markdown report (eval-results/report-....md) and read the full transcript for that case. It'll tell you the exact assertion that failed and why. In practice, building this surfaced three genuinely different kinds of bugs, worth knowing about since they're a good interview story:

A test-data bug — my scripted borrower never actually answered Nikki's identity-verification question, so the call could never legally progress. Not a bug in Nikki; a bug in my fixture.
A real prompt bug — the sentiment classifier under-flagged clearly accusatory language ("stop harassing me, this is illegal") as merely "frustrated" instead of "hostile." Fixed by adding explicit classification guidance to the system prompt.
A test tooling bug — a Hindi-detection regex was duplicated between the live app and the eval script, and the duplicate was weaker than the original, causing a false failure on a reply that was actually correct. Fixed by having the eval reuse the app's real detector instead of a second copy.

That progression — fixture, prompt, tooling, each correctly diagnosed and fixed at the right layer instead of lumped together — is the actual value of having this harness, more than any single green checkmark.

🌐 Language Support

Three modes, selectable live in the console:

English — fixed
Hindi-English — fixed Hinglish, matching how real Indian collections calls actually sound
"Nikki asks" — opens bilingually ("Hello, Namaste — would you prefer English or Hindi?"), then locks onto whichever language the borrower signals, for the LLM reply, the Deepgram STT model, and the TTS voice, all three

☁️ Deploy to Vercel

npm i -g vercel
vercel

Add GROQ_API_KEY and DEEPGRAM_API_KEY in Settings → Environment Variables, then:

vercel --prod

Before relying on this in production: lib/db.ts uses the local filesystem, which is ephemeral on serverless — swap it for Vercel KV or a free Neon Postgres instance. The three functions in lib/db.ts (listCalls, saveCall, clearCalls) are the only surface to change.

Mic reliability actually improves on Vercel — speech recognition needs HTTPS, which *.vercel.app gets automatically.

🔧 Troubleshooting

Symptom	Cause	Fix
`npm install` fails on the `next` version	A stray character got into `package.json`	Confirm the line reads exactly `"next": "14.2.35"`
Agent doesn't respond in the app	Missing/wrong `GROQ_API_KEY`	Check `.env.local`, then restart `npm run dev` — env vars only load on server start
Mic transcription fails	Missing/wrong `DEEPGRAM_API_KEY`, or mic permission denied	Check the error banner under the input box — it tells you exactly which
`npm run eval` → every case shows `Connection error.`	Node can't reach `api.groq.com`, even though `curl` can	Almost always an IPv4/IPv6 routing issue. Already fixed in the `eval` script via `NODE_OPTIONS=--dns-result-order=ipv4first` — if you still see this, check for a VPN or proxy env var (`env \| grep -i proxy`)
Chrome's mic just clicks on then off	Chrome's built-in `SpeechRecognition` API is unrelated to this app — it silently streams to an undocumented Google service that can fail independent of your network	This app uses Deepgram instead specifically because of this unreliability; if you're testing Chrome's raw demo page and seeing this, it's a Chrome/Google issue, not this project

🗺️ Known Limitations & Roadmap

Limitation	Why it's there	Next step
No real telephony	Browser mic stands in for a phone line	Twilio Media Streams — same conversation engine, new I/O layer
No streaming responses	Full reply generated before TTS starts	Groq streaming + sentence-level TTS chunking
No barge-in / interruption	Borrower can't cut Nikki off mid-sentence	Client-side audio-level detection to cancel TTS
File-based call log	Fine for local dev, not serverless-safe	Swap `lib/db.ts` for Postgres/Vercel KV
Hindi TTS quality	Browser voice engine, not a dedicated Indic TTS model	ElevenLabs/Azure TTS (free tier too limited for reliable demo use)
Eval harness has no CI integration	Runs manually from the terminal	GitHub Actions workflow to run `npm run eval` on every PR

Built as a portfolio project targeting Voice AI / Forward Deployed Engineer roles.

Name		Name	Last commit message	Last commit date
app		app
components		components
data		data
eval-results		eval-results
lib		lib
scripts		scripts
.env.local.example		.env.local.example
.gitignore		.gitignore
README.md		README.md
next-env.d.ts		next-env.d.ts
next.config.js		next.config.js
package-lock.json		package-lock.json
package.json		package.json
postcss.config.js		postcss.config.js
tailwind.config.ts		tailwind.config.ts
tsconfig.json		tsconfig.json

Navigation Menu

Repository files navigation

🎙️ Nikki

Voice AI Collections Agent

💡 Why this exists

📋 Table of Contents

✨ Features

🏗️ Architecture

🔄 Call Flow — State Machine

🧰 Tech Stack

📁 Project Structure

🚀 Quick Start (End-to-End)

1. Clone and install

2. Get your API keys

3. Set up environment variables

4. Run the app

5. Run the eval harness

6. Deploy it live

🔑 Environment Variables

🏢 Client Configuration Layer

🛡️ Compliance Monitoring Layer

🧪 Eval Harness — Deep Dive

What it is

Why it exists

How it actually works, step by step

Why scripted lines, not an LLM-simulated borrower

The 9 test cases

What "assertions" means concretely

Reading the output

🌐 Language Support

☁️ Deploy to Vercel

🔧 Troubleshooting

🗺️ Known Limitations & Roadmap

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

🎙️ Nikki

Voice AI Collections Agent

💡 Why this exists

📋 Table of Contents

✨ Features

🏗️ Architecture

🔄 Call Flow — State Machine

🧰 Tech Stack

📁 Project Structure

🚀 Quick Start (End-to-End)

1. Clone and install

2. Get your API keys

3. Set up environment variables

4. Run the app

5. Run the eval harness

6. Deploy it live

🔑 Environment Variables

🏢 Client Configuration Layer

🛡️ Compliance Monitoring Layer

🧪 Eval Harness — Deep Dive

What it is

Why it exists

How it actually works, step by step

Why scripted lines, not an LLM-simulated borrower

The 9 test cases

What "assertions" means concretely

Reading the output

🌐 Language Support

☁️ Deploy to Vercel

🔧 Troubleshooting

🗺️ Known Limitations & Roadmap

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages