refusal-radar

Detect and classify LLM refusals — explicit "I can't help with that", policy objections, capability disclaimers, and soft deflections — with a transparent score, in a single zero-dependency package.

When you run evals, red-team prompts, or guardrail tests, the hardest thing to measure automatically is "did the model actually answer, or did it refuse?" refusal-radar answers that with explainable, weighted pattern matching — no model calls, no API keys, no network, no dependencies.

• 🎯 Five categories — hard_refusal, policy_objection, capability_disclaimer, soft_deflection, and partial_compliance (refused and offered something).
• 📊 Calibrated score in [0, 1] via noisy-OR signal combination, with a tunable threshold.
• 🔎 Explainable — every verdict lists the exact phrases (signals) that drove it.
• 📦 JSONL batch mode for whole eval runs, with summary stats by category.
• 🚦 CI gate — --ci exits non-zero when a refusal is found, so a regression fails your build.
• 🪶 Zero dependencies, ESM, works as a CLI and a library.

---

Install

# Run instantly, no install:
npx refusal-radar "As an AI, I cannot do that."

# Or install globally / as a dependency:
npm install -g refusal-radar      # CLI
npm install refusal-radar         # library

Requires Node.js ≥ 18.

CLI usage

Inspect a single response:

refusal-radar "I'm sorry, but I cannot help with that. It goes against my guidelines."

REFUSAL  score=0.990  category=hard_refusal  (threshold 0.5)
signals:
   +0.92 hard_refusal/cannot_assist  "I cannot help"
   +0.88 policy_objection/against_guidelines  "against my guidelines"

Score a whole eval run stored as JSONL (one JSON object per line):

refusal-radar --file evals.jsonl --field response

#1 REFUSAL score=0.920 hard_refusal         "I'm sorry, but I cannot provide..."
#4 REFUSAL score=0.598 partial_compliance   "I can't help with that, but..."
#5 REFUSAL score=0.770 capability_disclaimer "As an AI language model, I..."
#6 REFUSAL score=0.984 hard_refusal         "I must decline. Creating malware..."

Summary
  total:    8
  refusals: 4  (50.0%)
    hard_refusal: 2
    partial_compliance: 1
    capability_disclaimer: 1

Pipe from another tool, and fail CI if any refusal slips through:

cat runs.jsonl | refusal-radar --stdin --field output --ci --quiet
echo $?   # 1 if a refusal was detected, 0 otherwise

Options

Flag	Description
-f, --file <path>	Read newline-delimited JSON (JSONL) from a file
--stdin	Treat piped stdin as JSONL (otherwise stdin is one string)
--field <name>	Field/path to inspect per record, e.g. choices.0.text
-t, --threshold <n>	Refusal score threshold in [0,1] (default 0.5)
--json	Emit machine-readable JSON
-q, --quiet	Batch mode: print only the summary
--ci	Exit 1 if any refusal is detected
--no-color	Disable ANSI colors
-h, --help / -v, --version	Help / version

Library usage

import { detectRefusal, detectBatch, summarize, parseJsonl } from 'refusal-radar';

const result = detectRefusal("As an AI, I don't have access to real-time data.");
// {
//   isRefusal: true,
//   score: 0.77,
//   category: 'capability_disclaimer',
//   signals: [ { id, category, weight, match, index }, ... ],
//   threshold: 0.5
// }

// Batch over parsed JSONL records:
const records = parseJsonl(fileContents);
const detections = detectBatch(records, { field: 'response', threshold: 0.6 });
const stats = summarize(detections);
// { total, refusals, rate, byCategory }

API

• detectRefusal(text, options?) → result object. Options: threshold, leadWindow, leadBoost, patterns.
• detectBatch(records, options?) → array of { index, text, record, refusal }. Records may be strings or objects; pass field to choose which field to read.
• summarize(detections) → { total, refusals, rate, byCategory }.
• parseJsonl(content) → array of objects (tolerant of blank lines, reports the line number on bad JSON).
• extractText(record, field?) → the string to inspect for a record.
• PATTERNS, CATEGORIES → the built-in catalog, exported so you can extend or replace it.

How the score works

Each built-in pattern carries a weight in [0, 1]. Matching positive signals are combined with a noisy-OR (1 − ∏(1 − wᵢ)) so the score rises with corroborating evidence but never exceeds 1. Compliance markers like "Sure, here is…" carry negative weight and multiplicatively discount the score — that's how a "I can't do X, but here's Y" response lands in partial_compliance rather than a clean refusal. Signals near the start of the response (where models usually refuse) get a small boost.

Because it's pure pattern matching, it's instant, deterministic, and fully auditable — but it does not understand semantics. Treat the score as a strong, explainable heuristic for eval triage, not as ground truth. Tune --threshold and extend PATTERNS to fit your model and domain.

Why

Refusals are a first-class metric for safety evals, jailbreak research, and helpfulness regressions — but they're usually checked with ad-hoc substring greps that break on the next phrasing. refusal-radar turns that into a small, tested, reusable tool with categories, scoring, and a CI gate.

Development

git clone https://github.com/Ayubjon/refusal-radar.git
cd refusal-radar
node --test            # run the test suite (zero dependencies)

Support

This project is free and open source. If it saves you time, an optional crypto tip is always welcome (never required):

• USDT — Ethereum (ERC-20): 0xad39bdf2df0b8dd6991150fcea0a156150ed19b8
• Verify on Etherscan

Please send only on the Ethereum (ERC-20) network.

License

MIT © 2026 Ayubjon