feat(route): add the hamel blog #17428
Merged
+88
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Successfully generated as following: http://localhost:1200/hamel/blog - Success ✔️<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
<channel>
<title>Hamel's Blog</title>
<link>https://hamel.dev</link>
<atom:link href="http://localhost:1200/hamel/blog" rel="self" type="application/rss+xml"></atom:link>
<description>Hamel's Blog - Powered by RSSHub</description>
<generator>RSSHub</generator>
<webMaster>contact@rsshub.app (RSSHub)</webMaster>
<language>en</language>
<lastBuildDate>Sun, 03 Nov 2024 08:19:36 GMT</lastBuildDate>
<ttl>5</ttl>
<item>
<title>Creating a LLM-as-a-Judge That Drives Business Results</title>
<description><header id="title-block-header" class="quarto-title-block default">
<div class="quarto-title">
<h1 class="title">Creating a LLM-as-a-Judge That Drives Business Results</h1>
<div class="quarto-categories">
<div class="quarto-category">LLMs</div>
<div class="quarto-category">evals</div>
</div>
</div>
<div>
<div class="description">
A step-by-step guide with my learnings from 30+ AI implementations.
</div>
</div>
<div class="quarto-title-meta">
<div>
<div class="quarto-title-meta-heading">Author</div>
<div class="quarto-title-meta-contents">
<p>Hamel Husain </p>
</div>
</div>
<div>
<div class="quarto-title-meta-heading">Published</div>
<div class="quarto-title-meta-contents">
<p class="date">October 29, 2024</p>
</div>
</div>
</div>
</header>
<nav id="TOC-body" role="doc-toc">
<h2 id="toc-title">Table Of Contents</h2>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#the-problem-ai-teams-are-drowning-in-data" id="toc-the-problem-ai-teams-are-drowning-in-data">The Problem: AI Teams Are Drowning in Data</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-1-find-the-principal-domain-expert" id="toc-step-1-find-the-principal-domain-expert">Step 1: Find <em>The</em> Principal Domain Expert</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#next-steps" id="toc-next-steps">Next Steps</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-2-create-a-dataset" id="toc-step-2-create-a-dataset">Step 2: Create a Dataset</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#why-a-diverse-dataset-matters" id="toc-why-a-diverse-dataset-matters">Why a Diverse Dataset Matters</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#dimensions-for-structuring-your-dataset" id="toc-dimensions-for-structuring-your-dataset">Dimensions for Structuring Your Dataset</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#examples-of-features-scenarios-and-personas" id="toc-examples-of-features-scenarios-and-personas">Examples of Features, Scenarios, and Personas</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#this-taxonomy-is-not-universal" id="toc-this-taxonomy-is-not-universal">This taxonomy is not universal</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#generating-data" id="toc-generating-data">Generating Data</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#example-llm-prompts-for-generating-user-inputs" id="toc-example-llm-prompts-for-generating-user-inputs">Example LLM Prompts for Generating User Inputs</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#generating-synthetic-data" id="toc-generating-synthetic-data">Generating Synthetic Data</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#next-steps-1" id="toc-next-steps-1">Next Steps</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-3-direct-the-domain-expert-to-make-passfail-judgments-with-critiques" id="toc-step-3-direct-the-domain-expert-to-make-passfail-judgments-with-critiques">Step 3: Direct The Domain Expert to Make Pass/Fail Judgments with Critiques</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#why-are-simple-passfail-metrics-important" id="toc-why-are-simple-passfail-metrics-important">Why are simple pass/fail metrics important?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#the-role-of-critiques" id="toc-the-role-of-critiques">The Role of Critiques</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#examples-of-good-critiques" id="toc-examples-of-good-critiques">Examples of Good Critiques</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#dont-stray-from-binary-passfail-judgments-when-starting-out" id="toc-dont-stray-from-binary-passfail-judgments-when-starting-out">Don’t stray from binary pass/fail judgments when starting out</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#make-it-easy-for-the-domain-expert-to-review-data" id="toc-make-it-easy-for-the-domain-expert-to-review-data">Make it easy for the domain expert to review data</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-many-examples-do-you-need" id="toc-how-many-examples-do-you-need">How many examples do you need?</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-4-fix-errors" id="toc-step-4-fix-errors">Step 4: Fix Errors</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-5-build-your-llm-as-a-judge-iteratively" id="toc-step-5-build-your-llm-as-a-judge-iteratively">Step 5: Build Your LLM as A Judge, Iteratively</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#the-hidden-power-of-critiques" id="toc-the-hidden-power-of-critiques">The Hidden Power of Critiques</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#start-with-expert-examples" id="toc-start-with-expert-examples">Start with Expert Examples</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#keep-iterating-on-the-prompt-until-convergence-with-domain-expert" id="toc-keep-iterating-on-the-prompt-until-convergence-with-domain-expert">Keep Iterating on the Prompt Until Convergence With Domain Expert</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-to-optimize-the-llm-judge-prompt" id="toc-how-to-optimize-the-llm-judge-prompt">How to Optimize the LLM Judge Prompt?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#the-human-side-of-the-process" id="toc-the-human-side-of-the-process">The Human Side of the Process</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-often-should-you-evaluate" id="toc-how-often-should-you-evaluate">How Often Should You Evaluate?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#what-if-this-doesnt-work" id="toc-what-if-this-doesnt-work">What if this doesn’t work?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#mistakes-ive-noticed-in-llm-judge-prompts" id="toc-mistakes-ive-noticed-in-llm-judge-prompts">Mistakes I’ve noticed in LLM judge prompts</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-6-perform-error-analysis" id="toc-step-6-perform-error-analysis">Step 6: Perform Error Analysis</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#classify-traces" id="toc-classify-traces">Classify Traces</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#an-interactive-walkthrough-of-error-analysis" id="toc-an-interactive-walkthrough-of-error-analysis">An Interactive Walkthrough of Error Analysis</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#fix-your-errors-again" id="toc-fix-your-errors-again">Fix Your Errors, Again</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#doing-this-well-requires-data-literacy" id="toc-doing-this-well-requires-data-literacy">Doing this well requires data literacy</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-7-create-more-specialized-llm-judges-if-needed" id="toc-step-7-create-more-specialized-llm-judges-if-needed">Step 7: Create More Specialized LLM Judges (if needed)</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#recap-of-critique-shadowing" id="toc-recap-of-critique-shadowing">Recap of Critique Shadowing</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#its-not-the-judge-that-created-value-afterall" id="toc-its-not-the-judge-that-created-value-afterall">It’s Not The Judge That Created Value, Afterall</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#do-you-really-need-this" id="toc-do-you-really-need-this">Do You Really Need This?</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#faq" id="toc-faq">FAQ</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#if-i-have-a-good-judge-llm-isnt-that-also-the-llm-id-also-want-to-use" id="toc-if-i-have-a-good-judge-llm-isnt-that-also-the-llm-id-also-want-to-use">If I have a good judge LLM, isn’t that also the LLM I’d also want to use?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#do-you-recommend-fine-tuning-judges" id="toc-do-you-recommend-fine-tuning-judges">Do you recommend fine-tuning judges?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#whats-wrong-with-off-the-shelf-llm-judges" id="toc-whats-wrong-with-off-the-shelf-llm-judges">What’s wrong with off-the-shelf LLM judges?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-do-you-evaluate-the-llm-judge" id="toc-how-do-you-evaluate-the-llm-judge">How Do you evaluate the LLM judge?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#what-model-do-you-use-for-the-llm-judge" id="toc-what-model-do-you-use-for-the-llm-judge">What model do you use for the LLM judge?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#what-about-guardrails" id="toc-what-about-guardrails">What about guardrails?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#im-using-llm-as-a-judge-and-getting-tremendous-value-but-i-didnt-follow-this-approach." id="toc-im-using-llm-as-a-judge-and-getting-tremendous-value-but-i-didnt-follow-this-approach.">I’m using LLM as a judge, and getting tremendous value but I didn’t follow this approach.</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-do-you-choose-between-traditional-ml-techniques-llm-as-a-judge-and-human-annotations" id="toc-how-do-you-choose-between-traditional-ml-techniques-llm-as-a-judge-and-human-annotations">How do you choose between traditional ML techniques, LLM-as-a-judge and human annotations?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#can-you-make-judges-from-small-models" id="toc-can-you-make-judges-from-small-models">Can you make judges from small models?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-do-you-ensure-consistency-when-updating-your-llm-model" id="toc-how-do-you-ensure-consistency-when-updating-your-llm-model">How do you ensure consistency when updating your LLM model?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-do-you-phase-out-human-in-the-loop-to-scale-this" id="toc-how-do-you-phase-out-human-in-the-loop-to-scale-this">How do you phase out human in the loop to scale this?</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#resources" id="toc-resources">Resources</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#stay-connected" id="toc-stay-connected">Stay Connected</a></li>
</ul>
</nav>
<p>Earlier this year, I wrote <a href="https://hamel.dev/blog/posts/evals/">Your AI product needs evals</a>. Many of you asked, “How do I get started with LLM-as-a-judge?” This guide shares what I’ve learned after helping over <a href="https://parlance-labs.com/">30 companies</a> set up their evaluation systems.</p>
<section id="the-problem-ai-teams-are-drowning-in-data" class="level2">
<h2 class="anchored" data-anchor-id="the-problem-ai-teams-are-drowning-in-data">The Problem: AI Teams Are Drowning in Data</h2>
<p>Ever spend weeks building an AI system, only to realize you have no idea if it’s actually working? You’re not alone. I’ve noticed teams repeat the same mistakes when using LLMs to evaluate AI outputs:</p>
<ol type="1">
<li><strong>Too Many Metrics</strong>: Creating numerous measurements that become unmanageable.</li>
<li><strong>Arbitrary Scoring Systems</strong>: Using uncalibrated scales (like 1-5) across multiple dimensions, where the difference between scores is unclear and subjective. What makes something a 3 versus a 4? Nobody knows, and different evaluators often interpret these scales differently.</li>
<li><strong>Ignoring Domain Experts</strong>: Not involving the people who understand the subject matter deeply.</li>
<li><strong>Unvalidated Metrics</strong>: Using measurements that don’t truly reflect what matters to the users or the business.</li>
</ol>
<p>The result? Teams end up buried under mountains of metrics or data they don’t trust and can’t use. Progress grinds to a halt. Everyone gets frustrated.</p>
<p>For example, it’s not uncommon for me to see dashboards that look like this:</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://hamel.dev/blog/posts/llm-judge/blog_header.png" class="img-fluid figure-img" referrerpolicy="no-referrer"></p>
<figcaption>An illustrative example of a bad eval dashboard</figcaption>
</figure>
</div>
<p>Tracking a bunch of scores on a 1-5 scale is often a sign of a bad eval process (I’ll discuss why later). In this post, I’ll show you how to avoid these pitfalls. The solution is to use a technique that I call <strong>“Critique Shadowing”</strong>. Here’s how to do it, step by step.</p>
</section>
<section id="step-1-find-the-principal-domain-expert" class="level2">
<h2 class="anchored" data-anchor-id="step-1-find-the-principal-domain-expert">Step 1: Find <em>The</em> Principal Domain Expert</h2>
<p>In most organizations there is usually one (maybe two) key individuals whose judgment is crucial for the success of your AI product. These are the people with deep domain expertise or represent your target users. Identifying and involving this <strong>Principal Domain Expert</strong> early in the process is critical.</p>
<p><strong>Why is finding the right domain expert so important?</strong></p>
<ul>
<li><p><strong>They Set the Standard</strong>: This person not only defines what is acceptable technically, but also helps you understand if you’re building something users actually want.</p></li>
<li><p><strong>Capture Unspoken Expectations</strong>: By involving them, you uncover their preferences and expectations, which they might not be able to fully articulate upfront. Through the evaluation process, you help them clarify what a “passable” AI interaction looks like.</p></li>
<li><p><strong>Consistency in Judgment</strong>: People in your organization may have different opinions about the AI’s performance. Focusing on the principal expert ensures that evaluations are consistent and aligned with the most critical standards.</p></li>
<li><p><strong>Sense of Ownership</strong>: Involving the expert gives them a stake in the AI’s development. They feel invested because they’ve had a hand in shaping it. In the end, they are more likely to approve of the AI.</p></li>
</ul>
<p><strong>Examples of Principal Domain Experts:</strong></p>
<ul>
<li>A <strong>psychologist</strong> for a mental health AI assistant.</li>
<li>A <strong>lawyer</strong> for an AI that analyzes legal documents.</li>
<li>A <strong>customer service director</strong> for a support chatbot.</li>
<li>A <strong>lead teacher or curriculum developer</strong> for an educational AI tool.</li>
</ul>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Exceptions
</div>
</div>
<div class="callout-body-container callout-body">
<p>In a smaller company, this might be the CEO or founder. If you are an independent developer, you should be the domain expert (but be honest with yourself about your expertise).</p>
<p>If you must rely on leadership, you should regularly validate their assumptions against real user feedback.</p>
</div>
</div>
<p>Many developers attempt to act as the domain expert themselves, or find a convenient proxy (ex: their superior). This is a recipe for disaster. People will have varying opinions about what is acceptable, and you can’t make everyone happy. What’s important is that your principal domain expert is satisfied.</p>
<p><strong>Remember:</strong> This doesn’t have to take a lot of the domain expert’s time. Later in this post, I’ll discuss how you can make the process efficient. Their involvement is absolutely critical to the AI’s success.</p>
<section id="next-steps" class="level3">
<h3 class="anchored" data-anchor-id="next-steps">Next Steps</h3>
<p>Once you’ve found your expert, we need to give them the right data to review. Let’s talk about how to do that next.</p>
</section>
</section>
<section id="step-2-create-a-dataset" class="level2">
<h2 class="anchored" data-anchor-id="step-2-create-a-dataset">Step 2: Create a Dataset</h2>
<p>With your principal domain expert on board, the next step is to build a dataset that captures problems that your AI will encounter. It’s important that the dataset is diverse and represents the types of interactions that your AI will have in production.</p>
<section id="why-a-diverse-dataset-matters" class="level3">
<h3 class="anchored" data-anchor-id="why-a-diverse-dataset-matters">Why a Diverse Dataset Matters</h3>
<ul>
<li><strong>Comprehensive Testing</strong>: Ensures your AI is evaluated across a wide range of situations.</li>
<li><strong>Realistic Interactions</strong>: Reflects actual user behavior for more relevant evaluations.</li>
<li><strong>Identifies Weaknesses</strong>: Helps uncover areas where the AI may struggle or produce errors.</li>
</ul>
</section>
<section id="dimensions-for-structuring-your-dataset" class="level3">
<h3 class="anchored" data-anchor-id="dimensions-for-structuring-your-dataset">Dimensions for Structuring Your Dataset</h3>
<p>You want to define dimensions that make sense for your use case. For example, here are ones that I often use for B2C applications:</p>
<ol type="1">
<li><strong>Features</strong>: Specific functionalities of your AI product.</li>
<li><strong>Scenarios</strong>: Situations or problems the AI may encounter and needs to handle.</li>
<li><strong>Personas</strong>: Representative user profiles with distinct characteristics and needs.</li>
</ol>
</section>
<section id="examples-of-features-scenarios-and-personas" class="level3">
<h3 class="anchored" data-anchor-id="examples-of-features-scenarios-and-personas">Examples of Features, Scenarios, and Personas</h3>
<section id="features" class="level4">
<h4 class="anchored" data-anchor-id="features">Features</h4>
<table class="caption-top table">
<colgroup>
<col style="width: 30%">
<col style="width: 69%">
</colgroup>
<thead>
<tr class="header">
<th><strong>Feature</strong></th>
<th><strong>Description</strong></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>Email Summarization</strong></td>
<td>Condensing lengthy emails into key points.</td>
</tr>
<tr class="even">
<td><strong>Meeting Scheduler</strong></td>
<td>Automating the scheduling of meetings across time zones.</td>
</tr>
<tr class="odd">
<td><strong>Order Tracking</strong></td>
<td>Providing shipment status and delivery updates.</td>
</tr>
<tr class="even">
<td><strong>Contact Search</strong></td>
<td>Finding and retrieving contact information from a database.</td>
</tr>
<tr class="odd">
<td><strong>Language Translation</strong></td>
<td>Translating text between languages.</td>
</tr>
<tr class="even">
<td><strong>Content Recommendation</strong></td>
<td>Suggesting articles or products based on user interests.</td>
</tr>
</tbody>
</table>
</section>
<section id="scenarios" class="level4">
<h4 class="anchored" data-anchor-id="scenarios">Scenarios</h4>
<p>Scenarios are situations the AI needs to handle, (not based on the outcome of the AI’s response).</p>
<table class="caption-top table">
<colgroup>
<col style="width: 31%">
<col style="width: 68%">
</colgroup>
<thead>
<tr class="header">
<th><strong>Scenario</strong></th>
<th><strong>Description</strong></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>Multiple Matches Found</strong></td>
<td>User’s request yields multiple results that need narrowing down. For example: User asks “Where’s my order?” but has three active orders (#123, #124, #125). AI must help identify which specific order they’re asking about.</td>
</tr>
<tr class="even">
<td><strong>No Matches Found</strong></td>
<td>User’s request yields no results, requiring alternatives or corrections. For example: User searches for order #ABC-123 which doesn’t exist. AI should explain valid order formats and suggest checking their confirmation email.</td>
</tr>
<tr class="odd">
<td><strong>Ambiguous Request</strong></td>
<td>User input lacks necessary specificity. For example: User says “I need to change my delivery” without specifying which order or what aspect of delivery (date, address, etc.) they want to change.</td>
</tr>
<tr class="even">
<td><strong>Invalid Data Provided</strong></td>
<td>User provides incorrect data type or format. For example: User tries to track a return using a regular order number instead of a return authorization (RMA) number.</td>
</tr>
<tr class="odd">
<td><strong>System Errors</strong></td>
<td>Technical issues prevent normal operation. For example: While looking up an order, the inventory database is temporarily unavailable. AI needs to explain the situation and provide alternatives.</td>
</tr>
<tr class="even">
<td><strong>Incomplete Information</strong></td>
<td>User omits required details. For example: User wants to initiate a return but hasn’t provided the order number or reason. AI needs to collect this information step by step.</td>
</tr>
<tr class="odd">
<td><strong>Unsupported Feature</strong></td>
<td>User requests functionality that doesn’t exist. For example: User asks to change payment method after order has shipped. AI must explain why this isn’t possible and suggest alternatives.</td>
</tr>
</tbody>
</table>
</section>
<section id="personas" class="level4">
<h4 class="anchored" data-anchor-id="personas">Personas</h4>
<table class="caption-top table">
<colgroup>
<col style="width: 27%">
<col style="width: 72%">
</colgroup>
<thead>
<tr class="header">
<th><strong>Persona</strong></th>
<th><strong>Description</strong></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>New User</strong></td>
<td>Unfamiliar with the system; requires guidance.</td>
</tr>
<tr class="even">
<td><strong>Expert User</strong></td>
<td>Experienced; expects efficiency and advanced features.</td>
</tr>
<tr class="odd">
<td><strong>Non-Native Speaker</strong></td>
<td>May have language barriers; uses non-standard expressions.</td>
</tr>
<tr class="even">
<td><strong>Busy Professional</strong></td>
<td>Values quick, concise responses; often multitasking.</td>
</tr>
<tr class="odd">
<td><strong>Technophobe</strong></td>
<td>Uncomfortable with technology; needs simple instructions.</td>
</tr>
<tr class="even">
<td><strong>Elderly User</strong></td>
<td>May not be tech-savvy; requires patience and clear guidance.</td>
</tr>
</tbody>
</table>
</section>
</section>
<section id="this-taxonomy-is-not-universal" class="level3">
<h3 class="anchored" data-anchor-id="this-taxonomy-is-not-universal">This taxonomy is not universal</h3>
<p>This taxonomy (features, scenarios, personas) is not universal. For example, it may not make sense to even have personas if users aren’t directly engaging with your AI. The idea is you should outline dimensions that make sense for your use case and generate data that covers them. You’ll likely refine these after the first round of evaluations.</p>
</section>
<section id="generating-data" class="level3">
<h3 class="anchored" data-anchor-id="generating-data">Generating Data</h3>
<p>To build your dataset, you can:</p>
<ul>
<li><strong>Use Existing Data</strong>: Sample real user interactions or behaviors from your AI system.</li>
<li><strong>Generate Synthetic Data</strong>: Use LLMs to create realistic user inputs covering various features, scenarios, and personas.</li>
</ul>
<p>Often, you’ll do a combination of both to ensure comprehensive coverage. Synthetic data is not as good as real data, but it’s a good starting point. Also, we are only using LLMs to generate the user inputs, not the LLM responses or internal system behavior.</p>
<p>Regardless of whether you use existing data or synthetic data, you want good coverage across the dimensions you’ve defined.</p>
<p><strong>Incorporating System Information</strong></p>
<p>When making test data, use your APIs and databases where appropriate. This will create realistic data and trigger the right scenarios. Sometimes you’ll need to write simple programs to get this information. That’s what the “Assumptions” column is referring to in the examples below.</p>
</section>
<section id="example-llm-prompts-for-generating-user-inputs" class="level3">
<h3 class="anchored" data-anchor-id="example-llm-prompts-for-generating-user-inputs">Example LLM Prompts for Generating User Inputs</h3>
<p>Here are some example prompts that illustrate how to use an LLM to generate synthetic <strong>user inputs</strong> for different combinations of features, scenarios, and personas:</p>
<table class="caption-top table">
<colgroup>
<col style="width: 3%">
<col style="width: 11%">
<col style="width: 11%">
<col style="width: 10%">
<col style="width: 35%">
<col style="width: 26%">
</colgroup>
<thead>
<tr class="header">
<th><strong>ID</strong></th>
<th><strong>Feature</strong></th>
<th><strong>Scenario</strong></th>
<th><strong>Persona</strong></th>
<th><strong>LLM Prompt to Generate User Input</strong></th>
<th>Assumptions (not directly in the prompt)</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>1</td>
<td><strong>Order Tracking</strong></td>
<td>Invalid Data Provided</td>
<td>Frustrated Customer</td>
<td>“Generate a user input from someone who is clearly irritated and impatient, using short, terse language to demand information about their order status for order number <strong>#1234567890</strong>. Include hints of previous negative experiences.”</td>
<td>Order number <strong>#1234567890</strong> does <strong>not</strong> exist in the system.</td>
</tr>
<tr class="even">
<td>2</td>
<td><strong>Contact Search</strong></td>
<td>Multiple Matches Found</td>
<td>New User</td>
<td>“Create a user input from someone who seems unfamiliar with the system, using hesitant language and asking for help to find contact information for a person named ‘Alex’. The user should appear unsure about what information is needed.”</td>
<td>Multiple contacts named ‘Alex’ exist in the system.</td>
</tr>
<tr class="odd">
<td>3</td>
<td><strong>Meeting Scheduler</strong></td>
<td>Ambiguous Request</td>
<td>Busy Professional</td>
<td>“Simulate a user input from someone who is clearly in a hurry, using abbreviated language and minimal details to request scheduling a meeting. The message should feel rushed and lack specific information.”</td>
<td>N/A</td>
</tr>
<tr class="even">
<td>4</td>
<td><strong>Content Recommendation</strong></td>
<td>No Matches Found</td>
<td>Expert User</td>
<td>“Produce a user input from someone who demonstrates in-depth knowledge of their industry, using specific terminology to request articles on sustainable supply chain management. Use the information in this article involving sustainable supply chain management to formulate a plausible query: {{article}}”</td>
<td>No articles on ‘Emerging trends in sustainable supply chain management’ exist in the system.</td>
</tr>
</tbody>
</table>
</section>
<section id="generating-synthetic-data" class="level3">
<h3 class="anchored" data-anchor-id="generating-synthetic-data">Generating Synthetic Data</h3>
<p>When generating synthetic data, you only need to create the user inputs. You then feed these inputs into your AI system to generate the AI’s responses. It’s important that you log everything so you can evaluate your AI. To recap, here’s the process:</p>
<ol type="1">
<li><strong>Generate User Inputs</strong>: Use the LLM prompts to create realistic user inputs.</li>
<li><strong>Feed Inputs into Your AI System</strong>: Input the user interactions into your AI as it currently exists.</li>
<li><strong>Capture AI Responses</strong>: Record the AI’s responses to form complete interactions.</li>
<li><strong>Organize the Interactions</strong>: Create a table to store the user inputs, AI responses, and relevant metadata.</li>
</ol>
<section id="how-much-data-should-you-generate" class="level4">
<h4 class="anchored" data-anchor-id="how-much-data-should-you-generate">How much data should you generate?</h4>
<p>There is no right answer here. At a minimum, you want to generate enough data so that you have examples for each combination of dimensions (in this toy example: features, scenarios, and personas). However, you also want to keep generating more data until you feel like you have stopped seeing new failure modes. The amount of data I generate varies significantly depending on the use case.</p>
</section>
<section id="does-synthetic-data-actually-work" class="level4">
<h4 class="anchored" data-anchor-id="does-synthetic-data-actually-work">Does synthetic data actually work?</h4>
<p>You might be skeptical of using synthetic data. After all, it’s not real data, so how can it be a good proxy? In my experience, it works surprisingly well. Some of my favorite AI products, like <a href="https://hex.tech/">Hex</a> use synthetic data to power their evals:</p>
<blockquote class="blockquote">
<p>“LLMs are surprisingly good at generating excellent - and diverse - examples of user prompts. This can be relevant for powering application features, and sneakily, for building Evals. If this sounds a bit like the Large Language Snake is eating its tail, I was just as surprised as you! All I can say is: it works, ship it.” <em><a href="https://www.linkedin.com/in/bryan-bischof/">Bryan Bischof</a>, Head of AI Engineering at Hex</em></p>
</blockquote>
</section>
</section>
<section id="next-steps-1" class="level3">
<h3 class="anchored" data-anchor-id="next-steps-1">Next Steps</h3>
<p>With your dataset ready, now comes the most important part: getting your principal domain expert to evaluate the interactions.</p>
</section>
</section>
<section id="step-3-direct-the-domain-expert-to-make-passfail-judgments-with-critiques" class="level2">
<h2 class="anchored" data-anchor-id="step-3-direct-the-domain-expert-to-make-passfail-judgments-with-critiques">Step 3: Direct The Domain Expert to Make Pass/Fail Judgments with Critiques</h2>
<p>The domain expert’s job is to focus on one thing: <strong>“Did the AI achieve the desired outcome?”</strong> No complex scoring scales or multiple metrics. Just a clear <strong>pass or fail</strong> decision. In addition to the pass/fail decision, the domain expert should write a critique that explains their reasoning.</p>
<section id="why-are-simple-passfail-metrics-important" class="level3">
<h3 class="anchored" data-anchor-id="why-are-simple-passfail-metrics-important">Why are simple pass/fail metrics important?</h3>
<ul>
<li><p><strong>Clarity and Focus</strong>: A binary decision forces everyone to consider what truly matters. It simplifies the evaluation to a single, crucial question.</p></li>
<li><p><strong>Actionable Insights</strong>: Pass/fail judgments are easy to interpret and act upon. They help you quickly identify whether the AI meets the user’s needs.</p></li>
<li><p><strong>Forces Articulation of Expectations</strong>: When domain experts must decide if an interaction passes or fails, they are compelled to articulate their expectations clearly. This process uncovers nuances and unspoken assumptions about how the AI should behave.</p></li>
<li><p><strong>Efficient Use of Resources</strong>: Keeps the evaluation process manageable, especially when starting out. You avoid getting bogged down in detailed metrics that might not be meaningful yet.</p></li>
</ul>
</section>
<section id="the-role-of-critiques" class="level3">
<h3 class="anchored" data-anchor-id="the-role-of-critiques">The Role of Critiques</h3>
<p>Alongside a binary pass/fail judgment, it’s important to write a detailed critique of the LLM-generated output. These critiques:</p>
<ul>
<li><p><strong>Captures Nuances</strong>: The critique allows you to note if something was mostly correct but had areas for improvement.</p></li>
<li><p><strong>Guide Improvement</strong>: Detailed feedback provides specific insights into how the AI can be enhanced.</p></li>
<li><p><strong>Balance Simplicity with Depth</strong>: While the pass/fail offers a clear verdict, the critique offers the depth needed to understand the reasoning behind the judgment.</p></li>
</ul>
<p><strong>Why Write Critiques?:</strong></p>
<p>In practice, domain experts may not have fully internalized all the judgment criteria. By forcing them to make a pass/fail decision and explain their reasoning, they clarify their expectations and provide valuable guidance for refining the AI.</p>
<p><em>“But my problem is complex!”</em> Trust me—starting simple forces you to focus on what truly matters. You can introduce more complexity later if needed.</p>
</section>
<section id="examples-of-good-critiques" class="level3">
<h3 class="anchored" data-anchor-id="examples-of-good-critiques">Examples of Good Critiques</h3>
<p>To illustrate how simple pass/fail judgments combined with detailed critiques work in practice, here’s a table showcasing examples of user interactions with an AI assistant. The table includes both <strong>passes</strong> and <strong>fails</strong>, with critiques explaining why the AI received that judgment. In cases where the AI passed despite critical issues, the critique highlights these aspects and justifies why it still passed overall. For failed interactions, the critique explains the critical elements that led to the failure.</p>
<table class="caption-top table">
<colgroup>
<col style="width: 51%">
<col style="width: 24%">
<col style="width: 24%">
</colgroup>
<thead>
<tr class="header">
<th><strong>User Interaction with AI</strong></th>
<th><strong>Judgment</strong></th>
<th><strong>Critique</strong></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>User:</strong> “I need to cancel my flight booking for next week.”<br><strong>AI:</strong> “Your flight booking for next week has been canceled. A confirmation email has been sent to you.”</td>
<td><strong>Pass</strong></td>
<td>The AI successfully executed the user’s request by canceling the flight and sending a confirmation, achieving the primary goal. <strong>However</strong>, it should have asked for confirmation before proceeding to prevent accidental cancellations. Despite this critical oversight, the AI fulfilled the user’s request, so it <strong>passes</strong>, but adding a confirmation step would improve user safety.</td>
</tr>
<tr class="even">
<td><strong>User:</strong> “What’s the capital of France?”<br><strong>AI:</strong> “The capital of France is Paris.”</td>
<td><strong>Pass</strong></td>
<td>The AI correctly answered the user’s question, providing accurate information promptly. <strong>Although</strong> the response is sufficient, the AI could enhance the interaction by offering additional interesting facts or asking if the user has more questions. Despite this opportunity for added engagement, the AI met the user’s primary need, so it <strong>passes</strong>.</td>
</tr>
<tr class="odd">
<td><strong>User:</strong> “I can’t log into my account; it says my password is incorrect.”<br><strong>AI:</strong> “Please reset your password by clicking ‘Forgot Password’.”</td>
<td><strong>Fail</strong></td>
<td>The AI provided a generic solution without acknowledging the user’s frustration or checking for other issues like account lockout. It failed to offer empathetic support or additional assistance. Due to the lack of personalized help and poor user experience, it <strong>fails</strong>.</td>
</tr>
<tr class="even">
<td><strong>User:</strong> “Book a table for two at an Italian restaurant tonight at 7 PM.”<br><strong>AI:</strong> “Table for two at Luigi’s Italian Restaurant booked for tonight at 7 PM. Confirmation number is 12345.”</td>
<td><strong>Pass</strong></td>
<td>The AI efficiently completed the booking as requested, providing all necessary details and a confirmation number. <strong>However</strong>, it could have asked if the user had seating preferences or dietary restrictions. The AI accomplished the primary task successfully, so it <strong>passes</strong>, with minor opportunities for enhanced service.</td>
</tr>
<tr class="odd">
<td><strong>User:</strong> “I need help with my billing statement.”<br><strong>AI:</strong> “Please provide your account number and full name.”</td>
<td><strong>Fail</strong></td>
<td>The AI requested sensitive information without verifying the user’s identity through secure means, potentially compromising security. It should guide the user through a secure authentication process first. Due to this critical oversight in user data protection, it <strong>fails</strong>.</td>
</tr>
</tbody>
</table>
<p>These examples demonstrate how the AI can receive both <strong>“Pass”</strong> and <strong>“Fail”</strong> judgments. In the critiques:</p>
<ul>
<li><p>For <strong>passes</strong>, we explain why the AI succeeded in meeting the user’s primary need, even if there were critical aspects that could be improved. We highlight these areas for enhancement while justifying the overall passing judgment.</p></li>
<li><p>For <strong>fails</strong>, we identify the critical elements that led to the failure, explaining why the AI did not meet the user’s main objective or compromised important factors like user experience or security.</p></li>
</ul>
<p>Most importantly, <strong>the critique should be detailed enough so that you can use it in a few-shot prompt for a LLM judge</strong>. In other words, it should be detailed enough that a new employee could understand it. Being too terse is a common mistake.</p>
<p>Note that the example user interactions with the AI are simplified for brevity - but you might need to give the domain expert more context to make a judgement. More on that later.</p>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Note
</div>
</div>
<div class="callout-body-container callout-body">
<p>At this point, you don’t need to perform a root cause analysis into the technical reasons behind why the AI failed. Many times, it’s useful to get a sense of overall behavior before diving into the weeds.</p>
</div>
</div>
</section>
<section id="dont-stray-from-binary-passfail-judgments-when-starting-out" class="level3">
<h3 class="anchored" data-anchor-id="dont-stray-from-binary-passfail-judgments-when-starting-out">Don’t stray from binary pass/fail judgments when starting out</h3>
<p>A common mistake is straying from binary pass/fail judgments. Let’s revisit the dashboard from earlier:</p>
<p><img src="https://hamel.dev/blog/posts/llm-judge/dashboard.png" class="img-fluid" referrerpolicy="no-referrer"></p>
<p>If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you’re doing it wrong. Let’s unpack why.</p>
<ol type="1">
<li><strong>It’s not actionable</strong>: People don’t know what to do with a 3 or 4. It’s not immediately obvious how this number is better than a 2. You need to be able to say “this interaction passed because…” and “this interaction failed because…”.</li>
<li>More often than not, <strong>these metrics do not matter</strong>. Every time I’ve analyzed data on domain expert judgments, they tend not to correlate with these kind of metrics. By having a domain expert make a binary judgment, you can figure out what truly matters.</li>
</ol>
<p>This is why I hate off the shelf metrics that come with many evaluation frameworks. They tend to lead people astray.</p>
<p><strong>Common Objections to Pass/Fail Judgments:</strong></p>
<ul>
<li>“The business said that these 8 dimensions are important, so we need to evaluate all of them.”</li>
<li>“We need to be able to say why an interaction passed or failed.”</li>
</ul>
<p>I can guarantee you that if someone says you need to measure 8 things on a 1-5 scale, they don’t know what they are looking for. They are just guessing. You have to let the domain expert drive and make a pass/fail judgment with critiques so you can figure out what truly matters. Stand your ground here.</p>
</section>
<section id="make-it-easy-for-the-domain-expert-to-review-data" class="level3">
<h3 class="anchored" data-anchor-id="make-it-easy-for-the-domain-expert-to-review-data">Make it easy for the domain expert to review data</h3>
<p>Finally, you need to remove all friction from reviewing data. I’ve written about this <a href="https://hamel.dev/notes/llm/finetuning/data_cleaning.html">here</a>. Sometimes, you can just use a spreadsheet. It’s a judgement call in terms of what is easiest for the domain expert. I found that I often have to provide additional context to help the domain expert understand the user interaction, such as:</p>
<ul>
<li>Metadata about the user, such as their location, subscription tier, etc.</li>
<li>Additional context about the system, such as the current time, inventory levels, etc.</li>
<li>Resources so you can check if the AI’s response is correct (ex: ability to search a database, etc.)</li>
</ul>
<p>All of this data needs to be presented on a single screen so the domain expert can review it without jumping through hoops. That’s why I recommend building <a href="https://hamel.dev/notes/llm/finetuning/data_cleaning.html">a simple web app</a> to review data.</p>
</section>
<section id="how-many-examples-do-you-need" class="level3">
<h3 class="anchored" data-anchor-id="how-many-examples-do-you-need">How many examples do you need?</h3>
<p>The number of examples you need depends on the complexity of the task. My heuristic is that I start with around 30 examples and keep going until I do not see any new failure modes. From there, I keep going until I’m not learning anything new.</p>
<p>Next, we’ll look at how to use this data to build an LLM judge.</p>
</section>
</section>
<section id="step-4-fix-errors" class="level2">
<h2 class="anchored" data-anchor-id="step-4-fix-errors">Step 4: Fix Errors</h2>
<p>After looking at the data, it’s likely you will find errors in your AI system. Instead of plowing ahead and building an LLM judge, you want to fix any obvious errors. Remember, the whole point of the LLM as a judge is to help you find these errors, so it’s totally fine if you find them earlier!</p>
<p>If you have already developed <a href="https://hamel.dev/blog/posts/evals">Level 1 evals as outlined in my previous post</a>, you should not have any pervasive errors. However, these errors can sometimes slip through the cracks. If you find pervasive errors, fix them and go back to step 3. Keep iterating until you feel like you have stabilized your system.</p>
</section>
<section id="step-5-build-your-llm-as-a-judge-iteratively" class="level2">
<h2 class="anchored" data-anchor-id="step-5-build-your-llm-as-a-judge-iteratively">Step 5: Build Your LLM as A Judge, Iteratively</h2>
<section id="the-hidden-power-of-critiques" class="level3">
<h3 class="anchored" data-anchor-id="the-hidden-power-of-critiques">The Hidden Power of Critiques</h3>
<p>You cannot write a good judge prompt until you’ve seen the data. <a href="https://arxiv.org/abs/2404.12272">The paper from Shankar et al.,</a> “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences” summarizes this well:</p>
<blockquote class="blockquote">
<p>to grade outputs,people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria. We dub this phenomenon criteria drift, and it implies thatit is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.</p>
</blockquote>
</section>
<section id="start-with-expert-examples" class="level3">
<h3 class="anchored" data-anchor-id="start-with-expert-examples">Start with Expert Examples</h3>
<p>Let me share a real-world example of building an LLM judge you can apply to your own use case. When I was helping Honeycomb build their <a href="https://www.honeycomb.io/blog/introducing-query-assistant">Query Assistant feature</a>, we needed a way to evaluate if the AI was generating good queries. Here’s what our LLM judge prompt looked like, including few-shot examples of critiques from our domain expert, <a href="https://x.com/_cartermp">Phillip</a>:</p>
<pre class="text"><code>You are a Honeycomb query evaluator with advanced capabilities to judge if a query is good or not.
You understand the nuances of the Honeycomb query language, including what is likely to be
most useful from an analytics perspective.
Here is information about the Honeycomb query language:
{{query_language_info}}
Here are some guidelines for evaluating queries:
{{guidelines}}
Example evaluations:
&lt;examples&gt;
&lt;example-1&gt;
&lt;nlq&gt;show me traces where ip is 10.0.2.90&lt;/nlq&gt;
&lt;query&gt;
{
"breakdowns": ["trace.trace_id"],
"calculations": [{"op": "COUNT"}],
"filters": [{"column": "net.host.ip", "op": "=", "value": "10.0.2.90"}]
}
&lt;/query&gt;
&lt;critique&gt;
{
"critique": "The query correctly filters for traces with an IP address of 10.0.2.90
and counts the occurrences of those traces, grouped by trace.trace_id. The response
is good as it meets the requirement of showing traces from a specific IP address
without additional complexities.",
"outcome": "good"
}
&lt;/critique&gt;
&lt;/example-1&gt;
&lt;example-2&gt;
&lt;nlq&gt;show me slowest trace&lt;/nlq&gt;
&lt;query&gt;
{
"calculations": [{"column": "duration_ms", "op": "MAX"}],
"orders": [{"column": "duration_ms", "op": "MAX", "order": "descending"}],
"limit": 1
}
&lt;/query&gt;
&lt;critique&gt;
{
"critique": "While the query attempts to find the slowest trace using MAX(duration_ms)
and ordering correctly, it fails to group by trace.trace_id. Without this grouping,
the query only shows the MAX(duration_ms) measurement over time, not the actual
slowest trace.",
"outcome": "bad"
}
&lt;/critique&gt;
&lt;/example-2&gt;
&lt;example-3&gt;
&lt;nlq&gt;count window-hash where window-hash exists per hour&lt;/nlq&gt;
&lt;query&gt;
{
"breakdowns": ["window-hash"],
"calculations": [{"op": "COUNT"}],
"filters": [{"column": "window-hash", "op": "exists"}],
"time_range": 3600
}
&lt;/query&gt;
&lt;critique&gt;
{
"critique": "While the query correctly counts window-hash occurrences, the time_range
of 3600 seconds (1 hour) is insufficient for per-hour analysis. When we say 'per hour',
we need a time_range of at least 36000 seconds to show meaningful hourly patterns.",
"outcome": "bad"
}
&lt;/critique&gt;
&lt;/example-3&gt;
&lt;/examples&gt;
For the following query, first write a detailed critique explaining your reasoning,
then provide a pass/fail judgment in the same format as above.
&lt;nlq&gt;{{user_input}}&lt;/nlq&gt;
&lt;query&gt;
{{generated_query}}
&lt;/query&gt;
&lt;critique&gt;</code></pre>
<p>Notice how each example includes:</p>
<ol type="1">
<li>The natural language query (NLQ) in <code>&lt;nlq&gt;</code> tags</li>
<li>The generated query in <code>&lt;query&gt;</code> tags</li>
<li>The critique and outcome in <code>&lt;critique&gt;</code> tags</li>
</ol>
<p>In the prompt above, the example critiques are fixed. An advanced approach is to include examples dynamically based upon the item you are judging. You can learn more in <a href="https://blog.langchain.dev/dosu-langsmith-no-prompt-eng/">this post about Continual In-Context Learning</a>.</p>
</section>
<section id="keep-iterating-on-the-prompt-until-convergence-with-domain-expert" class=& |
github-actions
bot
added
the
Auto: Route Test Complete
TonyRL
reviewed
Nov 4, 2024
fix(route): fix pr issue
|
Successfully generated as following: http://localhost:1200/hamel/blog - Success ✔️<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
<channel>
<title>Hamel's Blog</title>
<link>https://hamel.dev</link>
<atom:link href="http://localhost:1200/hamel/blog" rel="self" type="application/rss+xml"></atom:link>
<description>Hamel's Blog - Powered by RSSHub</description>
<generator>RSSHub</generator>
<webMaster>contact@rsshub.app (RSSHub)</webMaster>
<language>en</language>
<lastBuildDate>Tue, 05 Nov 2024 01:21:46 GMT</lastBuildDate>
<ttl>5</ttl>
<item>
<title>Creating a LLM-as-a-Judge That Drives Business Results</title>
<description><header id="title-block-header" class="quarto-title-block default">
<div class="quarto-title">
<h1 class="title">Creating a LLM-as-a-Judge That Drives Business Results</h1>
<div class="quarto-categories">
<div class="quarto-category">LLMs</div>
<div class="quarto-category">evals</div>
</div>
</div>
<div>
<div class="description">
A step-by-step guide with my learnings from 30+ AI implementations.
</div>
</div>
<div class="quarto-title-meta">
<div>
<div class="quarto-title-meta-heading">Author</div>
<div class="quarto-title-meta-contents">
<p>Hamel Husain </p>
</div>
</div>
<div>
<div class="quarto-title-meta-heading">Published</div>
<div class="quarto-title-meta-contents">
<p class="date">October 29, 2024</p>
</div>
</div>
</div>
</header>
<nav id="TOC-body" role="doc-toc">
<h2 id="toc-title">Table Of Contents</h2>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#the-problem-ai-teams-are-drowning-in-data" id="toc-the-problem-ai-teams-are-drowning-in-data">The Problem: AI Teams Are Drowning in Data</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-1-find-the-principal-domain-expert" id="toc-step-1-find-the-principal-domain-expert">Step 1: Find <em>The</em> Principal Domain Expert</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#next-steps" id="toc-next-steps">Next Steps</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-2-create-a-dataset" id="toc-step-2-create-a-dataset">Step 2: Create a Dataset</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#why-a-diverse-dataset-matters" id="toc-why-a-diverse-dataset-matters">Why a Diverse Dataset Matters</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#dimensions-for-structuring-your-dataset" id="toc-dimensions-for-structuring-your-dataset">Dimensions for Structuring Your Dataset</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#examples-of-features-scenarios-and-personas" id="toc-examples-of-features-scenarios-and-personas">Examples of Features, Scenarios, and Personas</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#this-taxonomy-is-not-universal" id="toc-this-taxonomy-is-not-universal">This taxonomy is not universal</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#generating-data" id="toc-generating-data">Generating Data</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#example-llm-prompts-for-generating-user-inputs" id="toc-example-llm-prompts-for-generating-user-inputs">Example LLM Prompts for Generating User Inputs</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#generating-synthetic-data" id="toc-generating-synthetic-data">Generating Synthetic Data</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#next-steps-1" id="toc-next-steps-1">Next Steps</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-3-direct-the-domain-expert-to-make-passfail-judgments-with-critiques" id="toc-step-3-direct-the-domain-expert-to-make-passfail-judgments-with-critiques">Step 3: Direct The Domain Expert to Make Pass/Fail Judgments with Critiques</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#why-are-simple-passfail-metrics-important" id="toc-why-are-simple-passfail-metrics-important">Why are simple pass/fail metrics important?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#the-role-of-critiques" id="toc-the-role-of-critiques">The Role of Critiques</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#examples-of-good-critiques" id="toc-examples-of-good-critiques">Examples of Good Critiques</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#dont-stray-from-binary-passfail-judgments-when-starting-out" id="toc-dont-stray-from-binary-passfail-judgments-when-starting-out">Don’t stray from binary pass/fail judgments when starting out</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#make-it-easy-for-the-domain-expert-to-review-data" id="toc-make-it-easy-for-the-domain-expert-to-review-data">Make it easy for the domain expert to review data</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-many-examples-do-you-need" id="toc-how-many-examples-do-you-need">How many examples do you need?</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-4-fix-errors" id="toc-step-4-fix-errors">Step 4: Fix Errors</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-5-build-your-llm-as-a-judge-iteratively" id="toc-step-5-build-your-llm-as-a-judge-iteratively">Step 5: Build Your LLM as A Judge, Iteratively</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#the-hidden-power-of-critiques" id="toc-the-hidden-power-of-critiques">The Hidden Power of Critiques</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#start-with-expert-examples" id="toc-start-with-expert-examples">Start with Expert Examples</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#keep-iterating-on-the-prompt-until-convergence-with-domain-expert" id="toc-keep-iterating-on-the-prompt-until-convergence-with-domain-expert">Keep Iterating on the Prompt Until Convergence With Domain Expert</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-to-optimize-the-llm-judge-prompt" id="toc-how-to-optimize-the-llm-judge-prompt">How to Optimize the LLM Judge Prompt?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#the-human-side-of-the-process" id="toc-the-human-side-of-the-process">The Human Side of the Process</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-often-should-you-evaluate" id="toc-how-often-should-you-evaluate">How Often Should You Evaluate?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#what-if-this-doesnt-work" id="toc-what-if-this-doesnt-work">What if this doesn’t work?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#mistakes-ive-noticed-in-llm-judge-prompts" id="toc-mistakes-ive-noticed-in-llm-judge-prompts">Mistakes I’ve noticed in LLM judge prompts</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-6-perform-error-analysis" id="toc-step-6-perform-error-analysis">Step 6: Perform Error Analysis</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#classify-traces" id="toc-classify-traces">Classify Traces</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#an-interactive-walkthrough-of-error-analysis" id="toc-an-interactive-walkthrough-of-error-analysis">An Interactive Walkthrough of Error Analysis</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#fix-your-errors-again" id="toc-fix-your-errors-again">Fix Your Errors, Again</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#doing-this-well-requires-data-literacy" id="toc-doing-this-well-requires-data-literacy">Doing this well requires data literacy</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#step-7-create-more-specialized-llm-judges-if-needed" id="toc-step-7-create-more-specialized-llm-judges-if-needed">Step 7: Create More Specialized LLM Judges (if needed)</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#recap-of-critique-shadowing" id="toc-recap-of-critique-shadowing">Recap of Critique Shadowing</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#its-not-the-judge-that-created-value-afterall" id="toc-its-not-the-judge-that-created-value-afterall">It’s Not The Judge That Created Value, Afterall</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#do-you-really-need-this" id="toc-do-you-really-need-this">Do You Really Need This?</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#faq" id="toc-faq">FAQ</a>
<ul>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#if-i-have-a-good-judge-llm-isnt-that-also-the-llm-id-also-want-to-use" id="toc-if-i-have-a-good-judge-llm-isnt-that-also-the-llm-id-also-want-to-use">If I have a good judge LLM, isn’t that also the LLM I’d also want to use?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#do-you-recommend-fine-tuning-judges" id="toc-do-you-recommend-fine-tuning-judges">Do you recommend fine-tuning judges?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#whats-wrong-with-off-the-shelf-llm-judges" id="toc-whats-wrong-with-off-the-shelf-llm-judges">What’s wrong with off-the-shelf LLM judges?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-do-you-evaluate-the-llm-judge" id="toc-how-do-you-evaluate-the-llm-judge">How Do you evaluate the LLM judge?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#what-model-do-you-use-for-the-llm-judge" id="toc-what-model-do-you-use-for-the-llm-judge">What model do you use for the LLM judge?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#what-about-guardrails" id="toc-what-about-guardrails">What about guardrails?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#im-using-llm-as-a-judge-and-getting-tremendous-value-but-i-didnt-follow-this-approach." id="toc-im-using-llm-as-a-judge-and-getting-tremendous-value-but-i-didnt-follow-this-approach.">I’m using LLM as a judge, and getting tremendous value but I didn’t follow this approach.</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-do-you-choose-between-traditional-ml-techniques-llm-as-a-judge-and-human-annotations" id="toc-how-do-you-choose-between-traditional-ml-techniques-llm-as-a-judge-and-human-annotations">How do you choose between traditional ML techniques, LLM-as-a-judge and human annotations?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#can-you-make-judges-from-small-models" id="toc-can-you-make-judges-from-small-models">Can you make judges from small models?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-do-you-ensure-consistency-when-updating-your-llm-model" id="toc-how-do-you-ensure-consistency-when-updating-your-llm-model">How do you ensure consistency when updating your LLM model?</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#how-do-you-phase-out-human-in-the-loop-to-scale-this" id="toc-how-do-you-phase-out-human-in-the-loop-to-scale-this">How do you phase out human in the loop to scale this?</a></li>
</ul></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#resources" id="toc-resources">Resources</a></li>
<li><a href="https://hamel.dev/blog/posts/llm-judge/index.html#stay-connected" id="toc-stay-connected">Stay Connected</a></li>
</ul>
</nav>
<p>Earlier this year, I wrote <a href="https://hamel.dev/blog/posts/evals/">Your AI product needs evals</a>. Many of you asked, “How do I get started with LLM-as-a-judge?” This guide shares what I’ve learned after helping over <a href="https://parlance-labs.com/">30 companies</a> set up their evaluation systems.</p>
<section id="the-problem-ai-teams-are-drowning-in-data" class="level2">
<h2 class="anchored" data-anchor-id="the-problem-ai-teams-are-drowning-in-data">The Problem: AI Teams Are Drowning in Data</h2>
<p>Ever spend weeks building an AI system, only to realize you have no idea if it’s actually working? You’re not alone. I’ve noticed teams repeat the same mistakes when using LLMs to evaluate AI outputs:</p>
<ol type="1">
<li><strong>Too Many Metrics</strong>: Creating numerous measurements that become unmanageable.</li>
<li><strong>Arbitrary Scoring Systems</strong>: Using uncalibrated scales (like 1-5) across multiple dimensions, where the difference between scores is unclear and subjective. What makes something a 3 versus a 4? Nobody knows, and different evaluators often interpret these scales differently.</li>
<li><strong>Ignoring Domain Experts</strong>: Not involving the people who understand the subject matter deeply.</li>
<li><strong>Unvalidated Metrics</strong>: Using measurements that don’t truly reflect what matters to the users or the business.</li>
</ol>
<p>The result? Teams end up buried under mountains of metrics or data they don’t trust and can’t use. Progress grinds to a halt. Everyone gets frustrated.</p>
<p>For example, it’s not uncommon for me to see dashboards that look like this:</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://hamel.dev/blog/posts/llm-judge/blog_header.png" class="img-fluid figure-img" referrerpolicy="no-referrer"></p>
<figcaption>An illustrative example of a bad eval dashboard</figcaption>
</figure>
</div>
<p>Tracking a bunch of scores on a 1-5 scale is often a sign of a bad eval process (I’ll discuss why later). In this post, I’ll show you how to avoid these pitfalls. The solution is to use a technique that I call <strong>“Critique Shadowing”</strong>. Here’s how to do it, step by step.</p>
</section>
<section id="step-1-find-the-principal-domain-expert" class="level2">
<h2 class="anchored" data-anchor-id="step-1-find-the-principal-domain-expert">Step 1: Find <em>The</em> Principal Domain Expert</h2>
<p>In most organizations there is usually one (maybe two) key individuals whose judgment is crucial for the success of your AI product. These are the people with deep domain expertise or represent your target users. Identifying and involving this <strong>Principal Domain Expert</strong> early in the process is critical.</p>
<p><strong>Why is finding the right domain expert so important?</strong></p>
<ul>
<li><p><strong>They Set the Standard</strong>: This person not only defines what is acceptable technically, but also helps you understand if you’re building something users actually want.</p></li>
<li><p><strong>Capture Unspoken Expectations</strong>: By involving them, you uncover their preferences and expectations, which they might not be able to fully articulate upfront. Through the evaluation process, you help them clarify what a “passable” AI interaction looks like.</p></li>
<li><p><strong>Consistency in Judgment</strong>: People in your organization may have different opinions about the AI’s performance. Focusing on the principal expert ensures that evaluations are consistent and aligned with the most critical standards.</p></li>
<li><p><strong>Sense of Ownership</strong>: Involving the expert gives them a stake in the AI’s development. They feel invested because they’ve had a hand in shaping it. In the end, they are more likely to approve of the AI.</p></li>
</ul>
<p><strong>Examples of Principal Domain Experts:</strong></p>
<ul>
<li>A <strong>psychologist</strong> for a mental health AI assistant.</li>
<li>A <strong>lawyer</strong> for an AI that analyzes legal documents.</li>
<li>A <strong>customer service director</strong> for a support chatbot.</li>
<li>A <strong>lead teacher or curriculum developer</strong> for an educational AI tool.</li>
</ul>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Exceptions
</div>
</div>
<div class="callout-body-container callout-body">
<p>In a smaller company, this might be the CEO or founder. If you are an independent developer, you should be the domain expert (but be honest with yourself about your expertise).</p>
<p>If you must rely on leadership, you should regularly validate their assumptions against real user feedback.</p>
</div>
</div>
<p>Many developers attempt to act as the domain expert themselves, or find a convenient proxy (ex: their superior). This is a recipe for disaster. People will have varying opinions about what is acceptable, and you can’t make everyone happy. What’s important is that your principal domain expert is satisfied.</p>
<p><strong>Remember:</strong> This doesn’t have to take a lot of the domain expert’s time. Later in this post, I’ll discuss how you can make the process efficient. Their involvement is absolutely critical to the AI’s success.</p>
<section id="next-steps" class="level3">
<h3 class="anchored" data-anchor-id="next-steps">Next Steps</h3>
<p>Once you’ve found your expert, we need to give them the right data to review. Let’s talk about how to do that next.</p>
</section>
</section>
<section id="step-2-create-a-dataset" class="level2">
<h2 class="anchored" data-anchor-id="step-2-create-a-dataset">Step 2: Create a Dataset</h2>
<p>With your principal domain expert on board, the next step is to build a dataset that captures problems that your AI will encounter. It’s important that the dataset is diverse and represents the types of interactions that your AI will have in production.</p>
<section id="why-a-diverse-dataset-matters" class="level3">
<h3 class="anchored" data-anchor-id="why-a-diverse-dataset-matters">Why a Diverse Dataset Matters</h3>
<ul>
<li><strong>Comprehensive Testing</strong>: Ensures your AI is evaluated across a wide range of situations.</li>
<li><strong>Realistic Interactions</strong>: Reflects actual user behavior for more relevant evaluations.</li>
<li><strong>Identifies Weaknesses</strong>: Helps uncover areas where the AI may struggle or produce errors.</li>
</ul>
</section>
<section id="dimensions-for-structuring-your-dataset" class="level3">
<h3 class="anchored" data-anchor-id="dimensions-for-structuring-your-dataset">Dimensions for Structuring Your Dataset</h3>
<p>You want to define dimensions that make sense for your use case. For example, here are ones that I often use for B2C applications:</p>
<ol type="1">
<li><strong>Features</strong>: Specific functionalities of your AI product.</li>
<li><strong>Scenarios</strong>: Situations or problems the AI may encounter and needs to handle.</li>
<li><strong>Personas</strong>: Representative user profiles with distinct characteristics and needs.</li>
</ol>
</section>
<section id="examples-of-features-scenarios-and-personas" class="level3">
<h3 class="anchored" data-anchor-id="examples-of-features-scenarios-and-personas">Examples of Features, Scenarios, and Personas</h3>
<section id="features" class="level4">
<h4 class="anchored" data-anchor-id="features">Features</h4>
<table class="caption-top table">
<colgroup>
<col style="width: 30%">
<col style="width: 69%">
</colgroup>
<thead>
<tr class="header">
<th><strong>Feature</strong></th>
<th><strong>Description</strong></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>Email Summarization</strong></td>
<td>Condensing lengthy emails into key points.</td>
</tr>
<tr class="even">
<td><strong>Meeting Scheduler</strong></td>
<td>Automating the scheduling of meetings across time zones.</td>
</tr>
<tr class="odd">
<td><strong>Order Tracking</strong></td>
<td>Providing shipment status and delivery updates.</td>
</tr>
<tr class="even">
<td><strong>Contact Search</strong></td>
<td>Finding and retrieving contact information from a database.</td>
</tr>
<tr class="odd">
<td><strong>Language Translation</strong></td>
<td>Translating text between languages.</td>
</tr>
<tr class="even">
<td><strong>Content Recommendation</strong></td>
<td>Suggesting articles or products based on user interests.</td>
</tr>
</tbody>
</table>
</section>
<section id="scenarios" class="level4">
<h4 class="anchored" data-anchor-id="scenarios">Scenarios</h4>
<p>Scenarios are situations the AI needs to handle, (not based on the outcome of the AI’s response).</p>
<table class="caption-top table">
<colgroup>
<col style="width: 31%">
<col style="width: 68%">
</colgroup>
<thead>
<tr class="header">
<th><strong>Scenario</strong></th>
<th><strong>Description</strong></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>Multiple Matches Found</strong></td>
<td>User’s request yields multiple results that need narrowing down. For example: User asks “Where’s my order?” but has three active orders (#123, #124, #125). AI must help identify which specific order they’re asking about.</td>
</tr>
<tr class="even">
<td><strong>No Matches Found</strong></td>
<td>User’s request yields no results, requiring alternatives or corrections. For example: User searches for order #ABC-123 which doesn’t exist. AI should explain valid order formats and suggest checking their confirmation email.</td>
</tr>
<tr class="odd">
<td><strong>Ambiguous Request</strong></td>
<td>User input lacks necessary specificity. For example: User says “I need to change my delivery” without specifying which order or what aspect of delivery (date, address, etc.) they want to change.</td>
</tr>
<tr class="even">
<td><strong>Invalid Data Provided</strong></td>
<td>User provides incorrect data type or format. For example: User tries to track a return using a regular order number instead of a return authorization (RMA) number.</td>
</tr>
<tr class="odd">
<td><strong>System Errors</strong></td>
<td>Technical issues prevent normal operation. For example: While looking up an order, the inventory database is temporarily unavailable. AI needs to explain the situation and provide alternatives.</td>
</tr>
<tr class="even">
<td><strong>Incomplete Information</strong></td>
<td>User omits required details. For example: User wants to initiate a return but hasn’t provided the order number or reason. AI needs to collect this information step by step.</td>
</tr>
<tr class="odd">
<td><strong>Unsupported Feature</strong></td>
<td>User requests functionality that doesn’t exist. For example: User asks to change payment method after order has shipped. AI must explain why this isn’t possible and suggest alternatives.</td>
</tr>
</tbody>
</table>
</section>
<section id="personas" class="level4">
<h4 class="anchored" data-anchor-id="personas">Personas</h4>
<table class="caption-top table">
<colgroup>
<col style="width: 27%">
<col style="width: 72%">
</colgroup>
<thead>
<tr class="header">
<th><strong>Persona</strong></th>
<th><strong>Description</strong></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>New User</strong></td>
<td>Unfamiliar with the system; requires guidance.</td>
</tr>
<tr class="even">
<td><strong>Expert User</strong></td>
<td>Experienced; expects efficiency and advanced features.</td>
</tr>
<tr class="odd">
<td><strong>Non-Native Speaker</strong></td>
<td>May have language barriers; uses non-standard expressions.</td>
</tr>
<tr class="even">
<td><strong>Busy Professional</strong></td>
<td>Values quick, concise responses; often multitasking.</td>
</tr>
<tr class="odd">
<td><strong>Technophobe</strong></td>
<td>Uncomfortable with technology; needs simple instructions.</td>
</tr>
<tr class="even">
<td><strong>Elderly User</strong></td>
<td>May not be tech-savvy; requires patience and clear guidance.</td>
</tr>
</tbody>
</table>
</section>
</section>
<section id="this-taxonomy-is-not-universal" class="level3">
<h3 class="anchored" data-anchor-id="this-taxonomy-is-not-universal">This taxonomy is not universal</h3>
<p>This taxonomy (features, scenarios, personas) is not universal. For example, it may not make sense to even have personas if users aren’t directly engaging with your AI. The idea is you should outline dimensions that make sense for your use case and generate data that covers them. You’ll likely refine these after the first round of evaluations.</p>
</section>
<section id="generating-data" class="level3">
<h3 class="anchored" data-anchor-id="generating-data">Generating Data</h3>
<p>To build your dataset, you can:</p>
<ul>
<li><strong>Use Existing Data</strong>: Sample real user interactions or behaviors from your AI system.</li>
<li><strong>Generate Synthetic Data</strong>: Use LLMs to create realistic user inputs covering various features, scenarios, and personas.</li>
</ul>
<p>Often, you’ll do a combination of both to ensure comprehensive coverage. Synthetic data is not as good as real data, but it’s a good starting point. Also, we are only using LLMs to generate the user inputs, not the LLM responses or internal system behavior.</p>
<p>Regardless of whether you use existing data or synthetic data, you want good coverage across the dimensions you’ve defined.</p>
<p><strong>Incorporating System Information</strong></p>
<p>When making test data, use your APIs and databases where appropriate. This will create realistic data and trigger the right scenarios. Sometimes you’ll need to write simple programs to get this information. That’s what the “Assumptions” column is referring to in the examples below.</p>
</section>
<section id="example-llm-prompts-for-generating-user-inputs" class="level3">
<h3 class="anchored" data-anchor-id="example-llm-prompts-for-generating-user-inputs">Example LLM Prompts for Generating User Inputs</h3>
<p>Here are some example prompts that illustrate how to use an LLM to generate synthetic <strong>user inputs</strong> for different combinations of features, scenarios, and personas:</p>
<table class="caption-top table">
<colgroup>
<col style="width: 3%">
<col style="width: 11%">
<col style="width: 11%">
<col style="width: 10%">
<col style="width: 35%">
<col style="width: 26%">
</colgroup>
<thead>
<tr class="header">
<th><strong>ID</strong></th>
<th><strong>Feature</strong></th>
<th><strong>Scenario</strong></th>
<th><strong>Persona</strong></th>
<th><strong>LLM Prompt to Generate User Input</strong></th>
<th>Assumptions (not directly in the prompt)</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>1</td>
<td><strong>Order Tracking</strong></td>
<td>Invalid Data Provided</td>
<td>Frustrated Customer</td>
<td>“Generate a user input from someone who is clearly irritated and impatient, using short, terse language to demand information about their order status for order number <strong>#1234567890</strong>. Include hints of previous negative experiences.”</td>
<td>Order number <strong>#1234567890</strong> does <strong>not</strong> exist in the system.</td>
</tr>
<tr class="even">
<td>2</td>
<td><strong>Contact Search</strong></td>
<td>Multiple Matches Found</td>
<td>New User</td>
<td>“Create a user input from someone who seems unfamiliar with the system, using hesitant language and asking for help to find contact information for a person named ‘Alex’. The user should appear unsure about what information is needed.”</td>
<td>Multiple contacts named ‘Alex’ exist in the system.</td>
</tr>
<tr class="odd">
<td>3</td>
<td><strong>Meeting Scheduler</strong></td>
<td>Ambiguous Request</td>
<td>Busy Professional</td>
<td>“Simulate a user input from someone who is clearly in a hurry, using abbreviated language and minimal details to request scheduling a meeting. The message should feel rushed and lack specific information.”</td>
<td>N/A</td>
</tr>
<tr class="even">
<td>4</td>
<td><strong>Content Recommendation</strong></td>
<td>No Matches Found</td>
<td>Expert User</td>
<td>“Produce a user input from someone who demonstrates in-depth knowledge of their industry, using specific terminology to request articles on sustainable supply chain management. Use the information in this article involving sustainable supply chain management to formulate a plausible query: {{article}}”</td>
<td>No articles on ‘Emerging trends in sustainable supply chain management’ exist in the system.</td>
</tr>
</tbody>
</table>
</section>
<section id="generating-synthetic-data" class="level3">
<h3 class="anchored" data-anchor-id="generating-synthetic-data">Generating Synthetic Data</h3>
<p>When generating synthetic data, you only need to create the user inputs. You then feed these inputs into your AI system to generate the AI’s responses. It’s important that you log everything so you can evaluate your AI. To recap, here’s the process:</p>
<ol type="1">
<li><strong>Generate User Inputs</strong>: Use the LLM prompts to create realistic user inputs.</li>
<li><strong>Feed Inputs into Your AI System</strong>: Input the user interactions into your AI as it currently exists.</li>
<li><strong>Capture AI Responses</strong>: Record the AI’s responses to form complete interactions.</li>
<li><strong>Organize the Interactions</strong>: Create a table to store the user inputs, AI responses, and relevant metadata.</li>
</ol>
<section id="how-much-data-should-you-generate" class="level4">
<h4 class="anchored" data-anchor-id="how-much-data-should-you-generate">How much data should you generate?</h4>
<p>There is no right answer here. At a minimum, you want to generate enough data so that you have examples for each combination of dimensions (in this toy example: features, scenarios, and personas). However, you also want to keep generating more data until you feel like you have stopped seeing new failure modes. The amount of data I generate varies significantly depending on the use case.</p>
</section>
<section id="does-synthetic-data-actually-work" class="level4">
<h4 class="anchored" data-anchor-id="does-synthetic-data-actually-work">Does synthetic data actually work?</h4>
<p>You might be skeptical of using synthetic data. After all, it’s not real data, so how can it be a good proxy? In my experience, it works surprisingly well. Some of my favorite AI products, like <a href="https://hex.tech/">Hex</a> use synthetic data to power their evals:</p>
<blockquote class="blockquote">
<p>“LLMs are surprisingly good at generating excellent - and diverse - examples of user prompts. This can be relevant for powering application features, and sneakily, for building Evals. If this sounds a bit like the Large Language Snake is eating its tail, I was just as surprised as you! All I can say is: it works, ship it.” <em><a href="https://www.linkedin.com/in/bryan-bischof/">Bryan Bischof</a>, Head of AI Engineering at Hex</em></p>
</blockquote>
</section>
</section>
<section id="next-steps-1" class="level3">
<h3 class="anchored" data-anchor-id="next-steps-1">Next Steps</h3>
<p>With your dataset ready, now comes the most important part: getting your principal domain expert to evaluate the interactions.</p>
</section>
</section>
<section id="step-3-direct-the-domain-expert-to-make-passfail-judgments-with-critiques" class="level2">
<h2 class="anchored" data-anchor-id="step-3-direct-the-domain-expert-to-make-passfail-judgments-with-critiques">Step 3: Direct The Domain Expert to Make Pass/Fail Judgments with Critiques</h2>
<p>The domain expert’s job is to focus on one thing: <strong>“Did the AI achieve the desired outcome?”</strong> No complex scoring scales or multiple metrics. Just a clear <strong>pass or fail</strong> decision. In addition to the pass/fail decision, the domain expert should write a critique that explains their reasoning.</p>
<section id="why-are-simple-passfail-metrics-important" class="level3">
<h3 class="anchored" data-anchor-id="why-are-simple-passfail-metrics-important">Why are simple pass/fail metrics important?</h3>
<ul>
<li><p><strong>Clarity and Focus</strong>: A binary decision forces everyone to consider what truly matters. It simplifies the evaluation to a single, crucial question.</p></li>
<li><p><strong>Actionable Insights</strong>: Pass/fail judgments are easy to interpret and act upon. They help you quickly identify whether the AI meets the user’s needs.</p></li>
<li><p><strong>Forces Articulation of Expectations</strong>: When domain experts must decide if an interaction passes or fails, they are compelled to articulate their expectations clearly. This process uncovers nuances and unspoken assumptions about how the AI should behave.</p></li>
<li><p><strong>Efficient Use of Resources</strong>: Keeps the evaluation process manageable, especially when starting out. You avoid getting bogged down in detailed metrics that might not be meaningful yet.</p></li>
</ul>
</section>
<section id="the-role-of-critiques" class="level3">
<h3 class="anchored" data-anchor-id="the-role-of-critiques">The Role of Critiques</h3>
<p>Alongside a binary pass/fail judgment, it’s important to write a detailed critique of the LLM-generated output. These critiques:</p>
<ul>
<li><p><strong>Captures Nuances</strong>: The critique allows you to note if something was mostly correct but had areas for improvement.</p></li>
<li><p><strong>Guide Improvement</strong>: Detailed feedback provides specific insights into how the AI can be enhanced.</p></li>
<li><p><strong>Balance Simplicity with Depth</strong>: While the pass/fail offers a clear verdict, the critique offers the depth needed to understand the reasoning behind the judgment.</p></li>
</ul>
<p><strong>Why Write Critiques?:</strong></p>
<p>In practice, domain experts may not have fully internalized all the judgment criteria. By forcing them to make a pass/fail decision and explain their reasoning, they clarify their expectations and provide valuable guidance for refining the AI.</p>
<p><em>“But my problem is complex!”</em> Trust me—starting simple forces you to focus on what truly matters. You can introduce more complexity later if needed.</p>
</section>
<section id="examples-of-good-critiques" class="level3">
<h3 class="anchored" data-anchor-id="examples-of-good-critiques">Examples of Good Critiques</h3>
<p>To illustrate how simple pass/fail judgments combined with detailed critiques work in practice, here’s a table showcasing examples of user interactions with an AI assistant. The table includes both <strong>passes</strong> and <strong>fails</strong>, with critiques explaining why the AI received that judgment. In cases where the AI passed despite critical issues, the critique highlights these aspects and justifies why it still passed overall. For failed interactions, the critique explains the critical elements that led to the failure.</p>
<table class="caption-top table">
<colgroup>
<col style="width: 51%">
<col style="width: 24%">
<col style="width: 24%">
</colgroup>
<thead>
<tr class="header">
<th><strong>User Interaction with AI</strong></th>
<th><strong>Judgment</strong></th>
<th><strong>Critique</strong></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>User:</strong> “I need to cancel my flight booking for next week.”<br><strong>AI:</strong> “Your flight booking for next week has been canceled. A confirmation email has been sent to you.”</td>
<td><strong>Pass</strong></td>
<td>The AI successfully executed the user’s request by canceling the flight and sending a confirmation, achieving the primary goal. <strong>However</strong>, it should have asked for confirmation before proceeding to prevent accidental cancellations. Despite this critical oversight, the AI fulfilled the user’s request, so it <strong>passes</strong>, but adding a confirmation step would improve user safety.</td>
</tr>
<tr class="even">
<td><strong>User:</strong> “What’s the capital of France?”<br><strong>AI:</strong> “The capital of France is Paris.”</td>
<td><strong>Pass</strong></td>
<td>The AI correctly answered the user’s question, providing accurate information promptly. <strong>Although</strong> the response is sufficient, the AI could enhance the interaction by offering additional interesting facts or asking if the user has more questions. Despite this opportunity for added engagement, the AI met the user’s primary need, so it <strong>passes</strong>.</td>
</tr>
<tr class="odd">
<td><strong>User:</strong> “I can’t log into my account; it says my password is incorrect.”<br><strong>AI:</strong> “Please reset your password by clicking ‘Forgot Password’.”</td>
<td><strong>Fail</strong></td>
<td>The AI provided a generic solution without acknowledging the user’s frustration or checking for other issues like account lockout. It failed to offer empathetic support or additional assistance. Due to the lack of personalized help and poor user experience, it <strong>fails</strong>.</td>
</tr>
<tr class="even">
<td><strong>User:</strong> “Book a table for two at an Italian restaurant tonight at 7 PM.”<br><strong>AI:</strong> “Table for two at Luigi’s Italian Restaurant booked for tonight at 7 PM. Confirmation number is 12345.”</td>
<td><strong>Pass</strong></td>
<td>The AI efficiently completed the booking as requested, providing all necessary details and a confirmation number. <strong>However</strong>, it could have asked if the user had seating preferences or dietary restrictions. The AI accomplished the primary task successfully, so it <strong>passes</strong>, with minor opportunities for enhanced service.</td>
</tr>
<tr class="odd">
<td><strong>User:</strong> “I need help with my billing statement.”<br><strong>AI:</strong> “Please provide your account number and full name.”</td>
<td><strong>Fail</strong></td>
<td>The AI requested sensitive information without verifying the user’s identity through secure means, potentially compromising security. It should guide the user through a secure authentication process first. Due to this critical oversight in user data protection, it <strong>fails</strong>.</td>
</tr>
</tbody>
</table>
<p>These examples demonstrate how the AI can receive both <strong>“Pass”</strong> and <strong>“Fail”</strong> judgments. In the critiques:</p>
<ul>
<li><p>For <strong>passes</strong>, we explain why the AI succeeded in meeting the user’s primary need, even if there were critical aspects that could be improved. We highlight these areas for enhancement while justifying the overall passing judgment.</p></li>
<li><p>For <strong>fails</strong>, we identify the critical elements that led to the failure, explaining why the AI did not meet the user’s main objective or compromised important factors like user experience or security.</p></li>
</ul>
<p>Most importantly, <strong>the critique should be detailed enough so that you can use it in a few-shot prompt for a LLM judge</strong>. In other words, it should be detailed enough that a new employee could understand it. Being too terse is a common mistake.</p>
<p>Note that the example user interactions with the AI are simplified for brevity - but you might need to give the domain expert more context to make a judgement. More on that later.</p>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Note
</div>
</div>
<div class="callout-body-container callout-body">
<p>At this point, you don’t need to perform a root cause analysis into the technical reasons behind why the AI failed. Many times, it’s useful to get a sense of overall behavior before diving into the weeds.</p>
</div>
</div>
</section>
<section id="dont-stray-from-binary-passfail-judgments-when-starting-out" class="level3">
<h3 class="anchored" data-anchor-id="dont-stray-from-binary-passfail-judgments-when-starting-out">Don’t stray from binary pass/fail judgments when starting out</h3>
<p>A common mistake is straying from binary pass/fail judgments. Let’s revisit the dashboard from earlier:</p>
<p><img src="https://hamel.dev/blog/posts/llm-judge/dashboard.png" class="img-fluid" referrerpolicy="no-referrer"></p>
<p>If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you’re doing it wrong. Let’s unpack why.</p>
<ol type="1">
<li><strong>It’s not actionable</strong>: People don’t know what to do with a 3 or 4. It’s not immediately obvious how this number is better than a 2. You need to be able to say “this interaction passed because…” and “this interaction failed because…”.</li>
<li>More often than not, <strong>these metrics do not matter</strong>. Every time I’ve analyzed data on domain expert judgments, they tend not to correlate with these kind of metrics. By having a domain expert make a binary judgment, you can figure out what truly matters.</li>
</ol>
<p>This is why I hate off the shelf metrics that come with many evaluation frameworks. They tend to lead people astray.</p>
<p><strong>Common Objections to Pass/Fail Judgments:</strong></p>
<ul>
<li>“The business said that these 8 dimensions are important, so we need to evaluate all of them.”</li>
<li>“We need to be able to say why an interaction passed or failed.”</li>
</ul>
<p>I can guarantee you that if someone says you need to measure 8 things on a 1-5 scale, they don’t know what they are looking for. They are just guessing. You have to let the domain expert drive and make a pass/fail judgment with critiques so you can figure out what truly matters. Stand your ground here.</p>
</section>
<section id="make-it-easy-for-the-domain-expert-to-review-data" class="level3">
<h3 class="anchored" data-anchor-id="make-it-easy-for-the-domain-expert-to-review-data">Make it easy for the domain expert to review data</h3>
<p>Finally, you need to remove all friction from reviewing data. I’ve written about this <a href="https://hamel.dev/notes/llm/finetuning/data_cleaning.html">here</a>. Sometimes, you can just use a spreadsheet. It’s a judgement call in terms of what is easiest for the domain expert. I found that I often have to provide additional context to help the domain expert understand the user interaction, such as:</p>
<ul>
<li>Metadata about the user, such as their location, subscription tier, etc.</li>
<li>Additional context about the system, such as the current time, inventory levels, etc.</li>
<li>Resources so you can check if the AI’s response is correct (ex: ability to search a database, etc.)</li>
</ul>
<p>All of this data needs to be presented on a single screen so the domain expert can review it without jumping through hoops. That’s why I recommend building <a href="https://hamel.dev/notes/llm/finetuning/data_cleaning.html">a simple web app</a> to review data.</p>
</section>
<section id="how-many-examples-do-you-need" class="level3">
<h3 class="anchored" data-anchor-id="how-many-examples-do-you-need">How many examples do you need?</h3>
<p>The number of examples you need depends on the complexity of the task. My heuristic is that I start with around 30 examples and keep going until I do not see any new failure modes. From there, I keep going until I’m not learning anything new.</p>
<p>Next, we’ll look at how to use this data to build an LLM judge.</p>
</section>
</section>
<section id="step-4-fix-errors" class="level2">
<h2 class="anchored" data-anchor-id="step-4-fix-errors">Step 4: Fix Errors</h2>
<p>After looking at the data, it’s likely you will find errors in your AI system. Instead of plowing ahead and building an LLM judge, you want to fix any obvious errors. Remember, the whole point of the LLM as a judge is to help you find these errors, so it’s totally fine if you find them earlier!</p>
<p>If you have already developed <a href="https://hamel.dev/blog/posts/evals">Level 1 evals as outlined in my previous post</a>, you should not have any pervasive errors. However, these errors can sometimes slip through the cracks. If you find pervasive errors, fix them and go back to step 3. Keep iterating until you feel like you have stabilized your system.</p>
</section>
<section id="step-5-build-your-llm-as-a-judge-iteratively" class="level2">
<h2 class="anchored" data-anchor-id="step-5-build-your-llm-as-a-judge-iteratively">Step 5: Build Your LLM as A Judge, Iteratively</h2>
<section id="the-hidden-power-of-critiques" class="level3">
<h3 class="anchored" data-anchor-id="the-hidden-power-of-critiques">The Hidden Power of Critiques</h3>
<p>You cannot write a good judge prompt until you’ve seen the data. <a href="https://arxiv.org/abs/2404.12272">The paper from Shankar et al.,</a> “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences” summarizes this well:</p>
<blockquote class="blockquote">
<p>to grade outputs,people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria. We dub this phenomenon criteria drift, and it implies thatit is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.</p>
</blockquote>
</section>
<section id="start-with-expert-examples" class="level3">
<h3 class="anchored" data-anchor-id="start-with-expert-examples">Start with Expert Examples</h3>
<p>Let me share a real-world example of building an LLM judge you can apply to your own use case. When I was helping Honeycomb build their <a href="https://www.honeycomb.io/blog/introducing-query-assistant">Query Assistant feature</a>, we needed a way to evaluate if the AI was generating good queries. Here’s what our LLM judge prompt looked like, including few-shot examples of critiques from our domain expert, <a href="https://x.com/_cartermp">Phillip</a>:</p>
<pre class="text"><code>You are a Honeycomb query evaluator with advanced capabilities to judge if a query is good or not.
You understand the nuances of the Honeycomb query language, including what is likely to be
most useful from an analytics perspective.
Here is information about the Honeycomb query language:
{{query_language_info}}
Here are some guidelines for evaluating queries:
{{guidelines}}
Example evaluations:
&lt;examples&gt;
&lt;example-1&gt;
&lt;nlq&gt;show me traces where ip is 10.0.2.90&lt;/nlq&gt;
&lt;query&gt;
{
"breakdowns": ["trace.trace_id"],
"calculations": [{"op": "COUNT"}],
"filters": [{"column": "net.host.ip", "op": "=", "value": "10.0.2.90"}]
}
&lt;/query&gt;
&lt;critique&gt;
{
"critique": "The query correctly filters for traces with an IP address of 10.0.2.90
and counts the occurrences of those traces, grouped by trace.trace_id. The response
is good as it meets the requirement of showing traces from a specific IP address
without additional complexities.",
"outcome": "good"
}
&lt;/critique&gt;
&lt;/example-1&gt;
&lt;example-2&gt;
&lt;nlq&gt;show me slowest trace&lt;/nlq&gt;
&lt;query&gt;
{
"calculations": [{"column": "duration_ms", "op": "MAX"}],
"orders": [{"column": "duration_ms", "op": "MAX", "order": "descending"}],
"limit": 1
}
&lt;/query&gt;
&lt;critique&gt;
{
"critique": "While the query attempts to find the slowest trace using MAX(duration_ms)
and ordering correctly, it fails to group by trace.trace_id. Without this grouping,
the query only shows the MAX(duration_ms) measurement over time, not the actual
slowest trace.",
"outcome": "bad"
}
&lt;/critique&gt;
&lt;/example-2&gt;
&lt;example-3&gt;
&lt;nlq&gt;count window-hash where window-hash exists per hour&lt;/nlq&gt;
&lt;query&gt;
{
"breakdowns": ["window-hash"],
"calculations": [{"op": "COUNT"}],
"filters": [{"column": "window-hash", "op": "exists"}],
"time_range": 3600
}
&lt;/query&gt;
&lt;critique&gt;
{
"critique": "While the query correctly counts window-hash occurrences, the time_range
of 3600 seconds (1 hour) is insufficient for per-hour analysis. When we say 'per hour',
we need a time_range of at least 36000 seconds to show meaningful hourly patterns.",
"outcome": "bad"
}
&lt;/critique&gt;
&lt;/example-3&gt;
&lt;/examples&gt;
For the following query, first write a detailed critique explaining your reasoning,
then provide a pass/fail judgment in the same format as above.
&lt;nlq&gt;{{user_input}}&lt;/nlq&gt;
&lt;query&gt;
{{generated_query}}
&lt;/query&gt;
&lt;critique&gt;</code></pre>
<p>Notice how each example includes:</p>
<ol type="1">
<li>The natural language query (NLQ) in <code>&lt;nlq&gt;</code> tags</li>
<li>The generated query in <code>&lt;query&gt;</code> tags</li>
<li>The critique and outcome in <code>&lt;critique&gt;</code> tags</li>
</ol>
<p>In the prompt above, the example critiques are fixed. An advanced approach is to include examples dynamically based upon the item you are judging. You can learn more in <a href="https://blog.langchain.dev/dosu-langsmith-no-prompt-eng/">this post about Continual In-Context Learning</a>.</p>
</section>
<section id="keep-iterating-on-the-prompt-until-convergence-with-domain-expert" class=& |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Involved Issue / 该 PR 相关 Issue
Close #
Example for the Proposed Route(s) / 路由地址示例
New RSS Route Checklist / 新 RSS 路由检查表
PuppeteerNote / 说明
Hamel Husain's blog (hamel.dev) delves into AI, data science, and machine learning with a strong focus on practical applications and open-source contributions. Hamel, a prominent figure in AI and GitHub development, shares tutorials, insights on AI tooling, and real-world case studies, offering valuable guidance for developers and data scientists alike.
Hamel Husain 的博客（hamel.dev）探讨了 AI、数据科学和机器学习领域，特别关注实际应用和开源项目。Hamel 作为 AI 和 GitHub 开发的知名人物，分享了工具教程、AI 实践见解和真实案例，为开发者和数据科学家提供了宝贵的指导。