Vasari is a local, open source tool for evaluating AI features (agents, chatbots, RAG). Review LLM traces, run error analysis (open and axial coding), and validate an LLM as a judge against human labels with a confusion matrix, TPR/TNR and Cohen's kappa. Inspired by Hamel Husain's AI evals.
machine-learning annotation chatbot confusion-matrix observability product-management human-in-the-loop ai-agents error-analysis rag data-labeling opentelemetry prompt-engineering evals llm-evaluation llm-as-a-judge ai-evals hamel-husain
-
Updated
Jul 12, 2026 - TypeScript