Braintrust SDK

Braintrust is a platform for evaluating and shipping AI products. To learn more about Braintrust or sign up for free, visit our website or check out the docs.

This repository contains the JavaScript SDK for Braintrust. The SDK includes utilities to:

• Log experiments and datasets to Braintrust
• Run evaluations (via the Eval framework)

Quickstart

First, install the Braintrust SDK:

npm install braintrust autoevals

or

yarn add braintrust autoevals

Then, create a file named tutorial.eval.ts with the following code:

import { Eval } from "braintrust";
import { LevenshteinScorer } from "autoevals";

Eval("Say Hi Bot", {
  data: () => {
    return [
      {
        input: "Foo",
        expected: "Hi Foo",
      },
      {
        input: "Bar",
        expected: "Hello Bar",
      },
    ]; // Replace with your eval dataset
  },
  task: (input) => {
    return "Hi " + input; // Replace with your LLM call
  },
  scores: [LevenshteinScorer],
});

Then, run the following command:

BRAINTRUST_API_KEY=<YOUR_API_KEY> \
    npx braintrust eval tutorial.eval.ts

Integrations

Braintrust provides integrations with several popular AI development tools and platforms:

• LangChain.js: A callback handler to automatically log LangChain.js executions to Braintrust. Learn more
• Val Town: Examples and templates for using Braintrust with Val Town's serverless JavaScript/TypeScript environment. Learn more
• Vercel AI SDK: Integration with Vercel's AI SDK for building AI-powered applications. Learn more

Documentation

For more information, check out the docs:

• TypeScript