Added LLM evaluator to example projects (#1589)

D-K-P · web-flow · commit d52e3d7d571f · 2025-01-08T21:16:56.000Z
* Added repo links

* Added links to remix / nextjs webhooks examples

* Added a Realtime learn more snippet

* Added the batch llm evaluator docs

* Improved snippet

* Updated intro
diff --git a/docs/guides/example-projects/batch-llm-evaluator.mdx b/docs/guides/example-projects/batch-llm-evaluator.mdx
@@ -0,0 +1,49 @@
+---
+title: "Batch LLM Evaluator"
+sidebarTitle: "Batch LLM Evaluator"
+description: "This example project evaluates multiple LLM models using the Vercel AI SDK and streams updates to the frontend using Trigger.dev Realtime."
+---
+
+import RealtimeLearnMore from "/snippets/realtime-learn-more.mdx";
+
+## Overview
+
+This demo is a full stack example that uses the following:
+
+- A [Next.js](https://nextjs.org/) app with [Prisma](https://www.prisma.io/) for the database.
+- Trigger.dev [Realtime](https://trigger.dev/launchweek/0/realtime) to stream updates to the frontend.
+- Work with multiple LLM models using the Vercel [AI SDK](https://sdk.vercel.ai/docs/introduction). (OpenAI, Anthropic, XAI)
+- Distribute tasks across multiple tasks using the new [`batch.triggerByTaskAndWait`](https://trigger.dev/docs/triggering#batch-triggerbytaskandwait) method.
+
+## GitHub repo
+
+<Card
+  title="View the Batch LLM Evaluator repo"
+  icon="GitHub"
+  href="https://github.com/triggerdotdev/examples/tree/main/batch-llm-evaluator"
+>
+  Click here to view the full code for this project in our examples repository on GitHub. You can
+  fork it and use it as a starting point for your own project.
+</Card>
+
+## Video
+
+<video
+  controls
+  className="w-full aspect-video"
+  src="https://content.trigger.dev/batch-llm-evaluator.mp4"
+></video>
+
+## Relevant code
+
+- View the Trigger.dev task code in the [src/trigger/batch.ts](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/trigger/batch.ts) file.
+- The `evaluateModels` task uses the `batch.triggerByTaskAndWait` method to distribute the task to the different LLM models.
+- It then passes the results through to a `summarizeEvals` task that calculates some dummy "tags" for each LLM response.
+- We use a [useRealtimeRunsWithTag](https://trigger.dev/docs/frontend/react-hooks/realtime#userealtimerunswithtag) hook to subscribe to the different evaluation tasks runs in the [src/components/llm-evaluator.tsx](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/components/llm-evaluator.tsx) file.
+- We then pass the relevant run down into three different components for the different models:
+  - The `AnthropicEval` component: [src/components/evals/Anthropic.tsx](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/components/evals/Anthropic.tsx)
+  - The `XAIEval` component: [src/components/evals/XAI.tsx](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/components/evals/XAI.tsx)
+  - The `OpenAIEval` component: [src/components/evals/OpenAI.tsx](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/components/evals/OpenAI.tsx)
+- Each of these components then uses [useRealtimeRunWithStreams](https://trigger.dev/docs/frontend/react-hooks/realtime#userealtimerunwithstreams) to subscribe to the different LLM responses.
+
+<RealtimeLearnMore />
diff --git a/docs/guides/example-projects/realtime-fal-ai.mdx b/docs/guides/example-projects/realtime-fal-ai.mdx
@@ -4,6 +4,8 @@ sidebarTitle: "Realtime image gen with Fal.ai"
 description: "This example project generates an image from a prompt using Fal.ai and shows the progress of the task on the frontend using Trigger.dev Realtime."
 ---
 
+import RealtimeLearnMore from "/snippets/realtime-learn-more.mdx";
+
 ## Overview
 
 This full stack Next.js project showcases the following:
@@ -40,3 +42,5 @@ This video walks through the process of creating this task in a Next.js project.
     allowfullscreen
   />
 </div>
+
+<RealtimeLearnMore />
diff --git a/docs/guides/examples/fal-ai-realtime.mdx b/docs/guides/examples/fal-ai-realtime.mdx
@@ -4,6 +4,17 @@ sidebarTitle: "Fal.ai with Realtime"
 description: "This example task generates an image from a prompt using Fal.ai and shows the progress of the task on the frontend using Trigger.dev Realtime."
 ---
 
+## GitHub repo
+
+<Card
+  title="View the project on GitHub"
+  icon="GitHub"
+  href="https://github.com/triggerdotdev/examples/tree/main/realtime-fal-ai-image-generation"
+>
+  Click here to view the full code for this project in our examples repository on GitHub. You can
+  fork it and use it as a starting point for your own project.
+</Card>
+
 ## Walkthrough
 
 This video walks through the process of creating this task in a Next.js project.
diff --git a/docs/guides/frameworks/nextjs-webhooks.mdx b/docs/guides/frameworks/nextjs-webhooks.mdx
@@ -11,6 +11,17 @@ import VercelDocsCards from "/snippets/vercel-docs-cards.mdx";
 - [A Next.js project, set up with Trigger.dev](/guides/frameworks/nextjs)
 - [cURL](https://curl.se/) installed on your local machine. This will be used to send a POST request to your webhook handler.
 
+## GitHub repo
+
+<Card
+  title="View the project on GitHub"
+  icon="GitHub"
+  href="https://github.com/triggerdotdev/examples/tree/main/nextjs-webhooks/my-app"
+>
+  Click here to view the full code for this project in our examples repository on GitHub. You can
+  fork it and use it as a starting point for your own project.
+</Card>
+
 ## Adding the webhook handler
 
 The webhook handler in this guide will be an API route.
diff --git a/docs/guides/frameworks/remix-webhooks.mdx b/docs/guides/frameworks/remix-webhooks.mdx
@@ -9,6 +9,17 @@ description: "Learn how to trigger a task from a webhook in a Remix app."
 - [A Remix project, set up with Trigger.dev](/guides/frameworks/remix)
 - [cURL](https://curl.se/) installed on your local machine. This will be used to send a POST request to your webhook handler.
 
+## GitHub repo
+
+<Card
+  title="View the project on GitHub"
+  icon="GitHub"
+  href="https://github.com/triggerdotdev/examples/tree/main/remix-webhooks"
+>
+  Click here to view the full code for this project in our examples repository on GitHub. You can
+  fork it and use it as a starting point for your own project.
+</Card>
+
 ## Adding the webhook handler
 
 The webhook handler in this guide will be an API route. Create a new file `app/routes/api.webhook-handler.ts` or `app/routes/api.webhook-handler.js`.
diff --git a/docs/guides/frameworks/supabase-edge-functions-basic.mdx b/docs/guides/frameworks/supabase-edge-functions-basic.mdx
@@ -17,11 +17,6 @@ import NextjsTroubleshootingButtonSyntax from "/snippets/nextjs-button-syntax.md
 import WorkerFailedToStartWhenRunningDevCommand from "/snippets/worker-failed-to-start.mdx";
 import SupabaseDocsCards from "/snippets/supabase-docs-cards.mdx";
 
-<Info>
-  The project created in this guide can be found in this [GitHub
-  repo](https://github.com/triggerdotdev/example-projects/tree/main/supabase).
-</Info>
-
 ## Overview
 
 Supabase edge functions allow you to trigger tasks either when an event is sent from a third party (e.g. when a new Stripe payment is processed, when a new user signs up to a service, etc), or when there are any changes or updates to your Supabase database.
@@ -36,6 +31,17 @@ This guide shows you how to set up and deploy a simple Supabase edge function ex
 - [Create a Trigger.dev account](https://cloud.trigger.dev)
 - [Create a new Trigger.dev project](/guides/dashboard/creating-a-project)
 
+## GitHub repo
+
+<Card
+  title="View the project on GitHub"
+  icon="GitHub"
+  href="https://github.com/triggerdotdev/examples/tree/main/supabase-edge-functions"
+>
+  Click here to view the full code for this project in our examples repository on GitHub. You can
+  fork it and use it as a starting point for your own project.
+</Card>
+
 ## Initial setup
 
 <Steps>
diff --git a/docs/guides/frameworks/supabase-edge-functions-database-webhooks.mdx b/docs/guides/frameworks/supabase-edge-functions-database-webhooks.mdx
@@ -13,18 +13,14 @@ import NextjsTroubleshootingButtonSyntax from "/snippets/nextjs-button-syntax.md
 import WorkerFailedToStartWhenRunningDevCommand from "/snippets/worker-failed-to-start.mdx";
 import SupabaseDocsCards from "/snippets/supabase-docs-cards.mdx";
 
-<Info>
-  The project created in this guide can be found in this [GitHub
-  repo](https://github.com/triggerdotdev/example-projects/tree/main/supabase).
-</Info>
-
 ## Overview
 
 Supabase and Trigger.dev can be used together to create powerful workflows triggered by real-time changes in your database tables:
 
 - A Supabase Database Webhook triggers an Edge Function when a row including a video URL is inserted into a table
 - The Edge Function triggers a Trigger.dev task, passing the `video_url` column data from the new table row as the payload
 - The Trigger.dev task then:
+
   - Uses [FFmpeg](https://www.ffmpeg.org/) to extract the audio track from a video URL
   - Uses [Deepgram](https://deepgram.com) to transcribe the extracted audio
   - Updates the original table row using the `record.id` in Supabase with the new transcription using `update`
@@ -38,6 +34,17 @@ Supabase and Trigger.dev can be used together to create powerful workflows trigg
 - [Create a new Trigger.dev project](/guides/dashboard/creating-a-project)
 - [Create a new Deepgram account](https://deepgram.com/) and get your API key from the dashboard
 
+## GitHub repo
+
+<Card
+  title="View the project on GitHub"
+  icon="GitHub"
+  href="https://github.com/triggerdotdev/examples/tree/main/supabase-edge-functions"
+>
+  Click here to view the full code for this project in our examples repository on GitHub. You can
+  fork it and use it as a starting point for your own project.
+</Card>
+
 ## Initial setup
 
 <Steps>
diff --git a/docs/guides/introduction.mdx b/docs/guides/introduction.mdx
@@ -40,6 +40,7 @@ Example projects are full projects with example repos you can fork and use. Thes
 | Example project                                                              | Description                                                                                                    | Framework | GitHub Repo                                                                                           |
 | :--------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------- | :-------- | :---------------------------------------------------------------------------------------------------- |
 | [Realtime Fal.ai image generation](/guides/example-projects/realtime-fal-ai) | Generate an image from a prompt using Fal.ai and show the progress of the task on the frontend using Realtime. | Next.js   | [View the repo](https://github.com/triggerdotdev/examples/tree/main/realtime-fal-ai-image-generation) |
+| [Batch LLM Evaluator](/guides/example-projects/batch-llm-evaluator)          | Evaluate multiple LLM models and stream the results to the frontend.                                           | Next.js   | [View the repo](https://github.com/triggerdotdev/examples/tree/main/batch-llm-evaluator)              |
 
 ## Example tasks
 
diff --git a/docs/mint.json b/docs/mint.json
@@ -348,7 +348,10 @@
     },
     {
       "group": "Example projects",
-      "pages": ["guides/example-projects/realtime-fal-ai"]
+      "pages": [
+        "guides/example-projects/realtime-fal-ai",
+        "guides/example-projects/batch-llm-evaluator"
+      ]
     },
     {
       "group": "Example tasks",
diff --git a/docs/snippets/realtime-learn-more.mdx b/docs/snippets/realtime-learn-more.mdx
@@ -0,0 +1,7 @@
+## Learn more about Trigger.dev Realtime
+
+To learn more, take a look at the following resources:
+
+- [Realtime docs](https://trigger.dev/docs/realtime) - learn more about Realtime.
+- [Batch Trigger docs](https://trigger.dev/docs/triggering) - learn more about Batch Triggers.
+- [React hooks](https://trigger.dev/docs/frontend/react-hooks) - learn more about React hooks.

Original file line number	Diff line number	Diff line change
`@@ -348,7 +348,10 @@`
`348`	`348`	`},`
`349`	`349`	`{`
`350`	`350`	`"group": "Example projects",`
`351`		`- "pages": ["guides/example-projects/realtime-fal-ai"]`
	`351`	`+ "pages": [`
	`352`	`+ "guides/example-projects/realtime-fal-ai",`
	`353`	`+ "guides/example-projects/batch-llm-evaluator"`
	`354`	`+ ]`
`352`	`355`	`},`
`353`	`356`	`{`
`354`	`357`	`"group": "Example tasks",`