-
Notifications
You must be signed in to change notification settings - Fork 834
chore: add docs on configuring custom output rendering for annotation queues #1267
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -3,36 +3,25 @@ title: Custom output rendering | |||||
| sidebarTitle: Custom output rendering | ||||||
| --- | ||||||
|
|
||||||
| This page describes how to use custom output rendering to visualize outputs from [_experiments_](/langsmith/evaluation-concepts#experiment) using your own custom HTML pages. | ||||||
|
|
||||||
| You'll learn how to: | ||||||
|
|
||||||
| - **[Configure custom rendering](#configure-custom-output-rendering)** in the LangSmith UI or via the API. | ||||||
| - **[Build a custom renderer](#build-a-custom-renderer)** to display output data. | ||||||
| - **[Understand where custom rendering appears](#where-custom-rendering-appears)** in LangSmith. | ||||||
|
|
||||||
| ## Overview | ||||||
|
|
||||||
| By default, LangSmith displays outputs in a standard JSON format. With custom output rendering, you can replace this default view with your own HTML page that receives the output data via the postMessage API and renders it however you choose. | ||||||
|
|
||||||
| This is particularly useful for: | ||||||
| Custom output rendering allows you to visualize run outputs and dataset reference outputs using your own custom HTML pages. This is particularly useful for: | ||||||
|
|
||||||
| - **Domain-specific formatting**: Display medical records, legal documents, or other specialized data types in their native format. | ||||||
| - **Custom visualizations**: Create charts, graphs, or diagrams from numeric or structured output data. | ||||||
|
|
||||||
| Custom rendering is configured per-[_dataset_](/langsmith/evaluation-concepts#datasets). | ||||||
|
|
||||||
| ## Prerequisites | ||||||
| In this page you'll learn how to: | ||||||
|
|
||||||
| <Check> | ||||||
| - A LangSmith dataset | ||||||
| - A web page that can receive postMessage events (can be hosted anywhere or run locally) | ||||||
| - Basic knowledge of HTML/JavaScript and the postMessage API | ||||||
| </Check> | ||||||
| - **[Configure custom rendering](#configure-custom-output-rendering)** in the LangSmith UI. | ||||||
| - **[Build a custom renderer](#build-a-custom-renderer)** to display output data. | ||||||
| - **[Understand where custom rendering appears](#where-custom-rendering-appears)** in LangSmith. | ||||||
|
|
||||||
| ## Configure custom output rendering | ||||||
|
|
||||||
| ### In the LangSmith UI | ||||||
| Configure custom rendering at two levels: | ||||||
|
|
||||||
| - **For datasets**: Apply custom rendering to all runs associated with that dataset, wherever they appear—in experiments, run detail panes, or annotation queues. | ||||||
| - **For annotation queues**: Apply custom rendering to all runs within a specific annotation queue, regardless of which dataset they come from. This takes precedence over dataset-level configuration. | ||||||
|
|
||||||
| ### For datasets | ||||||
|
|
||||||
| To configure custom output rendering for a dataset: | ||||||
|
|
||||||
|
|
@@ -47,44 +36,22 @@ To configure custom output rendering for a dataset: | |||||
|
|
||||||
|  | ||||||
|
|
||||||
| ### Via the API | ||||||
| ### For annotation queues | ||||||
|
|
||||||
| You can also configure custom rendering programmatically by updating the dataset metadata: | ||||||
| To configure custom output rendering for an annotation queue: | ||||||
|
|
||||||
| <CodeGroup> | ||||||
| ```python Python | ||||||
| from langsmith import Client | ||||||
|  | ||||||
|
|
||||||
| client = Client() | ||||||
|
|
||||||
| # Update dataset metadata with iframe config | ||||||
| client.update_dataset( | ||||||
| dataset_id="your-dataset-id", | ||||||
| metadata={ | ||||||
| "iframe_config": { | ||||||
| "enabled": True, | ||||||
| "url": "https://your-domain.com/output-renderer.html" | ||||||
| } | ||||||
| } | ||||||
| ) | ||||||
| ``` | ||||||
|
|
||||||
| ```typescript TypeScript | ||||||
| import { Client } from "langsmith"; | ||||||
|
|
||||||
| const client = new Client(); | ||||||
| 1. Navigate to the **Annotation Queues** page. | ||||||
| 2. Click on an existing annotation queue or create a new one. | ||||||
| 3. In the annotation queue settings pane, scroll to the **Custom Output Rendering** section. | ||||||
| 4. Toggle **Enable custom output rendering**. | ||||||
| 5. Enter the webpage URL in the **URL** field. | ||||||
| 6. Click **Save** or **Create**. | ||||||
|
|
||||||
| // Update dataset metadata with iframe config | ||||||
| await client.updateDataset("your-dataset-id", { | ||||||
| metadata: { | ||||||
| iframe_config: { | ||||||
| enabled: true, | ||||||
| url: "https://your-domain.com/output-renderer.html" | ||||||
| } | ||||||
| } | ||||||
| }); | ||||||
| ``` | ||||||
| </CodeGroup> | ||||||
| <Info> | ||||||
| When custom rendering is configured at both levels, annotation queue configuration takes precedence over dataset configuration for runs viewed within that queue. | ||||||
| </Info> | ||||||
|
|
||||||
| ## Build a custom renderer | ||||||
|
|
||||||
|
|
@@ -163,6 +130,6 @@ When enabled, your custom rendering will replace the default output view in: | |||||
|
|
||||||
|  | ||||||
|
|
||||||
| - **Annotation queues**: When experiments are added to annotation queues for review: | ||||||
| - **Annotation queues**: When reviewing runs in annotation queues: | ||||||
|
||||||
| - **Annotation queues**: When reviewing runs in annotation queues: | |
| - **Annotation queues**: When you review runs in annotation queues: |
Binary file added
BIN
+380 KB
src/langsmith/images/annotation-queue-custom-output-rendering-settings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nitpick] The precedence note appears twice (here and in the Info box at line 52-54). Consider removing the precedence mention from line 22 to avoid redundancy, keeping only the more prominent Info callout.