Skip to content

Commit 2a96e6f

Browse files
sahusiddharthsahusiddharth
authored
Docs/updating the LangSmith docs (#1828)
Fixes #1791
1 parent f6ebf99 commit 2a96e6f

File tree

7 files changed

+109
-79
lines changed

7 files changed

+109
-79
lines changed

‎docs/_static/langsmith_dashboard.png

619 KB
Loading

‎docs/howtos/integrations/_langsmith.md

-75
This file was deleted.

‎docs/howtos/integrations/langsmith.md

+104
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# LangSmith
2+
3+
[LangSmith](https://docs.smith.langchain.com/) is an advanced tool designed to enhance the development and deployment of applications utilizing large language models (LLMs). It provides a comprehensive framework for tracing, analyzing, and optimizing LLM workflows, making it easier for developers to manage complex interactions within their applications.
4+
5+
This tutorial explains how to log traces of Ragas evaluations using LangSmith. Since Ragas is built on LangChain, you only need to set up LangSmith, and it will handle logging the traces automatically.
6+
7+
## Setting Up LangSmith
8+
9+
To set up LangSmith, make sure you set the following environment variables (refer to the [LangSmith documentation](https://docs.smith.langchain.com/#quick-start) for more details):
10+
11+
```bash
12+
export LANGCHAIN_TRACING_V2=true
13+
export LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
14+
export LANGCHAIN_API_KEY=<your-api-key>
15+
export LANGCHAIN_PROJECT=<your-project> # Defaults to "default" if not set
16+
```
17+
18+
## Getting the Dataset
19+
20+
When creating evaluation dataset or evaluating instance, ensure the terminology matches the schema used in `SingleTurnSample` or `MultiTurnSample`.
21+
22+
23+
```python
24+
from ragas import EvaluationDataset
25+
26+
27+
dataset = [
28+
{
29+
"user_input": "Which CEO is widely recognized for democratizing AI education through platforms like Coursera?",
30+
"retrieved_contexts": [
31+
"Andrew Ng, CEO of Landing AI, is known for his pioneering work in deep learning and for democratizing AI education through Coursera."
32+
],
33+
"response": "Andrew Ng is widely recognized for democratizing AI education through platforms like Coursera.",
34+
"reference": "Andrew Ng, CEO of Landing AI, is known for democratizing AI education through Coursera.",
35+
},
36+
{
37+
"user_input": "Who is Sam Altman?",
38+
"retrieved_contexts": [
39+
"Sam Altman, CEO of OpenAI, has advanced AI research and advocates for safe, beneficial AI technologies."
40+
],
41+
"response": "Sam Altman is the CEO of OpenAI and advocates for safe, beneficial AI technologies.",
42+
"reference": "Sam Altman, CEO of OpenAI, has advanced AI research and advocates for safe AI.",
43+
},
44+
{
45+
"user_input": "Who is Demis Hassabis and how did he gain prominence?",
46+
"retrieved_contexts": [
47+
"Demis Hassabis, CEO of DeepMind, is known for developing systems like AlphaGo that master complex games."
48+
],
49+
"response": "Demis Hassabis is the CEO of DeepMind, known for developing systems like AlphaGo.",
50+
"reference": "Demis Hassabis, CEO of DeepMind, is known for developing AlphaGo.",
51+
},
52+
{
53+
"user_input": "Who is the CEO of Google and Alphabet Inc., praised for leading innovation across Google's product ecosystem?",
54+
"retrieved_contexts": [
55+
"Sundar Pichai, CEO of Google and Alphabet Inc., leads innovation across Google's product ecosystem."
56+
],
57+
"response": "Sundar Pichai is the CEO of Google and Alphabet Inc., praised for leading innovation across Google's product ecosystem.",
58+
"reference": "Sundar Pichai, CEO of Google and Alphabet Inc., leads innovation across Google's product ecosystem.",
59+
},
60+
{
61+
"user_input": "How did Arvind Krishna transform IBM?",
62+
"retrieved_contexts": [
63+
"Arvind Krishna, CEO of IBM, transformed the company by focusing on cloud computing and AI solutions."
64+
],
65+
"response": "Arvind Krishna transformed IBM by focusing on cloud computing and AI solutions.",
66+
"reference": "Arvind Krishna, CEO of IBM, transformed the company through cloud computing and AI.",
67+
},
68+
]
69+
70+
evaluation_dataset = EvaluationDataset.from_list(dataset)
71+
```
72+
73+
## Tracing ragas metrics
74+
75+
Run the Ragas evaluations on your dataset, and the traces will appear in your LangSmith dashboard under the specified project name or "default."
76+
77+
78+
```python
79+
from ragas import evaluate
80+
from ragas.llms import LangchainLLMWrapper
81+
from langchain_openai import ChatOpenAI
82+
from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness
83+
84+
llm = ChatOpenAI(model="gpt-4o-mini")
85+
evaluator_llm = LangchainLLMWrapper(llm)
86+
87+
result = evaluate(
88+
dataset=evaluation_dataset,
89+
metrics=[LLMContextRecall(), Faithfulness(), FactualCorrectness()],
90+
llm=evaluator_llm,
91+
)
92+
93+
result
94+
```
95+
96+
Output
97+
```
98+
Evaluating: 0%| | 0/15 [00:00<?, ?it/s]
99+
100+
{'context_recall': 1.0000, 'faithfulness': 0.9333, 'factual_correctness': 0.8520}
101+
```
102+
103+
## LangSmith Dashboard
104+
![jpeg](../../_static/langsmith_dashboard.png)

‎mkdocs.yml

+2-1
Original file line numberDiff line numberDiff line change
@@ -98,10 +98,11 @@ nav:
9898
- Debug LLM Based Metrics: howtos/applications/_metrics_llm_calls.md
9999
- Integrations:
100100
- howtos/integrations/index.md
101-
- LlamaIndex: howtos/integrations/_llamaindex.md
102101
- Arize: howtos/integrations/_arize.md
103102
- LangChain: howtos/integrations/langchain.md
104103
- LangGraph: howtos/integrations/_langgraph_agent_evaluation.md
104+
- LangSmith: howtos/integrations/langsmith.md
105+
- LlamaIndex: howtos/integrations/_llamaindex.md
105106
- Migrations:
106107
- From v0.1 to v0.2: howtos/migrations/migrate_from_v01_to_v02.md
107108
- 📖 References:

‎src/ragas/metrics/_answer_relevance.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
from __future__ import annotations
22

3+
import asyncio
34
import logging
45
import typing as t
56
from dataclasses import dataclass, field
@@ -16,7 +17,6 @@
1617
SingleTurnMetric,
1718
)
1819
from ragas.prompt import PydanticPrompt
19-
import asyncio
2020

2121
logger = logging.getLogger(__name__)
2222

‎src/ragas/sdk.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -91,4 +91,4 @@ def check_api_response(response: requests.Response) -> None:
9191

9292

9393
def build_evaluation_app_url(app_url: str, run_id: str) -> str:
94-
return f"{app_url}/dashboard/alignment/evaluation/{run_id}"
94+
return f"{app_url}/dashboard/alignment/evaluation/{run_id}"

‎src/ragas/testset/synthesizers/testset_schema.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
SingleTurnSample,
1717
)
1818
from ragas.exceptions import UploadException
19-
from ragas.sdk import upload_packet, get_app_url
19+
from ragas.sdk import get_app_url, upload_packet
2020

2121

2222
class TestsetSample(BaseSample):

0 commit comments

Comments
 (0)