You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
> 🚀 Dedicated solutions to evaluate, monitor and improve performance of LLM & RAG application in production including custom models for production quality monitoring.[Talk to founders](https://cal.com/shahul-ragas/30min)
35
+
[Ragas](https://www.ragas.io/) supercharges your LLM application evaluations with tools to objectively measure performance, synthesize test case scenarios, and gain insights by leveraging production data.
36
+
37
+
Evaluating and testing LLM applications is a challenging, time-consuming, and often boring process. Ragas aims provide a suite of tools that could supercharge your evaluation workflows and make it more efficient and fun using state-of-the-art research. We are also building an open ecosystem, that fosters sharing of ideas to make the evaluation process better and collaborates with other tools in the market to make it seamless you.
42
38
43
-
Ragas is a framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines. RAG denotes a class of LLM applications that use external data to augment the LLM’s context. There are existing tools and frameworks that help you build these pipelines but evaluating it and quantifying your pipeline performance can be hard. This is where Ragas (RAG Assessment) comes in.
39
+
## Key Features
44
40
45
-
Ragas provides you with the tools based on the latest research for evaluating LLM-generated text to give you insights about your RAG pipeline. Ragas can be integrated with your CI/CD to provide continuous checks to ensure performance.
41
+
-**Metrics**: Different LLM based and non LLM based metrics to objectively evaluate your LLM applications such as RAG, Agents, etc.
42
+
-**Test Data Generation**: Synthesize high-quality datasets covering wide variety of scenarios for comprehensive testing of your LLM applications.
43
+
-**Integrations**: Seamless integration with all major LLM applications frameworks like langchain and observability tools.
This is a small example program you can run to see ragas in action!
64
-
65
-
```python
66
-
67
-
from datasets import Dataset
68
-
import os
69
-
from ragas import evaluate
70
-
from ragas.metrics import faithfulness, answer_correctness
71
-
72
-
os.environ["OPENAI_API_KEY"] ="your-openai-key"
73
-
74
-
data_samples = {
75
-
'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
76
-
'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
77
-
'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'],
78
-
['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
79
-
'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
**Metrics can be classified into two categories based on the mechanism used underneath the hood**:
19
18
20
19
**LLM-based metrics**: These metrics use LLM underneath to do the evaluation. There might be one or more LLM calls that are performed to arrive at the score or result. These metrics can be somewhat non deterministic as the LLM might not always return the same result for the same input. On the other hand, these metrics has shown to be more accurate and closer to human evaluation.
0 commit comments