How to integrate with latest LangChain?

[cdennison]

The existing documentation has several issues:
https://docs.ragas.io/en/latest/howtos/integrations/langchain.html

1. The imports are wrong see ModuleNotFoundError: No module named 'ragas.langchain' #571
2. I have no idea how to use VectorstoreIndexCreator - all the LangChain examples I see use Chroma or FAISS
3. The example from the LangChain cookbooks is also broken https://github.com/langchain-ai/langsmith-cookbook/blob/main/testing-examples/ragas/ragas.ipynb

Your Question
Does Ragas plan to offer long terms support for LangChain (if not you can ignore the rest I'll have to abandon Ragas)? Can you please review the following code to see if it's correct? It's based on https://docs.ragas.io/en/latest/howtos/applications/compare_llms.html and https://docs.smith.langchain.com/old/cookbook/hub-examples/retrieval-qa-chain

Code Examples

ragas==0.1.14
langchain==0.2.14

from langchain_community.vectorstores import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.document_loaders import TextLoader
from langchain.chains import RetrievalQA

get a token: https://platform.openai.com/account/api-keys
from getpass import getpass
os.environ["OPENAI_API_KEY"] = getpass()

llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)

def build_query_engine(llm):
  loader = TextLoader("./notebooks_nyc_wikipedia_nyc_text.txt")
  data = loader.load()

  text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)
  all_splits = text_splitter.split_documents(data)

  vectorstore = Chroma.from_documents(documents=all_splits, embedding=OpenAIEmbeddings())

  return RetrievalQA.from_chain_type(
      llm,
      retriever=vectorstore.as_retriever(),
      return_source_documents=True
  )

#Ragas data
eval_questions = [
  "What is the population of New York City as of 2020?",
  "Which borough of New York City has the highest population?",
  "What is the economic significance of New York City?",
  "How did New York City get its name?",
  "What is the significance of the Statue of Liberty in New York City?",
]

eval_answers = [
  "8,804,190",
  "Brooklyn",
  "New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.",
  "New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.",
  "The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.",
]

examples = [
  {"query": q, "ground_truth": [eval_answers[i]]}
  for i, q in enumerate(eval_questions)
]

# run the queries as a batch for efficiency
qa_chain = build_query_engine(llm)
predictions = qa_chain.batch(examples)

#Ragas code starts here
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
  faithfulness,
  answer_relevancy,
  answer_correctness,
)

metrics = [
  faithfulness,
  answer_relevancy,
  answer_correctness,
]

#these seems cumbersome to have to convert the predictions/results differently for every type of integration
def generate_responses(predictions):
  answers = []
  contexts = []
  test_answers = []
  test_questions = []
  for r in predictions:
    test_questions.append(r['query'])
    answers.append(r['result'])
    test_answers.append(str(r['ground_truth'][0]))
    contexts.append([c.page_content for c in r['source_documents']])

  dataset_dict = {
    "question": test_questions,
    "answer": answers,
    "contexts": contexts,
    "ground_truth": test_answers
  }

  return Dataset.from_dict(dataset_dict)

res = evaluate(
    generate_responses(predictions),
    metrics=metrics,
)

print(res)

#result
#{'faithfulness': 0.8714, 'answer_relevancy': 0.9611, 'answer_correctness': 0.5776}

[dosubot]

Hello @cdennison! I am here to assist you.

Similar Issues and Solutions

1. Import Issues:

   • Issue: ModuleNotFoundError for ragas.langchain.
   • Solution: Ensure you have the correct imports and that the Ragas library is properly installed. Refer to the latest documentation and GitHub issues for updates ^[1].

2. Integration with LangChain:

   • Issue: Problems with integrating Ragas with LangChain, specifically with VectorstoreIndexCreator.
   • Solution: Follow the updated integration steps provided in the Ragas documentation. Ensure you are using the correct imports and methods ^[2].

Corrected Code Example

Here's a revised version of your code, ensuring correct imports and usage of VectorstoreIndexCreator:

import os
from getpass import getpass
from dotenv import load_dotenv
import nest_asyncio
from langchain_community.document_loaders import TextLoader
from langchain.indexes import VectorstoreIndexCreator
from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas.langchain.evalchain import RagasEvaluatorChain
from ragas.metrics import faithfulness, answer_relevancy, answer_correctness
from datasets import Dataset
from ragas import evaluate

# Load environment variables
load_dotenv()
os.environ["OPENAI_API_KEY"] = getpass()

# Initialize OpenAI API client
api_key = os.environ.get("OPENAI_API_KEY")
openai.api_key = api_key

nest_asyncio.apply()

# Initialize LLM
llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)

# Build query engine
def build_query_engine(llm):
    loader = TextLoader("./notebooks_nyc_wikipedia_nyc_text.txt")
    data = loader.load()

    text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)
    all_splits = text_splitter.split_documents(data)

    vectorstore = Chroma.from_documents(documents=all_splits, embedding=OpenAIEmbeddings())

    return RetrievalQA.from_chain_type(
        llm,
        retriever=vectorstore.as_retriever(),
        return_source_documents=True
    )

# Evaluation questions and answers
eval_questions = [
    "What is the population of New York City as of 2020?",
    "Which borough of New York City has the highest population?",
    "What is the economic significance of New York City?",
    "How did New York City get its name?",
    "What is the significance of the Statue of Liberty in New York City?",
]

eval_answers = [
    "8,804,190",
    "Brooklyn",
    "New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.",
    "New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.",
    "The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.",
]

examples = [
    {"query": q, "ground_truth": [eval_answers[i]]}
    for i, q in enumerate(eval_questions)
]

# Run the queries as a batch for efficiency
qa_chain = build_query_engine(llm)
predictions = qa_chain.batch(examples)

# Generate responses
def generate_responses(predictions):
    answers = []
    contexts = []
    test_answers = []
    test_questions = []
    for r in predictions:
        test_questions.append(r['query'])
        answers.append(r['result'])
        test_answers.append(str(r['ground_truth'][0]))
        contexts.append([c.page_content for c in r['source_documents']])

    dataset_dict = {
        "question": test_questions,
        "answer": answers,
        "contexts": contexts,
        "ground_truth": test_answers
    }

    return Dataset.from_dict(dataset_dict)

# Evaluate using Ragas
metrics = [faithfulness, answer_relevancy, answer_correctness]
res = evaluate(generate_responses(predictions), metrics=metrics)
print(res)

This code should address the issues with imports and ensure the correct usage of VectorstoreIndexCreator and other components. Make sure to follow the latest documentation and updates from Ragas and LangChain for any changes or additional guidance ^[2][1].

To continue talking to Dosu, mention @dosu.