Testset Generation Failure: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

[Zolastic]

Description:

I am encountering multiple warnings and errors while attempting to generate a test set using the ragas library with LangChain, Pydantic v2, and OpenAI. The main issues stem from deprecated imports and namespace conflicts. Despite following potential fixes and utilizing nest_asyncio, the issue persists.

Dependencies:

Here are the relevant dependencies from my pyproject.toml:

[tool.poetry.dependencies]
python = ">=3.12,<3.13"
ragas = "^0.1.18"
langchain = "^0.3.0"
langchain-community = "^0.3.0"
langchain-openai = "^0.2.0"
bs4 = "^0.0.2"
beautifulsoup4 = "^4.12.3"
unstructured = "^0.15.12"
libmagic = "^1.0"
python-magic-bin = "^0.4.14"
nest-asyncio = "^1.6.0"

Steps to Reproduce:

1. Set up the environment using Python 3.12 and the dependencies listed above.
2. Use the following code to load documents from a webpage and generate a test set:

import os
from langchain_community.document_loaders import WebBaseLoader
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

import nest_asyncio
nest_asyncio.apply()

os.environ["OPENAI_API_KEY"] = "openai-api-key"

loader = WebBaseLoader("https://www.nba.com/stats/players")
documents = loader.load()

generator_llm = ChatOpenAI(model="gpt-4o-mini-2024-07-18")
critic_llm = ChatOpenAI(model="gpt-4o-mini-2024-07-18")
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

try:
    test_set = generator.generate_with_langchain_docs(documents, test_size=3, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}, raise_exceptions=False)
    data_frame = test_set.to_pandas()
    data_frame.to_csv('testset.csv', index=False)
except Exception as e:
    print(f"Error during test set generation: {e}")

Error Output:

USER_AGENT environment variable not set, consider setting it to identify your requests.
/.../pydantic/_fields.py:132: UserWarning: Field "model_name" in _VertexAIBase has conflict with protected namespace "model_".
/.../pydantic/_fields.py:132: UserWarning: Field "model_name" in _VertexAICommon has conflict with protected namespace "model_".
LangChainDeprecationWarning: As of langchain-core 0.3.0, LangChain uses pydantic v2 internally...
Error during test set generation: The runner thread which was running the jobs raised an exception. Read the traceback above to debug it.

What I Have Tried:

• Added nest_asyncio.apply() to manage async conflicts.
• Checked compatibility between LangChain and Pydantic, replacing deprecated imports where applicable.
• Attempted to use raise_exceptions=False as suggested and is_async=False, but the problem persists.

Environment:

• Python Version: 3.12
• Pydantic Version: 2.x
• LangChain Version: 0.3.0
• Ragas Version: 0.1.18

Expected Behavior:

The test set should be generated without warnings or namespace conflicts, and it should create the testset.csv file successfully.

Actual Behavior:

The process halts with a deprecation warning and a namespace conflict related to pydantic and model_. It raises an exception during test set generation.

Any guidance on how to resolve these namespace conflicts and LangChain compatibility issues would be greatly appreciated.

[dosubot]

Hey @Zolastic! I'm here to help you tackle any bugs, answer questions, and even become a contributor. Let's work together to resolve the issue you're facing.

I found a similar closed issue that might be relevant to your problem: running quickstart.ipynb fails. This issue was closed by a pull request: fix: llamaIndex issues on September 26, 2023 ^[1].

To resolve the namespace conflicts and LangChain compatibility issues when generating a test set using the ragas library with LangChain, Pydantic v2, and OpenAI, you need to ensure that the correct imports and class references are used. Specifically, you should use the appropriate wrappers and ensure that the correct classes from LangChain and OpenAI are referenced.

Here is the relevant code snippet that shows how to correctly set up the TestsetGenerator class to handle LangChain and OpenAI models:

from langchain_openai.chat_models import ChatOpenAI
from langchain_openai.embeddings import OpenAIEmbeddings
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings.base import LangchainEmbeddingsWrapper
from ragas.testset.docstore import InMemoryDocumentStore
from ragas.testset.extractor import KeyphraseExtractor
from ragas.llms.prompt import Prompt

@dataclass
class TestsetGenerator:
    generator_llm: BaseRagasLLM
    critic_llm: BaseRagasLLM
    embeddings: BaseRagasEmbeddings
    docstore: DocumentStore

    @classmethod
    def from_langchain(
        cls,
        generator_llm: LangchainLLM,
        critic_llm: LangchainLLM,
        embeddings: LangchainEmbeddings,
        docstore: t.Optional[DocumentStore] = None,
        run_config: RunConfig = RunConfig(),
        chunk_size: int = 1024,
    ) -> "TestsetGenerator":
        generator_llm_model = LangchainLLMWrapper(generator_llm)
        critic_llm_model = LangchainLLMWrapper(critic_llm)
        embeddings_model = LangchainEmbeddingsWrapper(embeddings)

        keyphrase_extractor = KeyphraseExtractor(llm=generator_llm_model)
        if docstore is None:
            from langchain.text_splitter import TokenTextSplitter

            splitter = TokenTextSplitter(chunk_size=chunk_size, chunk_overlap=0)
            docstore = InMemoryDocumentStore(
                splitter=splitter,
                embeddings=embeddings_model,
                extractor=keyphrase_extractor,
                run_config=run_config,
            )
        return cls(
            generator_llm=generator_llm_model,
            critic_llm=critic_llm_model,
            embeddings=embeddings_model,
            docstore=docstore,
        )

    @classmethod
    @deprecated("0.1.4", removal="0.2.0", alternative="from_langchain")
    def with_openai(
        cls,
        generator_llm: str = "gpt-4o-mini",
        critic_llm: str = "gpt-4o",
        embeddings: str = "text-embedding-ada-002",
        docstore: t.Optional[DocumentStore] = None,
        chunk_size: int = 1024,
    ) -> "TestsetGenerator":
        generator_llm_model = ChatOpenAI(model=generator_llm)
        critic_llm_model = ChatOpenAI(model=critic_llm)
        embeddings_model = OpenAIEmbeddings(model=embeddings)

        return cls.from_langchain(
            generator_llm=generator_llm_model,
            critic_llm=critic_llm_model,
            embeddings=embeddings_model,
            docstore=docstore,
            chunk_size=chunk_size,
        )

Additionally, you can leverage the Prompt class from the ragas.llms.prompt module to create various prompts for generating and refining questions, extracting keyphrases, and more. This setup ensures that the TestsetGenerator class can handle both LangChain and OpenAI models by using the appropriate wrappers and class references ^[2][3].

By following these steps, you should be able to resolve the namespace conflicts and compatibility issues you're encountering.

To continue talking to Dosu, mention @dosu.

[Zolastic]

Unfortunately, this does not fix the error that appears when running

generator.generate_with_langchain_docs(documents, test_size=3, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}, raise_exceptions=False)

error:

The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass raise_exceptions=False incase you want to show only a warning message instead.

I am learning Ragas and am referring to the documentation and this GitHub file

[Zolastic]

Full Error Traceback:

I'm encountering the following error when trying to generate a test set using generate_with_langchain_docs. It appears related to the ragas docstore, and despite using raise_exceptions=False, the issue persists.

Error Traceback:

Traceback (most recent call last): 
  File "/Users/username/Documents/projects/learning/ragas-basics/testset-generation-2.py", line 32, in <module>
    testset = generator.generate_with_langchain_docs(documents, test_size=2, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}, raise_exceptions=False)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/username/Documents/projects/learning/ragas-basics/.venv/lib/python3.12/site-packages/ragas/testset/generator.py", line 206, in generate_with_langchain_docs
    self.docstore.add_documents(
  File "/Users/username/Documents/projects/learning/ragas-basics/.venv/lib/python3.12/site-packages/ragas/testset/docstore.py", line 214, in add_documents
    self.add_nodes(nodes, show_progress=show_progress)
  File "/Users/username/Documents/projects/learning/ragas-basics/.venv/lib/python3.12/site-packages/ragas/testset/docstore.py", line 253, in add_nodes
    raise ExceptionInRunner()
ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` in case you want to show only a warning message instead.

Here is the code I’m using:

from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

from langchain_community.document_loaders import WebBaseLoader

import nest_asyncio
nest_asyncio.apply()

loader = WebBaseLoader("https://www.nba.com/stats/players")
documents = loader.load()

# generator with openai models
generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm = ChatOpenAI(model="gpt-4")
embeddings = OpenAIEmbeddings()


generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

# generate testset
testset = generator.generate_with_langchain_docs(documents, test_size=2, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}, raise_exceptions=False)

df = testset.to_pandas()
print(df)   

I’ve tried the following:

• Setting raise_exceptions=False
• Using nest_asyncio and is_async=False

Still facing the issue. Would appreciate any help to debug this!

@dosu: If you can provide any insights or assistance here, that would be great! Others are welcome to chime in as well.

[dosubot]

To resolve the ExceptionInRunner error when generating a test set using generate_with_langchain_docs in the ragas library, despite setting raise_exceptions=False and using nest_asyncio, you need to ensure that the test_data_rows are properly handled even when exceptions are raised. The relevant code snippet shows that ExceptionInRunner is raised if test_data_rows is empty:

try:
    test_data_rows = exec.results()
    if not test_data_rows:
        raise ExceptionInRunner()
except ValueError as e:
    raise e

Since raise_exceptions is set to False, the Executor should not raise exceptions, but it seems that the ExceptionInRunner is still being raised if test_data_rows is empty. To handle this, you can modify the code to check for raise_exceptions before raising ExceptionInRunner:

try:
    test_data_rows = exec.results()
    if not test_data_rows and raise_exceptions:
        raise ExceptionInRunner()
except ValueError as e:
    raise e

This way, ExceptionInRunner will only be raised if raise_exceptions is True.

Here is the modified code snippet:

try:
    test_data_rows = exec.results()
    if not test_data_rows and raise_exceptions:
        raise ExceptionInRunner()
except ValueError as e:
    raise e

This change ensures that the ExceptionInRunner is not raised when raise_exceptions is set to False. Additionally, the ExceptionInRunner class in the ragas library is designed to provide a warning message when raise_exceptions=False, which aligns with this approach:

class ExceptionInRunner(RagasException):
    """
    Exception raised when an exception is raised in the executor.
    """

    def __init__(self):
        msg = "The runner thread which was running the jobs raised an exception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` in case you want to show only a warning message instead."
        super().__init__(msg)

By making this adjustment, you should be able to avoid the ExceptionInRunner error when raise_exceptions is set to False ^[1][2].

This is a tough nut to crack, could really use your help on this, @jjmachan.

To continue talking to Dosu, mention @dosu.

[Zolastic]

Issue Update:

I was able to solve the issue by editing the docstore.py file under the testset folder. However, I'm not entirely sure if my solution is the best or if there’s a more appropriate way to resolve it. Here’s a summary of the changes I made:

1. Changes to Node Class:

Before (Original Code):

class Node(Document):
    keyphrases: t.List[str] = Field(default_factory=list, repr=False)
    relationships: t.Dict[Direction, t.Any] = Field(default_factory=dict, repr=False)
    doc_similarity: t.Optional[float] = Field(default=None, repr=False)
    wins: int = 0

After (My Changes):

class Node(Document):
    keyphrases: t.List[str] = Field(default_factory=list, repr=False)
    relationships: t.Dict[Direction, t.Any] = {}  # Changed Field to a standard dict
    doc_similarity: t.Optional[float] = Field(default=None, repr=False)
    wins: int = 0

2. Changes to add_documents Method:

Before (Original Code):

def add_documents(self, docs: t.Sequence[Document], show_progress=True):
    # split documents with self.splitter into smaller nodes
    nodes = [
        Node.from_langchain_document(d)
        for d in self.splitter.transform_documents(docs)
    ]
    self.add_nodes(nodes, show_progress=show_progress)

After (My Changes):

def add_documents(self, docs: t.Sequence[Document], show_progress=True):
    """
    Add documents in batch mode.
    """
    assert self.embeddings is not None, "Embeddings must be set"

    # split documents with self.splitter into smaller nodes
    docs = self.splitter.transform_documents(docs)

    # Create the nodes and set the filename for each one
    nodes = []
    for i, d in enumerate(docs):
        node = Node.from_langchain_document(d)
        node.metadata['filename'] = f"document_{i}.txt"  # Set custom filename
        node.embedding = None  # Clear embedding for each node
        node.keyphrases = []   # Clear keyphrases for each node
        nodes.append(node)

    print("nodes in add_documents: ", nodes[0].embedding)
    self.add_nodes(nodes, show_progress=show_progress)

Why I Made These Changes:

1. Node Class Change:
   I was encountering an issue where modifying the relationships field led to errors because Field (from Pydantic) does not support item assignment. By changing relationships to a standard dictionary ({}), I was able to avoid this error.

2. add_documents Method Change:

   • I explicitly cleared the embedding and keyphrases for each node to ensure they start with fresh values before processing.

Question:

Are these changes the correct approach for fixing the issue? Is there a better or more standard way to address these issues within the library’s architecture?

@dosu, do you have any insights or suggestions on how to approach this? Any feedback or guidance from others would be greatly appreciated!

[dosubot]

Your changes address the issues you encountered, but let's review them to ensure they align with the library's architecture and best practices.

1. Changes to Node Class

Original Code:

class Node(Document):
    keyphrases: t.List[str] = Field(default_factory=list, repr=False)
    relationships: t.Dict[Direction, t.Any] = Field(default_factory=dict, repr=False)
    doc_similarity: t.Optional[float] = Field(default=None, repr=False)
    wins: int = 0

Your Changes:

class Node(Document):
    keyphrases: t.List[str] = Field(default_factory=list, repr=False)
    relationships: t.Dict[Direction, t.Any] = {}  # Changed Field to a standard dict
    doc_similarity: t.Optional[float] = Field(default=None, repr=False)
    wins: int = 0

Analysis:

• Issue with Field: The original use of Field with default_factory=dict is correct for Pydantic models. The error you encountered might be due to how the relationships field is being modified elsewhere in the code.
• Standard Dictionary: Changing relationships to a standard dictionary ({}) avoids the Pydantic-specific behavior but loses some benefits of Pydantic's validation and serialization.

Recommendation:

Instead of changing relationships to a standard dictionary, ensure that any modifications to this field are done in a way that Pydantic supports. For example, if you need to update the dictionary, use methods that Pydantic models support.

2. Changes to add_documents Method

Original Code:

def add_documents(self, docs: t.Sequence[Document], show_progress=True):
    # split documents with self.splitter into smaller nodes
    nodes = [
        Node.from_langchain_document(d)
        for d in self.splitter.transform_documents(docs)
    ]
    self.add_nodes(nodes, show_progress=show_progress)

Your Changes:

def add_documents(self, docs: t.Sequence[Document], show_progress=True):
    """
    Add documents in batch mode.
    """
    assert self.embeddings is not None, "Embeddings must be set"

    # split documents with self.splitter into smaller nodes
    docs = self.splitter.transform_documents(docs)

    # Create the nodes and set the filename for each one
    nodes = []
    for i, d in enumerate(docs):
        node = Node.from_langchain_document(d)
        node.metadata['filename'] = f"document_{i}.txt"  # Set custom filename
        node.embedding = None  # Clear embedding for each node
        node.keyphrases = []   # Clear keyphrases for each node
        nodes.append(node)

    print("nodes in add_documents: ", nodes[0].embedding)
    self.add_nodes(nodes, show_progress=show_progress)

Analysis:

• Clearing Embeddings and Keyphrases: Explicitly clearing embedding and keyphrases ensures that each node starts with fresh values, which can help avoid issues with stale data.
• Setting Custom Filenames: Adding a custom filename to each node's metadata is a good practice for tracking and debugging.

Recommendation:

Your changes to the add_documents method are reasonable. However, ensure that clearing embedding and keyphrases aligns with the intended use of the library. If these fields are expected to be set elsewhere, clearing them might not be necessary.

Final Thoughts:

• Node Class: Consider keeping the Field for relationships and ensure modifications are Pydantic-compatible.
• add_documents Method: Your changes are fine, but double-check if clearing embedding and keyphrases is necessary for your use case.

If you encounter further issues, it might be helpful to review how relationships are being modified and ensure that any updates to Pydantic fields are done correctly.

To continue talking to Dosu, mention @dosu.

[os1ma]

I've encountered the same error. It seems that the default field values of ragas Node and Document, which inherit from langchain_core.pydantic_v1 BaseModel, might be causing the issue. In langchain v0.3, pydantic v0.2 is used internally, and there might be a problem with combining this with langchain_core.pydantic_v1.

[Zolastic]

Thank you for the suggestion! 🙏

I changed my langchain version to 0.2.11 and langchain-openai to 0.1.8, and it worked perfectly. It seems the issue was indeed related to the version mismatch with pydantic. I’ll proceed with these versions for now.

Thanks again for the help!