TestsetGenerator in local Ollama

[hanifa2102]

[ ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
I am not able to do testset generation using the local Ollama models.

Ragas version: 0.2.16.dev2+gb284337
Python version: 3.11.5

Code to Reproduce

    from llama_index.llms.ollama import Ollama
    from llama_index.embeddings.huggingface import HuggingFaceEmbedding
    from ragas.llms import LangchainLLMWrapper
    from ragas.embeddings import LangchainEmbeddingsWrapper


    generator_llm = LangchainLLMWrapper(Ollama(model='gemma3:27',request_timeout=120.0,temperature=0))
    generator_embeddings = LangchainEmbeddingsWrapper(HuggingFaceEmbedding(model_name="mixedbread-ai/mxbai-embed-large-v1",truncate_dim=512,local_files_only=False,embed_batch_size=8,trust_remote_code=True))

    from ragas.testset import TestsetGenerator

    generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
    dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
    return dataset

Error trace

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[17], [line 1](vscode-notebook-cell:?execution_count=17&line=1)
----> [1](vscode-notebook-cell:?execution_count=17&line=1) dataset=langchain_local(docs)

File ~/llm/AI-Assistant/johndoe/../src/eval/ragas_eval.py:50, in langchain_local(docs)
     [47](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/src/eval/ragas_eval.py:47) from ragas.testset import TestsetGenerator
     [49](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/src/eval/ragas_eval.py:49) generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
---> [50](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/src/eval/ragas_eval.py:50) dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
     [51](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/src/eval/ragas_eval.py:51) return dataset

File ~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:188, in TestsetGenerator.generate_with_langchain_docs(self, documents, testset_size, transforms, transforms_llm, transforms_embedding_model, query_distribution, run_config, callbacks, with_debugging_logs, raise_exceptions)
    [185](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:185) apply_transforms(kg, transforms)
    [186](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:186) self.knowledge_graph = kg
--> [188](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:188) return self.generate(
    [189](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:189)     testset_size=testset_size,
    [190](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:190)     query_distribution=query_distribution,
    [191](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:191)     run_config=run_config,
    [192](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:192)     callbacks=callbacks,
    [193](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:193)     with_debugging_logs=with_debugging_logs,
    [194](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:194)     raise_exceptions=raise_exceptions,
    [195](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:195) )

File ~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:369, in TestsetGenerator.generate(self, testset_size, query_distribution, num_personas, run_config, batch_size, callbacks, token_usage_parser, with_debugging_logs, raise_exceptions)
    [366](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:366)     patch_logger("ragas.experimental.testset.transforms", logging.DEBUG)
    [368](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:368) if self.persona_list is None:
--> [369](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:369)     self.persona_list = generate_personas_from_kg(
    [370](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:370)         llm=self.llm,
    [371](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:371)         kg=self.knowledge_graph,
    [372](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:372)         num_personas=num_personas,
    [373](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:373)         callbacks=callbacks,
    [374](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:374)     )
    [375](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:375) else:
    [376](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:376)     random.shuffle(self.persona_list)

File ~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:95, in generate_personas_from_kg(kg, llm, persona_generation_prompt, num_personas, filter_fn, callbacks)
     [93](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:93) nodes = [node for node in kg.nodes if filter_fn(node)]
     [94](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:94) if len(nodes) == 0:
---> [95](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:95)     raise ValueError(
     [96](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:96)         "No nodes that satisfied the given filer. Try changing the filter."
     [97](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:97)     )
     [99](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:99) summaries = [node.properties.get("summary") for node in nodes]
    [100](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:100) summaries = [summary for summary in summaries if isinstance(summary, str)]

ValueError: No nodes that satisfied the given filer. Try changing the filter.

Expected behavior
TestsetGenerator similar to the OpenAI call.

Additional context

1. In this case, I have tried with Langchain and Ollama, but my final goal would be to LLamaindex and Ollama instead. But wrt to the original tutorial, just wanted to change one variable (OpenAI to Ollama) only first.
2. Also checked chatgpt

Replace Ollama with a LangChain LLM that supports agenerate_prompt, like ChatOpenAI

Any ideas if ragas has any plans to support a locall llm, as there is restriction for our project to use cloud/web based llms.

[hanifa2102]

Just to add, not sure, if it is a bug. Mod feel free to change the label, if required.