Expensive agentic failure loop

[ahgraber]

[x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
In some cases / with some models, the instructions to generate json result in something like

Here's the generated abstract conceptual question in the requested JSON format::

{"text": ...}

Would you like me to explain in more detail?

The built-in validator will fail this because it cannot parse the json, and then make several repeated requests to try to conform the response to the specified schema. If this is an error associated with an LLM's default response preferences, the agentic retry loop is basically a waste of tokens.

Ragas version: 0.2.1
Python version:

Code to Reproduce
Use Anthropic Claude 3.5 Sonnet with the new testset generator

Error trace
error.txt

Expected behavior
Intelligent extraction of json object from text blob

Additional context
Add any other context about the problem here.

[ahgraber]

I'd like to take this on; I have an idea for a simple heuristic-based json extractor that should have a harmless fail mode.

[jjmachan]

hey @ahgraber that would be really useful - do give it a shot.

I'm guessing a simple json re is what you have in mind? would be helpful! One more thing I wanted to check was if the JSON decoding errors are being passed into the prompts

the other main improvement is throwing an error when generation stops due to max-tokens - this one I'm trying to fix

Hopefully we'll be able to make the json parser much better 🙂 ❤️

[ahgraber]

I don't even think I'll need regex -- basically I'm planning to search for paired [] or {}.

I've noticed that newer models tend to return complete JSON (provided there's no truncation for token count), but the models are frequently verbose and will try to be too helpful e.g., "Here's the json requested: {...} Please ask for further explanation". Langchain's default json/markdown parser fails in this case when the json is embedded in text and not delimited with a '```json'.

I'll keep an eye on my LMStudio logs and see if I can find the rendered repair prompt.

[dosubot]

The issue you're encountering seems to be related to the JSON validation and extraction process in Ragas version 0.2.1. The current implementation uses the RagasOutputParser class, which attempts to parse the output string to ensure it adheres to the specified JSON schema. If the output string doesn't meet the JSON format requirements, it uses a retry mechanism to correct it, which can lead to repeated requests and token wastage if the initial response is not in the expected format ^[1].

Your idea of implementing a heuristic-based JSON extractor could potentially help in intelligently extracting JSON objects from text blobs, reducing unnecessary retries. This approach might involve identifying and isolating JSON-like structures within the text, which could then be parsed separately.

To continue talking to Dosu, mention @dosu.

[ahgraber]

@jjmachan See attached log for repair loop from a HeadlinesExtractor transformation.

parse-errors.log

1. It does not look like json decoding error information is being included in the repair prompt
2. The original text is included with increasing levels of escape characters, ultimately leading (in my case) to a crash due to exceeding the context length.