Fix: Tool action_input containing markdown with JSON

[mikkoh]

This PR fixes a bug where if a Tool returns JSON markdown in the action_input string then ChatConversationalAgentOutputParser.parse fails to parse the action_input properly.

An example text/message that fails to parse:

{"action":"ToolWithJson","action_input":"The tool input ```json\\n{\\"yes\\":true}\\n```"}

This PR optimistically tries to parse the text/message as JSON and if that fails then it attempts to find the JSON in the string.

Before this PR the above example fails with:

SyntaxError: Unexpected token '\', "\n{\"yes\":true}\n" is not valid JSON

This PR also adds further error handling to return that either action or action_input could not be found.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
langchainjs-docs	✅ Ready (Inspect)	Visit Preview	May 10, 2023 8:26pm

[jacoblee93]

The agent is prompted to return JSON in a markdown block, so we're actually being optimistic for an invalid response here?

[jacoblee93]

My gut says we should try to improve the prompt to more strictly output markdown - can you give an end-to-end example of an input that caused this?

[mikkoh]

Fair.

It should be noted I'm new to langchainjs and I could be doing things wrong.

Here's a brief description of what I'm doing without going into too many details since the details are confidential to work at Shopify.

• We create an agent using initializeAgentExecutorWithOptions
• This agent has a tool which on success returns JSON from it's _call method
• For simplicity and to carry the example from above the JSON that is returned by the _call method is '{"yes":true}'. Nothing else is returned just simply the string'{"yes": true}'. In reality for our use-case the JSON is way more complex than this

Here's a snippet of the verbose output modified to match the yes: true example:

RESPONSE FORMAT INSTRUCTIONS
----------------------------

When responding to me please, please output a response in one of two formats:

**Option 1:**
Use this if you want the human to use a tool.
Markdown code snippet formatted in the following schema:

```json
{
    "action": string \ The action to take. Must be one of JSONGiver
AnotherThing
    "action_input": string \ The input to the action
}
```

**Option #2:**
Use this if you want to respond directly to the human. Markdown code snippet formatted in the following schema:

```json
{
    "action": "Final Answer",
    "action_input": string \ You should put what you want to return to use here
}
```

USER'S INPUT
--------------------
Here is the user's input (remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else):

Can you give me JSON information?
AI: {
    "action": "JSONGiver",
    "action_input": "yes"
}
Human: TOOL RESPONSE:
---------------------
{"yes":true}

USER'S INPUT
--------------------

Okay, so what is the response to my original question? If using information from tools, you must say it explicitly - I have forgotten all TOOL RESPONSES! Remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else.

LLM d880e074-2d65-49ce-8b6b-a3cbdcbb201c finished: {"generations":[[{"text":"{\n    \"action\": \"Final Answer\",\n    \"action_input\": \"The JSON is:\\n\\n```json\\n{\"yes\": true}\\n```\"\n}"}}]]}

Finished chain.
[14:34:00.532] ERROR: Unexpected token \ in JSON at position 0

[jacoblee93]

If you don't explicitly specify that the tool returns JSON output does it help?

If you're looking for the overall output of the agent to be in a different format, an output or output fixing parser may be helpful:

https://js.langchain.com/docs/modules/prompts/output_parsers/

Output parsers contain a prompt that nudges the chain to return a specific format, and output fixing parsers will attempt to fix the output of a given run by passing mismatched output into another LLM.

[mikkoh]

I actually don't explicitly specify I'm returning JSON. I just return the JSON as string. I don't include the any markdown declarations that would specify the returned value is JSON. I just return '{"yes":true}'

The agent wraps my code with the markdown specifier and then when the Final Answer is parsed it fails.

I'll take a look at Output parsers to understand how they could be helpful.

[jacoblee93]

Ah gotcha - an output fixing parser may help you yeah. But I think one or two prompt tweaks may fix it in a more sustainable way - I can give it a shot later today.

If you can share a full code sample that consistently repros this it would speed me up significantly?

[HenryHengZJ]

Following this since we are having quite a lot of this error when agent failed to parse the tools output as JSON. Attaching the logs here:

[jacoblee93]

Hey @mikkoh and @HenryHengZJ I opened #1292 to address this with some prompt tweaks and a retry mechanism for poorly formatted agent action outputs - have a look at that one and I'll close this if it looks reasonable to you!

[HenryHengZJ]

Hey @mikkoh and @HenryHengZJ I opened #1292 to address this with some prompt tweaks and a retry mechanism for poorly formatted agent action outputs - have a look at that one and I'll close this if it looks reasonable to you!

Do we have the retry mechanism there? So far I don't see the error anymore, think the prompt and output parser kinda fixed it

[jacoblee93]

Yes, though it's not merged and released yet.

Great to hear that!

[jacoblee93]

#1292 was merged last week - if this recurs please open an issue.