LangChain Agent Fails to Recognize Tool Names with Descriptions and Incomplete Operation Addition

[eunhye1kim]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

import os
import requests
import yaml

os.environ["OPENAI_API_KEY"] = "sk-REDACTED"

from langchain_community.agent_toolkits.openapi import planner
from langchain_openai.chat_models import ChatOpenAI

from langchain_community.agent_toolkits.openapi.spec import reduce_openapi_spec
from langchain.requests import RequestsWrapper
from requests.packages.urllib3.exceptions import InsecureRequestWarning

# Ignore SSL warnings
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

with open("/home/ehkim/git/testprj/code_snippet/swagger.yaml") as f:
    data = yaml.load(f, Loader=yaml.FullLoader)
swagger_api_spec = reduce_openapi_spec(data)

def construct_superset_aut_headers(url=None):
    import requests
    url = "https://superset.mydomain.com/api/v1/security/login"
    payload = {
        "username": "myusername",
        "password": "mypassword",
        "provider": "db",
        "refresh": True
    }
    headers = {
        "Content-Type": "application/json"
    }

    response = requests.post(url, json=payload, headers=headers, verify=False)
    data = response.json()
    return {"Authorization": f"Bearer {data['access_token']}"}

from langchain.globals import set_debug
set_debug(True)

llm = ChatOpenAI(model='gpt-4o')
swagger_requests_wrapper = RequestsWrapper(headers=construct_superset_aut_headers(), verify=False)
superset_agent = planner.create_openapi_agent(
    swagger_api_spec,
    swagger_requests_wrapper,
    llm,
    allowed_operations=["GET", "POST", "PUT", "DELETE", "PATCH"],
    allow_dangerous_requests=True,
    agent_executor_kwargs={'handle_parsing_errors':True},
    handle_parsing_errors=True
)

superset_agent.run(
    """
    1. Get the dataset using the following information. (tool: requests_post, API: /api/v1/dataset/get_or_create/, database_id: 1, table_name: raw_esg_volume, response : {{'result' : {{'table_id': (dataset_id)}}}})
    2. Retrieve the dataset information obtained in step 1. (tool: requests_get, API: /api/v1/dataset/dataset/{{dataset_id}}/, params: None)
    3. Create a chart referencing the dataset obtained in step 2. The chart should plot the trend of total, online_news, and (total - online_news) values as a line chart. (tool: requests_post, API: /api/v1/chart/, database_id: 1)
    4. Return the URL of the created chart. https://superset.mydomain.com/explore/?slice_id={{chart_id}}
    When specifying the action, only write the tool name without any additional explanation.
    """
)

In this file, I used swagger.yaml file from https://superset.demo.datahubproject.io/api/v1/_openapi.
It's json format, so I converted it with below code.

```python
import json
import yaml

# read file
with open('swagger.json', 'r') as json_file:
    json_data = json.load(json_file)

# write file
with open('swagger.yaml', 'w') as yaml_file:
    yaml.dump(json_data, yaml_file, default_flow_style=False)

Error Message and Stack Trace (if applicable)

There's no exception because of handle_parsing_error=True but failure to solve user's request.
The below is agent log.

[chain/start] [chain:AgentExecutor] Entering Chain run with input:
{
  "input": "\n    1. Get the dataset using the following information. (tool: requests_post, API: /api/v1/dataset/get_or_create/, database_id: 1, (syncopation)    "
}
[chain/start] [chain:AgentExecutor > chain:LLMChain] Entering Chain run with input:
{
  "input": "\n    1. Get the dataset using the following information. (tool: requests_post, API: /api/v1/dataset/get_or_create/, database_id: 1, (syncopation)    ",
  "agent_scratchpad": "",
  "stop": [
    "\nObservation:",
    "\n\tObservation:"
  ]
}

...
(syncopation)
(api_planner log)
(syncopation)
(api_controller log)
...

[chain/end] [chain:AgentExecutor > tool:api_controller > chain:AgentExecutor > chain:LLMChain] [2.73s] Exiting Chain run with output:
{
  "text": "Action: The action to take is to make a POST request to the `/api/v1/dataset/get_or_create/` endpoint to create or get the dataset for the table 'raw_esg_volume' in the database with ID 1.\nAction Input: \n```json\n{\n  \"url\": \"https://superset.mydomain.com/api/v1/dataset/get_or_create/\",\n  \"data\": {\n    \"database_id\": 1,\n    \"table_name\": \"raw_esg_volume\"\n  },\n  \"output_instructions\": \"Extract the table_id from the response\"\n}\n```"
}
[tool/start] [chain:AgentExecutor > tool:api_controller > chain:AgentExecutor > tool:invalid_tool] Entering Tool run with input:
"{'requested_tool_name': "The action to take is to make a POST request to the `/api/v1/dataset/get_or_create/` endpoint to create or get the dataset for the table 'raw_esg_volume' in the database with ID 1.", 'available_tool_names': ['requests_get', 'requests_post', 'requests_put']}"
[tool/end] [chain:AgentExecutor > tool:api_controller > chain:AgentExecutor > tool:invalid_tool] [0ms] Exiting Tool run with output:
"The action to take is to make a POST request to the `/api/v1/dataset/get_or_create/` endpoint to create or get the dataset for the table 'raw_esg_volume' in the database with ID 1. is not a valid tool, try one of [requests_get, requests_post, requests_put]."
[chain/start] [chain:AgentExecutor > tool:api_controller > chain:AgentExecutor > chain:LLMChain] Entering Chain run with input:
{
  "input": "1. POST /api/v1/dataset/get_or_create/ with params {'database_id': 1, 'table_name': 'raw_esg_volume'}",
  "agent_scratchpad": "Action: The action to take is to make a POST request to the `/api/v1/dataset/get_or_create/` endpoint to create or get the dataset for the table 'raw_esg_volume' in the database with ID 1.\nAction Input: \n```json\n{\n  \"url\": \"https://superset.mydomain.com/api/v1/dataset/get_or_create/\",\n  \"data\": {\n    \"database_id\": 1,\n    \"table_name\": \"raw_esg_volume\"\n  },\n  \"output_instructions\": \"Extract the table_id from the response\"\n}\n```\nObservation: The action to take is to make a POST request to the `/api/v1/dataset/get_or_create/` endpoint to create or get the dataset for the table 'raw_esg_volume' in the database with ID 1. is not a valid tool, try one of [requests_get, requests_post, requests_put].\nThought:",
  "stop": [
    "\nObservation:",
    "\n\tObservation:"
  ]
}
[llm/start] [chain:AgentExecutor > tool:api_controller > chain:AgentExecutor > chain:LLMChain > llm:ChatOpenAI] Entering LLM run with input:
{
  "prompts": [
    "Human: You are an agent that gets a sequence of API calls and given their documentation, should execute them and return the final response.\nIf you cannot complete them and run into issues, you should explain the issue. If you're unable to resolve an API call, you can retry the API call. When interacting with API objects, you should extract ids for inputs to other API calls but ids and names for outputs returned to the User.\n\n\nHere is documentation on the API:\nBase url: https://superset.mydomain.com/\nEndpoints:\n== Docs for POST /api/v1/dataset/get_or_create/ == \nrequestBody:\n  content:\n    application/json:\n      schema:\n        properties:\n          always_filter_main_dttm:\n            default: false\n            type: boolean\n          database:\n            type: integer\n          external_url:\n            nullable: true\n            type: string\n          is_managed_externally:\n            nullable: true\n            type: boolean\n          normalize_columns:\n            default: false\n            type: boolean\n          owners:\n            items:\n              type: integer\n            type: array\n          schema:\n            maxLength: 250\n            minLength: 0\n            nullable: true\n            type: string\n          sql:\n            nullable: true\n            type: string\n          table_name:\n            maxLength: 250\n            minLength: 1\n            type: string\n        required:\n        - database\n        - table_name\n        type: object\n  description: Dataset schema\n  required: true\n\n== Docs for POST /api/v1/dataset/get_or_create/ == \nrequestBody:\n  content:\n    application/json:\n      schema:\n        properties:\n          always_filter_main_dttm:\n            default: false\n            type: boolean\n          database_id:\n            description: ID of database table belongs to\n            type: integer\n          normalize_columns:\n            default: false\n            type: boolean\n          schema:\n            description: The schema the table belongs to\n            maxLength: 250\n            minLength: 0\n            nullable: true\n            type: string\n          table_name:\n            description: Name of table\n            type: string\n          template_params:\n            description: Template params for the table\n            type: string\n        required:\n        - database_id\n        - table_name\n        type: object\n  required: true\nresponses:\n  content:\n    application/json:\n      schema:\n        properties:\n          result:\n            properties:\n              table_id:\n                type: integer\n            type: object\n        type: object\n  description: The ID of the table\n\n\n\n\nHere are tools to execute requests against the API: requests_get: Use this to GET content from a website.\nInput to the tool should be a json string with 3 keys: \"url\", \"params\" and \"output_instructions\".\nThe value of \"url\" should be a string. \nThe value of \"params\" should be a dict of the needed and available parameters from the OpenAPI spec related to the endpoint. \nIf parameters are not needed, or not available, leave it empty.\nThe value of \"output_instructions\" should be instructions on what information to extract from the response, \nfor example the id(s) for a resource(s) that the GET request fetches.\n\nrequests_post: Use this when you want to POST to a website.\nInput to the tool should be a json string with 3 keys: \"url\", \"data\", and \"output_instructions\".\nThe value of \"url\" should be a string.\nThe value of \"data\" should be a dictionary of key-value pairs you want to POST to the url.\nThe value of \"output_instructions\" should be instructions on what information to extract from the response, for example the id(s) for a resource(s) that the POST request creates.\nAlways use double quotes for strings in the json string.\nrequests_put: Use this when you want to PUT to a website.\nInput to the tool should be a json string with 3 keys: \"url\", \"data\", and \"output_instructions\".\nThe value of \"url\" should be a string.\nThe value of \"data\" should be a dictionary of key-value pairs you want to PUT to the url.\nThe value of \"output_instructions\" should be instructions on what information to extract from the response, for example the id(s) for a resource(s) that the PUT request creates.\nAlways use double quotes for strings in the json string.\n\n\nStarting below, you should follow this format:\n\nPlan: the plan of API calls to execute\nThought: you should always think about what to do\nAction: the action to take, should be one of the tools [requests_get, requests_post, requests_put]\nAction Input: the input to the action\nObservation: the output of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I am finished executing the plan (or, I cannot finish executing the plan without knowing some other information.)\nFinal Answer: the final output from executing the plan or missing information I'd need to re-plan correctly.\n\n\nBegin!\n\nPlan: 1. POST /api/v1/dataset/get_or_create/ with params {'database_id': 1, 'table_name': 'raw_esg_volume'}\nThought:\nAction: The action to take is to make a POST request to the `/api/v1/dataset/get_or_create/` endpoint to create or get the dataset for the table 'raw_esg_volume' in the database with ID 1.\nAction Input: \n```json\n{\n  \"url\": \"https://superset.mydomain.com/api/v1/dataset/get_or_create/\",\n  \"data\": {\n    \"database_id\": 1,\n    \"table_name\": \"raw_esg_volume\"\n  },\n  \"output_instructions\": \"Extract the table_id from the response\"\n}\n```\nObservation: The action to take is to make a POST request to the `/api/v1/dataset/get_or_create/` endpoint to create or get the dataset for the table 'raw_esg_volume' in the database with ID 1. is not a valid tool, try one of [requests_get, requests_post, requests_put].\nThought:"
  ]
}
[llm/end] [chain:AgentExecutor > tool:api_controller > chain:AgentExecutor > chain:LLMChain > llm:ChatOpenAI] [4.12s] Exiting LLM run with output:
{
  "generations": [
    [
      {
        "text": "It looks like there was an error in the previous action. I will correct the action to use the appropriate tool, which is `requests_post`.\n\nPlan: 1. POST /api/v1/dataset/get_or_create/ with params {'database_id': 1, 'table_name': 'raw_esg_volume'}\nThought: Make a POST request to the `/api/v1/dataset/get_or_create/` endpoint to create or get the dataset for the table 'raw_esg_volume' in the database with ID 1.\nAction: Execute the corrected action using the `requests_post` tool.\nAction Input:\n```json\n{\n  \"url\": \"https://superset.mydomain.com/api/v1/dataset/get_or_create/\",\n  \"data\": {\n    \"database_id\": 1,\n    \"table_name\": \"raw_esg_volume\"\n  },\n  \"output_instructions\": \"Extract the table_id from the response\"\n}\n```",
        "generation_info": {
          "finish_reason": "stop",
          "logprobs": null
        },
        "type": "ChatGeneration",
        "message": {
          "lc": 1,
          "type": "constructor",
          "id": [
            "langchain",
            "schema",
            "messages",
            "AIMessage"
          ],
          "kwargs": {
            "content": "It looks like there was an error in the previous action. I will correct the action to use the appropriate tool, which is `requests_post`.\n\nPlan: 1. POST /api/v1/dataset/get_or_create/ with params {'database_id': 1, 'table_name': 'raw_esg_volume'}\nThought: Make a POST request to the `/api/v1/dataset/get_or_create/` endpoint to create or get the dataset for the table 'raw_esg_volume' in the database with ID 1.\nAction: Execute the corrected action using the `requests_post` tool.\nAction Input:\n```json\n{\n  \"url\": \"https://superset.mydomain.com/api/v1/dataset/get_or_create/\",\n  \"data\": {\n    \"database_id\": 1,\n    \"table_name\": \"raw_esg_volume\"\n  },\n  \"output_instructions\": \"Extract the table_id from the response\"\n}\n```",
            "response_metadata": {
              "token_usage": {
                "completion_tokens": 196,
                "prompt_tokens": 1296,
                "total_tokens": 1492
              },
              "model_name": "gpt-4o",
              "system_fingerprint": "fp_c4e5b6fa31",
              "finish_reason": "stop",
              "logprobs": null
            },
            "type": "ai",
            "id": "run-b38b50e3-b4d1-44ef-996a-76b132d46f79-0",
            "usage_metadata": {
              "input_tokens": 1296,
              "output_tokens": 196,
              "total_tokens": 1492
            },
            "tool_calls": [],
            "invalid_tool_calls": []
          }
        }
      }
    ]
  ],
  "llm_output": {
    "token_usage": {
      "completion_tokens": 196,
      "prompt_tokens": 1296,
      "total_tokens": 1492
    },
    "model_name": "gpt-4o",
    "system_fingerprint": "fp_c4e5b6fa31"
  },
  "run": null
}
[chain/end] [chain:AgentExecutor > tool:api_controller > chain:AgentExecutor > chain:LLMChain] [4.12s] Exiting Chain run with output:
{
  "text": "It looks like there was an error in the previous action. I will correct the action to use the appropriate tool, which is `requests_post`.\n\nPlan: 1. POST /api/v1/dataset/get_or_create/ with params {'database_id': 1, 'table_name': 'raw_esg_volume'}\nThought: Make a POST request to the `/api/v1/dataset/get_or_create/` endpoint to create or get the dataset for the table 'raw_esg_volume' in the database with ID 1.\nAction: Execute the corrected action using the `requests_post` tool.\nAction Input:\n```json\n{\n  \"url\": \"https://superset.mydomain.com/api/v1/dataset/get_or_create/\",\n  \"data\": {\n    \"database_id\": 1,\n    \"table_name\": \"raw_esg_volume\"\n  },\n  \"output_instructions\": \"Extract the table_id from the response\"\n}\n```"
}
[tool/start] [chain:AgentExecutor > tool:api_controller > chain:AgentExecutor > tool:invalid_tool] Entering Tool run with input:
"{'requested_tool_name': 'Execute the corrected action using the `requests_post` tool.', 'available_tool_names': ['requests_get', 'requests_post', 'requests_put']}"
[tool/end] [chain:AgentExecutor > tool:api_controller > chain:AgentExecutor > tool:invalid_tool] [0ms] Exiting Tool run with output:
"Execute the corrected action using the `requests_post` tool. is not a valid tool, try one of [requests_get, requests_post, requests_put]."

...
(loop)
...

Description

I expected two things.
One is that all five operations added to the planner.create_openapi_agent function are added to api_controller, and the other is that only the tool name is correctly entered in the response in the tool name field when executing the API.
However, as observed through logs, both did not work well.

System Info

platform : linux
python : 3.11

$ pip freeze | grep langchain
langchain==0.2.4
langchain-community==0.2.4
langchain-core==0.2.6
langchain-openai==0.1.8
langchain-text-splitters==0.2.1