Agent Intros

docs/source/notebooks/extraction/intro.ipynb

@@ -64,10 +64,10 @@
     "Each task corresponds to a LangSmith dataset with the following schema:\n",
     "\n",
     "Inputs:\n",
-    "- input: str - the input text\n",
+    "- `input: str` - the input text\n",
     "\n",
     "Outputs\n",
-    "- output: str - the expected extraction result, as a json object\n"
+    "- `output: str` - the expected extraction result, as a json object\n"
    ]
   },
   {
@@ -89,6 +89,7 @@
    "outputs": [],
    "source": [
     "from langchain.chat_models import ChatOpenAI\n",
+    "\n",
     "from langchain_benchmarks.extraction import get_eval_config\n",
     "\n",
     "eval_llm = ChatOpenAI(model=\"gpt-4\", model_kwargs={\"seed\": 42})\n",

docs/source/notebooks/retrieval/intro.ipynb

@@ -63,7 +63,6 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "\n",
     "### Task resources\n",
     "\n",
     "In addition to a name, daset_id, and description, each retrieval task provides a few helper functions you can use to configure your pipeline.\n",

docs/source/notebooks/tool_usage/intro.ipynb

@@ -0,0 +1,175 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "6728b05f-e3bb-487a-8818-e0d5d18b5501",
+   "metadata": {},
+   "source": [
+    "# Agent Tool Usage Tasks\n",
+    "\n",
+    "These tasks are meant to grade your agent's effectiveness at using tools to accomplish tasks.\n",
+    "\n",
+    "You can check an up-to-date list of tool usage tasks in the registry:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "a57e65d7-dbd6-4128-8260-f6ee3b43157c",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table>\n",
+       "<thead>\n",
+       "<tr><th>Name                              </th><th>Type         </th><th>Dataset ID                                                                                                                                                 </th><th>Description  </th></tr>\n",
+       "</thead>\n",
+       "<tbody>\n",
+       "<tr><td>Tool Usage - Typewriter (1 tool)  </td><td>ToolUsageTask</td><td><a href=\"https://smith.langchain.com/public/59577193-8938-4ccf-92a7-e8a96bcf4f86/d\" target=\"_blank\" rel=\"noopener\">59577193-8938-4ccf-92a7-e8a96bcf4f86</a></td><td>Environment with a single tool that accepts a single letter as input, and prints it on a piece of virtual paper.\n",
+       "\n",
+       "The objective of this task is to evaluate the ability of the model to use the provided tools to repeat a given input string.\n",
+       "\n",
+       "For example, if the string is 'abc', the tools 'a', 'b', and 'c' must be invoked in that order.\n",
+       "\n",
+       "The dataset includes examples of varying difficulty. The difficulty is measured by the length of the string.              </td></tr>\n",
+       "<tr><td>Tool Usage - Typewriter (26 tools)</td><td>ToolUsageTask</td><td><a href=\"https://smith.langchain.com/public/128af05e-aa00-4e3b-a958-d166dd450581/d\" target=\"_blank\" rel=\"noopener\">128af05e-aa00-4e3b-a958-d166dd450581</a></td><td>Environment with 26 tools each tool represents a letter of the alphabet.\n",
+       "\n",
+       "The objective of this task is to evaluate the model's ability the use tools\n",
+       "for a simple repetition task.\n",
+       "\n",
+       "For example, if the string is 'abc', the tools 'a', 'b', and 'c' must be invoked in that order.\n",
+       "\n",
+       "The dataset includes examples of varying difficulty. The difficulty is measured by the length of the string.\n",
+       "\n",
+       "This is a variation of the typer writer task, where 26 parameterless tools are\n",
+       "given instead of a single tool that takes a letter as an argument.              </td></tr>\n",
+       "<tr><td>Tool Usage - Relational Data      </td><td>ToolUsageTask</td><td><a href=\"https://smith.langchain.com/public/1d89f4b3-5f73-48cf-a127-2fdeb22f6d84/d\" target=\"_blank\" rel=\"noopener\">1d89f4b3-5f73-48cf-a127-2fdeb22f6d84</a></td><td>Environment with fake data about users and their locations and favorite foods.\n",
+       "\n",
+       "The environment provides a set of tools that can be used to query the data.\n",
+       "\n",
+       "The objective of this task is to evaluate the ability to use the provided tools to answer questions about relational data.\n",
+       "\n",
+       "The dataset contains 21 examples of varying difficulty. The difficulty is measured by the number of tools that need to be used to answer the question.\n",
+       "\n",
+       "Each example is composed of a question, a reference answer, and information about the sequence in which tools should be used to answer the question.\n",
+       "\n",
+       "Success is measured by the ability to answer the question correctly, and efficiently.              </td></tr>\n",
+       "<tr><td>Multiverse Math                   </td><td>ToolUsageTask</td><td><a href=\"https://smith.langchain.com/public/594f9f60-30a0-49bf-b075-f44beabf546a/d\" target=\"_blank\" rel=\"noopener\">594f9f60-30a0-49bf-b075-f44beabf546a</a></td><td>An environment that contains a few basic math operations, but with altered results.\n",
+       "\n",
+       "For example, multiplication of 5*3 will be re-interpreted as 5*3*1.1. The basic operations retain some basic properties, such as commutativity, associativity, and distributivity; however, the results are different than expected.\n",
+       "\n",
+       "The objective of this task is to evaluate the ability to use the provided tools to solve simple math questions and ignore any innate knowledge about math.              </td></tr>\n",
+       "</tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "Registry(tasks=[ToolUsageTask(name='Tool Usage - Typewriter (1 tool)', dataset_id='https://smith.langchain.com/public/59577193-8938-4ccf-92a7-e8a96bcf4f86/d', description=\"Environment with a single tool that accepts a single letter as input, and prints it on a piece of virtual paper.\\n\\nThe objective of this task is to evaluate the ability of the model to use the provided tools to repeat a given input string.\\n\\nFor example, if the string is 'abc', the tools 'a', 'b', and 'c' must be invoked in that order.\\n\\nThe dataset includes examples of varying difficulty. The difficulty is measured by the length of the string.\\n\", create_environment=<function get_environment at 0x132f9cea0>, instructions=\"Repeat the given string using the provided tools. Do not write anything else or provide any explanations. For example, if the string is 'abc', you must print the letters 'a', 'b', and 'c' one at a time and in that order. \"), ToolUsageTask(name='Tool Usage - Typewriter (26 tools)', dataset_id='https://smith.langchain.com/public/128af05e-aa00-4e3b-a958-d166dd450581/d', description=\"Environment with 26 tools each tool represents a letter of the alphabet.\\n\\nThe objective of this task is to evaluate the model's ability the use tools\\nfor a simple repetition task.\\n\\nFor example, if the string is 'abc', the tools 'a', 'b', and 'c' must be invoked in that order.\\n\\nThe dataset includes examples of varying difficulty. The difficulty is measured by the length of the string.\\n\\nThis is a variation of the typer writer task, where 26 parameterless tools are\\ngiven instead of a single tool that takes a letter as an argument.\\n\", create_environment=<function get_environment at 0x132f9d3a0>, instructions=\"Repeat the given string by using the provided tools. Do not write anything else or provide any explanations. For example, if the string is 'abc', you must invoke the tools 'a', 'b', and 'c' in that order. Please invoke the functions without any arguments.\"), ToolUsageTask(name='Tool Usage - Relational Data', dataset_id='https://smith.langchain.com/public/1d89f4b3-5f73-48cf-a127-2fdeb22f6d84/d', description='Environment with fake data about users and their locations and favorite foods.\\n\\nThe environment provides a set of tools that can be used to query the data.\\n\\nThe objective of this task is to evaluate the ability to use the provided tools to answer questions about relational data.\\n\\nThe dataset contains 21 examples of varying difficulty. The difficulty is measured by the number of tools that need to be used to answer the question.\\n\\nEach example is composed of a question, a reference answer, and information about the sequence in which tools should be used to answer the question.\\n\\nSuccess is measured by the ability to answer the question correctly, and efficiently.\\n', create_environment=<function get_environment at 0x132f9c9a0>, instructions=\"Please answer the user's question by using the tools provided. Do not guess the answer. Keep in mind that entities like users,foods and locations have both a name and an ID, which are not the same.\"), ToolUsageTask(name='Multiverse Math', dataset_id='https://smith.langchain.com/public/594f9f60-30a0-49bf-b075-f44beabf546a/d', description='An environment that contains a few basic math operations, but with altered results.\\n\\nFor example, multiplication of 5*3 will be re-interpreted as 5*3*1.1. The basic operations retain some basic properties, such as commutativity, associativity, and distributivity; however, the results are different than expected.\\n\\nThe objective of this task is to evaluate the ability to use the provided tools to solve simple math questions and ignore any innate knowledge about math.\\n', create_environment=<function get_environment at 0x132f9c2c0>, instructions='You are requested to solve math questions in an alternate mathematical universe. The operations have been altered to yield different results than expected. Do not guess the answer or rely on your  innate knowledge of math. Use the provided tools to answer the question. While associativity and commutativity apply, distributivity does not. Answer the question using the fewest possible tools. Only include the numeric response without any clarifications.')])"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from langchain_benchmarks import registry\n",
+    "\n",
+    "registry.filter(Type=\"ToolUsageTask\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9f54cdd3-67f6-43ba-a929-1a6ed1b01296",
+   "metadata": {},
+   "source": [
+    "### Task resources\n",
+    "\n",
+    "In addition to a name, daset_id, and description, the `tool_use` directory also has a shared agent factory you can use to get started:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3363e86d-3c86-4297-81b6-f73899be48b0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_benchmarks.tool_usage import agents\n",
+    "\n",
+    "agent_factory = agents.OpenAIAgentFactory(task, model=\"gpt-3.5-turbo-16k\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "994bb145-9b12-4a60-87da-003f44dd13e5",
+   "metadata": {},
+   "source": [
+    "They also define a `create_environment` method that returns a ToolUsageEnvironment object:\n",
+    "\n",
+    "```python\n",
+    "class ToolUsageEnvironment:\n",
+    "    \"\"\"An instance of an environment for tool usage.\"\"\"\n",
+    "\n",
+    "    tools: List[BaseTool]\n",
+    "    \"\"\"The tools that can be used in the environment.\"\"\"\n",
+    "\n",
+    "    read_state: Optional[Callable[[], Any]] = None\n",
+    "    \"\"\"A function that returns the current state of the environment.\"\"\"\n",
+    "```\n",
+    "\n",
+    "This is used to define the available tools for a given dataset and to let any evaluators read the world state grade the agent."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3d5e48c4-d5d0-4d19-9bab-61c01a512f21",
+   "metadata": {},
+   "source": [
+    "### Dataset schema\n",
+    "\n",
+    "Each task corresponds to a LangSmith dataset with the following schema:\n",
+    "\n",
+    "Inputs:\n",
+    "- `question: str` - the user question\n",
+    "\n",
+    "Outputs\n",
+    "- `expected_steps: list` - the expected order of tools used\n",
+    "- `reference: str` - the expected answer\n",
+    "\n",
+    "There may be additional output keys, such as:\n",
+    "- `order_matters`: bool - whether the order of tool invocations matters\n",
+    "- `state: any` - the end 'state' the environment should be in after a given data point"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cd9dea3f-68b5-47f7-9a12-c7b5eafc4a37",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/source/notebooks/tool_usage/multiverse_math.ipynb

@@ -22,19 +22,24 @@
     "    Please note that the modified operations are not guaranteed to even make sense in the real world since not all properties will be retained (e.g., distributive property)."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "03488ab1-31ed-41c2-8da2-46b02599b181",
+   "metadata": {},
+   "source": [
+    "For this code to work, please configure LangSmith environment variables with your credentials."
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "id": "8df805c7-02b2-4c59-8b15-507015f5a284",
-   "metadata": {
-    "tags": [
-     "remove-cell"
-    ]
-   },
+   "execution_count": 1,
+   "id": "1615b8ff-688a-4447-8c4c-d64ad02818ed",
+   "metadata": {},
    "outputs": [],
    "source": [
-    "%load_ext autoreload\n",
-    "%autoreload 2"
+    "import os\n",
+    "\n",
+    "os.environ[\"LANGCHAIN_API_KEY\"] = \"sk-...\"  # Your LangSmith API key"
    ]
   },
   {
@@ -49,14 +54,6 @@
     "from langchain_benchmarks import clone_public_dataset, registry"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "03488ab1-31ed-41c2-8da2-46b02599b181",
-   "metadata": {},
-   "source": [
-    "For this code to work, please configure LangSmith environment variables with your credentials."
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": 3,
@@ -475,7 +472,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.6"
+   "version": "3.11.2"
   }
  },
  "nbformat": 4,

docs/source/notebooks/tool_usage/relational_data.ipynb

@@ -1156,7 +1156,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.6"
+   "version": "3.11.2"
   }
  },
  "nbformat": 4,