LangGraph_and_LangSmith_Assignment_Version(1).ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"gJXW_DgiSebM"},"source":["# LangGraph and LangSmith - Agentic RAG Powered by LangChain\n","\n","In the following notebook we'll complete the following tasks:\n","\n","- 🤝 Breakout Room #1:\n","  1. Install required libraries\n","  2. Set Environment Variables\n","  3. Creating our Tool Belt\n","  4. Creating Our State\n","  5. Creating and Compiling A Graph!\n","  \n","\n","- 🤝 Breakout Room #2:\n","  1. Creating an Evaluation Dataset\n","  2. Adding Evaluators\n","  3. Evaluating"]},{"cell_type":"markdown","metadata":{"id":"djQ3nRAgoF67"},"source":["# 🤝 Breakout Room #1"]},{"cell_type":"markdown","metadata":{"id":"e7pQDUhUnIo8"},"source":["## LangGraph - Building Cyclic Applications with LangChain\n","\n","LangGraph is a tool that leverages LangChain Expression Language to build coordinated multi-actor and stateful applications that includes cyclic behaviour.\n","\n","### Why Cycles?\n","\n","In essence, we can think of a cycle in our graph as a more robust and customizable loop. It allows us to keep our application agent-forward while still giving the powerful functionality of traditional loops.\n","\n","Due to the inclusion of cycles over loops, we can also compose rather complex flows through our graph in a much more readable and natural fashion. Effetively allowing us to recreate appliation flowcharts in code in an almost 1-to-1 fashion.\n","\n","### Why LangGraph?\n","\n","Beyond the agent-forward approach - we can easily compose and combine traditional \"DAG\" (directed acyclic graph) chains with powerful cyclic behaviour due to the tight integration with LCEL. This means it's a natural extension to LangChain's core offerings!"]},{"cell_type":"markdown","metadata":{"id":"3_fLDElOVoop"},"source":["## Task 1:  Dependencies\n","\n","We'll first install all our required libraries."]},{"cell_type":"code","execution_count":1,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":23071,"status":"ok","timestamp":1711193380679,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"KaVwN269EttM","outputId":"7363555a-44d2-4463-b6e4-5b22892dc752"},"outputs":[{"name":"stdout","output_type":"stream","text":["\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m810.5/810.5 kB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m52.4/52.4 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m49.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m269.1/269.1 kB\u001b[0m \u001b[31m28.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m71.6/71.6 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m262.4/262.4 kB\u001b[0m \u001b[31m29.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m75.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m81.1/81.1 kB\u001b[0m \u001b[31m10.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.7/5.7 MB\u001b[0m \u001b[31m91.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.0/8.0 MB\u001b[0m \u001b[31m50.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m6.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.0/53.0 kB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m138.5/138.5 kB\u001b[0m \u001b[31m13.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m9.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.8/77.8 kB\u001b[0m \u001b[31m10.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25h  Building wheel for sgmllib3k (setup.py) ... \u001b[?25l\u001b[?25hdone\n"]}],"source":["!pip install -qU langchain langchain_openai langgraph arxiv duckduckgo-search"]},{"cell_type":"markdown","metadata":{"id":"wujPjGJuoPwg"},"source":["## Task 2: Environment Variables\n","\n","We'll want to set both our OpenAI API key and our LangSmith environment variables."]},{"cell_type":"code","execution_count":2,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4214,"status":"ok","timestamp":1711193384880,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"Jdh8CoVWHRvs","outputId":"be8ed704-8f20-4fbb-f19b-314f07b503af"},"outputs":[{"name":"stdout","output_type":"stream","text":["OpenAI API Key:··········\n"]}],"source":["import os\n","import getpass\n","\n","os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")"]},{"cell_type":"code","execution_count":3,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":5939,"status":"ok","timestamp":1711193396482,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"Nv0glIDyHmRt","outputId":"3a7cec50-4d3f-411a-f10e-3847a110e72f"},"outputs":[{"name":"stdout","output_type":"stream","text":["LangSmith API Key: ··········\n"]}],"source":["from uuid import uuid4\n","\n","os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n","os.environ[\"LANGCHAIN_PROJECT\"] = f\"AIE1 - LangGraph - {uuid4().hex[0:8]}\"\n","os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass(\"LangSmith API Key: \")"]},{"cell_type":"markdown","metadata":{"id":"sBRyQmEAVzua"},"source":["## Task 3: Creating our Tool Belt\n","\n","As is usually the case, we'll want to equip our agent with a toolbelt to help answer questions and add external knowledge.\n","\n","There's a tonne of tools in the [LangChain Community Repo](https://github.com/langchain-ai/langchain/tree/master/libs/community/langchain_community/tools) but we'll stick to a couple just so we can observe the cyclic nature of LangGraph in action!\n","\n","We'll leverage:\n","\n","- [Duck Duck Go Web Search](https://github.com/langchain-ai/langchain/tree/master/libs/community/langchain_community/tools/ddg_search)\n","- [Arxiv](https://github.com/langchain-ai/langchain/tree/master/libs/community/langchain_community/tools/arxiv)"]},{"cell_type":"markdown","metadata":{"id":"2k6n_Dob2F46"},"source":["####🏗️ Activity #1:\n","\n","Please add the tools to use into our toolbelt.\n","\n","> NOTE: Each tool in our toolbelt should be a method."]},{"cell_type":"code","execution_count":4,"metadata":{"executionInfo":{"elapsed":1041,"status":"ok","timestamp":1711193427888,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"lAxaSvlfIeOg"},"outputs":[],"source":["from langchain_community.tools.ddg_search import DuckDuckGoSearchRun\n","from langchain_community.tools.arxiv.tool import ArxivQueryRun\n","\n","tool_belt = [\n","    DuckDuckGoSearchRun(),### YOUR TOOL HERE\n","    ArxivQueryRun() ### YOUR TOOL HERE\n","]"]},{"cell_type":"markdown","metadata":{"id":"1FdOjEslXdRR"},"source":["### Actioning with Tools\n","\n","Now that we've created our tool belt - we need to create a process that will let us leverage them when we need them.\n","\n","We'll use the built-in [`ToolExecutor`](https://github.com/langchain-ai/langgraph/blob/main/langgraph/prebuilt/tool_executor.py) to do so."]},{"cell_type":"code","execution_count":5,"metadata":{"executionInfo":{"elapsed":397,"status":"ok","timestamp":1711193432487,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"cFr1m80-JZsD"},"outputs":[],"source":["from langgraph.prebuilt import ToolExecutor\n","\n","tool_executor = ToolExecutor(tool_belt)"]},{"cell_type":"markdown","metadata":{"id":"VI-C669ZYVI5"},"source":["### Model\n","\n","Now we can set-up our model! We'll leverage the familiar OpenAI model suite for this example - but it's not *necessary* to use with LangGraph. LangGraph supports all models - though you might not find success with smaller models - as such, they recommend you stick with:\n","\n","- OpenAI's GPT-3.5 and GPT-4\n","- Anthropic's Claude\n","- Google's Gemini\n","\n","> NOTE: Because we're leveraging the OpenAI function calling API - we'll need to use OpenAI *for this specific example* (or any other service that exposes an OpenAI-style function calling API."]},{"cell_type":"code","execution_count":6,"metadata":{"executionInfo":{"elapsed":1072,"status":"ok","timestamp":1711193438435,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"QkNS8rNZJs4z"},"outputs":[],"source":["from langchain_openai import ChatOpenAI\n","\n","model = ChatOpenAI(temperature=0)"]},{"cell_type":"markdown","metadata":{"id":"Ugkj3GzuZpQv"},"source":["Now that we have our model set-up, let's \"put on the tool belt\", which is to say: We'll bind our LangChain formatted tools to the model in an OpenAI function calling format."]},{"cell_type":"code","execution_count":7,"metadata":{"executionInfo":{"elapsed":1,"status":"ok","timestamp":1711193439402,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"4OdMqFafZ_0V"},"outputs":[],"source":["from langchain_core.utils.function_calling import convert_to_openai_function\n","\n","functions = [convert_to_openai_function(t) for t in tool_belt]\n","model = model.bind_functions(functions)"]},{"cell_type":"code","execution_count":8,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":6,"status":"ok","timestamp":1711193443088,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"If90dSMxQzHx","outputId":"793544a6-8aca-4bdf-cd51-53eabafc1c94"},"outputs":[{"name":"stdout","output_type":"stream","text":["{'name': 'duckduckgo_search', 'description': 'A wrapper around DuckDuckGo Search. Useful for when you need to answer questions about current events. Input should be a search query.', 'parameters': {'type': 'object', 'properties': {'query': {'description': 'search query to look up', 'type': 'string'}}, 'required': ['query']}}\n"]}],"source":["print(functions[0])"]},{"cell_type":"markdown","metadata":{"id":"ERzuGo6W18Lr"},"source":["#### ❓ Question #1:\n","\n","How does the model determine which tool to use?\n","\n","Ans. The LLM based on the description and metadata of the functions or tools determines which tool or function to be called."]},{"cell_type":"markdown","metadata":{"id":"_296Ub96Z_H8"},"source":["## Putting the State in Stateful\n","\n","Earlier we used this phrasing:\n","\n","`coordinated multi-actor and stateful applications`\n","\n","So what does that \"stateful\" mean?\n","\n","To put it simply - we want to have some kind of object which we can pass around our application that holds information about what the current situation (state) is. Since our system will be constructed of many parts moving in a coordinated fashion - we want to be able to ensure we have some commonly understood idea of that state.\n","\n","LangGraph leverages a `StatefulGraph` which uses an `AgentState` object to pass information between the various nodes of the graph.\n","\n","There are more options than what we'll see below - but this `AgentState` object is one that is stored in a `TypedDict` with the key `messages` and the value is a `Sequence` of `BaseMessages` that will be appended to whenever the state changes.\n","\n","Let's think about a simple example to help understand exactly what this means (we'll simplify a great deal to try and clearly communicate what state is doing):\n","\n","1. We initialize our state object:\n","  - `{\"messages\" : []}`\n","2. Our user submits a query to our application.\n","  - New State: `HumanMessage(#1)`\n","  - `{\"messages\" : [HumanMessage(#1)}`\n","3. We pass our state object to an Agent node which is able to read the current state. It will use the last `HumanMessage` as input. It gets some kind of output which it will add to the state.\n","  - New State: `AgentMessage(#1, additional_kwargs {\"function_call\" : \"WebSearchTool\"})`\n","  - `{\"messages\" : [HumanMessage(#1), AgentMessage(#1, ...)]}`\n","4. We pass our state object to a \"conditional node\" (more on this later) which reads the last state to determine if we need to use a tool - which it can determine properly because of our provided object!"]},{"cell_type":"code","execution_count":9,"metadata":{"executionInfo":{"elapsed":409,"status":"ok","timestamp":1711193458072,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"mxL9b_NZKUdL"},"outputs":[],"source":["from typing import TypedDict, Annotated, Sequence\n","import operator\n","from langchain_core.messages import BaseMessage\n","\n","class AgentState(TypedDict):\n","  messages: Annotated[Sequence[BaseMessage], operator.add]"]},{"cell_type":"markdown","metadata":{"id":"vWsMhfO9grLu"},"source":["## It's Graphing Time!\n","\n","Now that we have state, and we have tools, and we have an LLM - we can finally start making our graph!\n","\n","Let's take a second to refresh ourselves about what a graph is in this context.\n","\n","Graphs, also called networks in some circles, are a collection of connected objects.\n","\n","The objects in question are typically called nodes, or vertices, and the connections are called edges.\n","\n","Let's look at a simple graph.\n","\n","![image](https://i.imgur.com/2NFLnIc.png)\n","\n","Here, we're using the coloured circles to represent the nodes and the yellow lines to represent the edges. In this case, we're looking at a fully connected graph - where each node is connected by an edge to each other node.\n","\n","If we were to think about nodes in the context of LangGraph - we would think of a function, or an LCEL runnable.\n","\n","If we were to think about edges in the context of LangGraph - we might think of them as \"paths to take\" or \"where to pass our state object next\".\n","\n","Let's create some nodes and expand on our diagram.\n","\n","> NOTE: Due to the tight integration with LCEL - we can comfortably create our nodes in an async fashion!"]},{"cell_type":"code","execution_count":10,"metadata":{"executionInfo":{"elapsed":395,"status":"ok","timestamp":1711193462329,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"91flJWtZLUrl"},"outputs":[],"source":["from langgraph.prebuilt import ToolInvocation\n","import json\n","from langchain_core.messages import FunctionMessage\n","\n","def call_model(state):\n","  messages = state[\"messages\"]\n","  response = model.invoke(messages)\n","  return {\"messages\" : [response]}\n","\n","def call_tool(state):\n","  last_message = state[\"messages\"][-1]\n","\n","  action = ToolInvocation(\n","      tool=last_message.additional_kwargs[\"function_call\"][\"name\"],\n","      tool_input=json.loads(\n","          last_message.additional_kwargs[\"function_call\"][\"arguments\"]\n","      )\n","  )\n","\n","  response = tool_executor.invoke(action)\n","\n","  function_message = FunctionMessage(content=str(response), name=action.tool)\n","\n","  return {\"messages\" : [function_message]}"]},{"cell_type":"markdown","metadata":{"id":"2bwR7MgWj3Wg"},"source":["Now we have two total nodes. We have:\n","\n","- `call_model` is a node that will...well...call the model\n","- `call_tool` is a node which will call a tool\n","\n","Let's start adding nodes! We'll update our diagram along the way to keep track of what this looks like!\n"]},{"cell_type":"code","execution_count":11,"metadata":{"executionInfo":{"elapsed":387,"status":"ok","timestamp":1711193468016,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"_vF4_lgtmQNo"},"outputs":[],"source":["from langgraph.graph import StateGraph, END\n","\n","workflow = StateGraph(AgentState)\n","\n","workflow.add_node(\"agent\", call_model)\n","workflow.add_node(\"action\", call_tool)"]},{"cell_type":"markdown","metadata":{"id":"b8CjRlbVmRpW"},"source":["Let's look at what we have so far:\n","\n","![image](https://i.imgur.com/md7inqG.png)"]},{"cell_type":"markdown","metadata":{"id":"uaXHpPeSnOWC"},"source":["Next, we'll add our entrypoint. All our entrypoint does is indicate which node is called first."]},{"cell_type":"code","execution_count":12,"metadata":{"executionInfo":{"elapsed":374,"status":"ok","timestamp":1711193473226,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"YGCbaYqRnmiw"},"outputs":[],"source":["workflow.set_entry_point(\"agent\")"]},{"cell_type":"markdown","metadata":{"id":"BUsfGoSpoF9U"},"source":["![image](https://i.imgur.com/wNixpJe.png)"]},{"cell_type":"markdown","metadata":{"id":"0Q_pQgHmoW0M"},"source":["Now we want to build a \"conditional edge\" which will use the output state of a node to determine which path to follow.\n","\n","We can help conceptualize this by thinking of our conditional edge as a conditional in a flowchart!\n","\n","Notice how our function simply checks if there is a \"function_call\" kwarg present.\n","\n","Then we create an edge where the origin node is our agent node and our destination node is *either* the action node or the END (finish the graph).\n","\n","It's important to highlight that the dictionary passed in as the third parameter (the mapping) should be created with the possible outputs of our conditional function in mind. In this case `should_continue` outputs either `\"end\"` or `\"continue\"` which are subsequently mapped to the action node or the END node."]},{"cell_type":"code","execution_count":13,"metadata":{"executionInfo":{"elapsed":566,"status":"ok","timestamp":1711193477545,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"1BZgb81VQf9o"},"outputs":[],"source":["def should_continue(state):\n","  last_message = state[\"messages\"][-1]\n","\n","  if \"function_call\" not in last_message.additional_kwargs:\n","    return \"end\"\n","\n","  return \"continue\"\n","\n","workflow.add_conditional_edges(\n","    \"agent\",\n","    should_continue,\n","    {\n","        \"continue\" : \"action\",\n","        \"end\" : END\n","    }\n",")"]},{"cell_type":"markdown","metadata":{"id":"-Cvhcf4jp0Ce"},"source":["Let's visualize what this looks like.\n","\n","![image](https://i.imgur.com/8ZNwKI5.png)"]},{"cell_type":"markdown","metadata":{"id":"yKCjWJCkrJb9"},"source":["Finally, we can add our last edge which will connect our action node to our agent node. This is because we *always* want our action node (which is used to call our tools) to return its output to our agent!"]},{"cell_type":"code","execution_count":14,"metadata":{"executionInfo":{"elapsed":409,"status":"ok","timestamp":1711193482107,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"UvcgbHf1rIXZ"},"outputs":[],"source":["workflow.add_edge(\"action\", \"agent\")"]},{"cell_type":"markdown","metadata":{"id":"EiWDwBQtrw7Z"},"source":["Let's look at the final visualization.\n","\n","![image](https://i.imgur.com/NWO7usO.png)"]},{"cell_type":"markdown","metadata":{"id":"KYqDpErlsCsu"},"source":["All that's left to do now is to compile our workflow - and we're off!"]},{"cell_type":"code","execution_count":15,"metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1711193486495,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"zt9-KS8DpzNx"},"outputs":[],"source":["app = workflow.compile()"]},{"cell_type":"markdown","metadata":{"id":"xhNWIwBL1W4Q"},"source":["#### ❓ Question #2:\n","\n","Is there any specific limit to how many times we can cycle?\n","\n","If not, how could we impose a limit to the number of cycles?\n","\n","Ans: There isn't a predefined maximum number of cycles. However, if we wish to set a cap on cycling, we could incorporate a conditional clause within the \"should_continue\" function."]},{"cell_type":"markdown","metadata":{"id":"VEYcTShCsPaa"},"source":["## Using Our Graph\n","\n","Now that we've created and compiled our graph - we can call it *just as we'd call any other* `Runnable`!\n","\n","Let's try out a few examples to see how it fairs:"]},{"cell_type":"code","execution_count":16,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":5630,"status":"ok","timestamp":1711193494016,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"Qn4n37PQRPII","outputId":"baaabc74-9b48-4e73-f684-abeb84ba9bf0"},"outputs":[{"data":{"text/plain":["{'messages': [HumanMessage(content='What is RAG in the context of Large Language Models? When did it break onto the scene?'),\n","  AIMessage(content='', additional_kwargs={'function_call': {'arguments': '{\"query\":\"RAG in the context of Large Language Models\"}', 'name': 'duckduckgo_search'}}, response_metadata={'token_usage': {'completion_tokens': 25, 'prompt_tokens': 171, 'total_tokens': 196}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3bc1b5746c', 'finish_reason': 'function_call', 'logprobs': None}),\n","  FunctionMessage(content='RAG stands for R etrieval- A ugmented G eneration. RAG enables large language models (LLM) to access and utilize up-to-date information. Hence, it improves the quality of relevance of the response from LLM. Below is a simple diagram of how RAG is implemented. The beauty of RAG lies in its ability to enable a language model to draw upon and leverage your own data to generate responses. While base models are traditionally trained on specific, point-in-time data, ensuring their effectiveness in performing tasks and adapting to the desired domain, they can struggle when faced with newer or current data. Key Takeaways. RAG is a relatively new artificial intelligence technique that can improve the quality of generative AI by allowing large language model (LLMs) to tap additional data resources without retraining. RAG models build knowledge repositories based on the organization\\'s own data, and the repositories can be continually updated to ... Retrieval-augmented generation (RAG) is a technique used to \"ground\" large language models (LLMs) with specific data sources, often sources that weren\\'t included in the models\\' original ... These advanced, generalized language models are trained on vast datasets, enabling them to understand and generate human-like text. In the context of RAG, LLMs are used to generate fully formed responses based on the user query and contextual information retrieved from the vector DBs during user queries. Querying', name='duckduckgo_search'),\n","  AIMessage(content='RAG stands for Retrieval-Augmented Generation. It is a technique used in the context of Large Language Models (LLMs) to improve the quality of generative AI by allowing LLMs to access and utilize up-to-date information from external data sources. RAG enables LLMs to generate responses based on the user query and contextual information retrieved from external data sources.\\n\\nRAG broke onto the scene as a relatively new artificial intelligence technique that can enhance the quality of generative AI by enabling LLMs to tap into additional data resources without the need for retraining. It allows LLMs to build knowledge repositories based on specific data sources, which can be continually updated to improve the relevance and accuracy of the generated responses.', response_metadata={'token_usage': {'completion_tokens': 147, 'prompt_tokens': 497, 'total_tokens': 644}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3bc1b5746c', 'finish_reason': 'stop', 'logprobs': None})]}"]},"execution_count":16,"metadata":{},"output_type":"execute_result"}],"source":["from langchain_core.messages import HumanMessage\n","\n","inputs = {\"messages\" : [HumanMessage(content=\"What is RAG in the context of Large Language Models? When did it break onto the scene?\")]}\n","\n","app.invoke(inputs)"]},{"cell_type":"markdown","metadata":{"id":"DBHnUtLSscRr"},"source":["Let's look at what happened:\n","\n","1. Our state object was populated with our request\n","2. The state object was passed into our entry point (agent node) and the agent node added an `AIMessage` to the state object and passed it along the conditional edge\n","3. The conditional edge received the state object, found the \"function_call\" `additional_kwarg`, and sent the state object to the action node\n","4. The action node added the response from the OpenAI function calling endpoint to the state object and passed it along the edge to the agent node\n","5. The agent node added a response to the state object and passed it along the conditional edge\n","6. The conditional edge received the state object, could not find the \"function_call\" `additional_kwarg` and passed the state object to END where we see it output in the cell above!\n","\n","Now let's look at an example that shows a multiple tool usage - all with the same flow!"]},{"cell_type":"code","execution_count":17,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":12650,"status":"ok","timestamp":1711193511347,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"afv2BuEsV5JG","outputId":"2bb95a16-1714-4c4e-ebb4-904ad05aa723"},"outputs":[{"data":{"text/plain":["{'messages': [HumanMessage(content='What is QLoRA in Machine Learning? Are their any papers that could help me understand? Once you have that information, can you look up the bio of the first author on the QLoRA paper?'),\n","  AIMessage(content='', additional_kwargs={'function_call': {'arguments': '{\"query\":\"QLoRA in Machine Learning\"}', 'name': 'duckduckgo_search'}}, response_metadata={'token_usage': {'completion_tokens': 22, 'prompt_tokens': 193, 'total_tokens': 215}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3bc1b5746c', 'finish_reason': 'function_call', 'logprobs': None}),\n","  FunctionMessage(content='Unlock the power of QLoRA with our definitive guide! Learn how to fine-tune the Falcon-7b model using PEFT for optimal AI performance. Step into the future of machine learning today. QLoRA is a fine-tuning method that combines Quantization and Low-Rank Adapters (LoRA). QLoRA is revolutionary in that it democratizes fine-tuning: it enables one to fine-tune massive models with billions of parameters on relatively small, highly available GPUs. ... QLoRA stands as a significant advancement in the field of machine learning ... Large Language Models (LLMs) are currently a hot topic in the field of machine learning. Imagine you\\'re an ML Engineer and your company has access to GPUs and open-source LLMs like LLAMA/Falcon. ... LoRA, Machine Learning, QLoRA, transformers. Categories: data-science, machine-learning. Updated: July 26, 2023. Share on Twitter Facebook ... \"Lora The Tuner\" By Daniel Warfield using MidJourney. All images by the author unless otherwise specified. Fine tuning is the process of tailoring a machine learning model to a specific application, which can be vital in achieving consistent and high quality performance. Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation.', name='duckduckgo_search'),\n","  AIMessage(content='', additional_kwargs={'function_call': {'arguments': '{\"query\":\"QLoRA in Machine Learning\"}', 'name': 'arxiv'}}, response_metadata={'token_usage': {'completion_tokens': 19, 'prompt_tokens': 546, 'total_tokens': 565}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3bc1b5746c', 'finish_reason': 'function_call', 'logprobs': None}),\n","  FunctionMessage(content=\"Published: 2023-05-23\\nTitle: QLoRA: Efficient Finetuning of Quantized LLMs\\nAuthors: Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer\\nSummary: We present QLoRA, an efficient finetuning approach that reduces memory usage\\nenough to finetune a 65B parameter model on a single 48GB GPU while preserving\\nfull 16-bit finetuning task performance. QLoRA backpropagates gradients through\\na frozen, 4-bit quantized pretrained language model into Low Rank\\nAdapters~(LoRA). Our best model family, which we name Guanaco, outperforms all\\nprevious openly released models on the Vicuna benchmark, reaching 99.3% of the\\nperformance level of ChatGPT while only requiring 24 hours of finetuning on a\\nsingle GPU. QLoRA introduces a number of innovations to save memory without\\nsacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is\\ninformation theoretically optimal for normally distributed weights (b) double\\nquantization to reduce the average memory footprint by quantizing the\\nquantization constants, and (c) paged optimziers to manage memory spikes. We\\nuse QLoRA to finetune more than 1,000 models, providing a detailed analysis of\\ninstruction following and chatbot performance across 8 instruction datasets,\\nmultiple model types (LLaMA, T5), and model scales that would be infeasible to\\nrun with regular finetuning (e.g. 33B and 65B parameter models). Our results\\nshow that QLoRA finetuning on a small high-quality dataset leads to\\nstate-of-the-art results, even when using smaller models than the previous\\nSoTA. We provide a detailed analysis of chatbot performance based on both human\\nand GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable\\nalternative to human evaluation. Furthermore, we find that current chatbot\\nbenchmarks are not trustworthy to accurately evaluate the performance levels of\\nchatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to\\nChatGPT. We release all of our models and code, including CUDA kernels for\\n4-bit training.\\n\\nPublished: 2023-12-31\\nTitle: Viz: A QLoRA-based Copyright Marketplace for Legally Compliant Generative AI\\nAuthors: Dipankar Sarkar\\nSummary: This paper aims to introduce and analyze the Viz system in a comprehensive\\nway, a novel system architecture that integrates Quantized Low-Rank Adapters\\n(QLoRA) to fine-tune large language models (LLM) within a legally compliant and\\nresource efficient marketplace. Viz represents a significant contribution to\\nthe field of artificial intelligence, particularly in addressing the challenges\\nof computational efficiency, legal compliance, and economic sustainability in\\nthe utilization and monetization of LLMs. The paper delineates the scholarly\\ndiscourse and developments that have informed the creation of Viz, focusing\\nprimarily on the advancements in LLM models, copyright issues in AI training\\n(NYT case, 2023), and the evolution of model fine-tuning techniques,\\nparticularly low-rank adapters and quantized low-rank adapters, to create a\\nsustainable and economically compliant framework for LLM utilization. The\\neconomic model it proposes benefits content creators, AI developers, and\\nend-users, delineating a harmonious integration of technology, economy, and\\nlaw, offering a comprehensive solution to the complex challenges of today's AI\\nlandscape.\\n\\nPublished: 2024-02-08\\nTitle: Accurate LoRA-Finetuning Quantization of LLMs via Information Retention\\nAuthors: Haotong Qin, Xudong Ma, Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xianglong Liu, Michele Magno\\nSummary: The LoRA-finetuning quantization of LLMs has been extensively studied to\\nobtain accurate yet compact LLMs for deployment on resource-constrained\\nhardware. However, existing methods cause the quantized LLM to severely degrade\\nand even fail to benefit from the finetuning of LoRA. This paper proposes a\\nnovel IR-QLoRA for pushing quantized LLMs with LoRA to be highly accurate\\nthrough information retention. The proposed IR-QLoRA mainly relies on two\\ntechnolog\", name='arxiv'),\n","  AIMessage(content='', additional_kwargs={'function_call': {'arguments': '{\"query\":\"Tim Dettmers bio\"}', 'name': 'duckduckgo_search'}}, response_metadata={'token_usage': {'completion_tokens': 21, 'prompt_tokens': 1515, 'total_tokens': 1536}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3bc1b5746c', 'finish_reason': 'function_call', 'logprobs': None}),\n","  FunctionMessage(content=\"Bio: Tim Dettmers's research focuses on making foundation models, such as ChatGPT, accessible to researchers and practitioners by reducing their resource requirements. This involves developing novel compression and networking algorithms and building systems that allow for memory-efficient, fast, and cheap deep learning. ... Bio: Tim Dettmers's research focuses on making foundation models, such as ChatGPT, accessible to researchers and practitioners by reducing their resource requirements. This involves developing novel compression and networking algorithms and building systems that allow for memory-efficient, fast, and cheap deep learning. ... Bio: Tim Dettmers's research focuses on making foundation models, such as ChatGPT, accessible to researchers and practitioners by reducing their resource requirements. This involves developing novel compression and networking algorithms and building systems that allow for memory-efficient, fast, and cheap deep learning. ... Bio: Tim Dettmers's research focuses on making foundation models, such as ChatGPT, accessible to researchers and practitioners by reducing their resource requirements. This involves developing novel compression and networking algorithms and building systems that allow for memory-efficient, fast, and cheap deep learning. ... Sponsored by Evolution AI: https://www.evolution.aiAbstract: Recent open-source large language models (LLMs) like LLaMA and Falcon are both high-quality and ...\", name='duckduckgo_search'),\n","  AIMessage(content='QLoRA is an efficient fine-tuning approach that combines Quantization and Low-Rank Adapters (LoRA) to reduce memory usage and enable fine-tuning of large language models on relatively small GPUs. One of the papers related to QLoRA is titled \"QLoRA: Efficient Finetuning of Quantized LLMs\" by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. The paper presents QLoRA as a method to finetune a 65B parameter model on a single 48GB GPU while maintaining performance.\\n\\nTim Dettmers is one of the authors of the QLoRA paper. His research focuses on making foundation models, such as ChatGPT, more accessible to researchers and practitioners by reducing their resource requirements. He works on developing compression and networking algorithms to enable memory-efficient, fast, and cost-effective deep learning.\\n\\nIf you would like more information or details on Tim Dettmers\\' bio, please let me know!', response_metadata={'token_usage': {'completion_tokens': 208, 'prompt_tokens': 1805, 'total_tokens': 2013}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3bc1b5746c', 'finish_reason': 'stop', 'logprobs': None})]}"]},"execution_count":17,"metadata":{},"output_type":"execute_result"}],"source":["inputs = {\"messages\" : [HumanMessage(content=\"What is QLoRA in Machine Learning? Are their any papers that could help me understand? Once you have that information, can you look up the bio of the first author on the QLoRA paper?\")]}\n","\n","app.invoke(inputs)"]},{"cell_type":"markdown","metadata":{"id":"CXzDlZVz1Hnf"},"source":["####🏗️ Activity #2:\n","\n","Please write out the steps the agent took to arrive at the correct answer.\n","\n","Answer:\n","\n","    1. The query was received.\n","    2. Then the agent assigns DuckDuckGo to search for information on QLoRA in the context of machine learning.\n","    3. After the agent receives responses or feedback from DuckDuckGo, the agent then made a request to Arxiv to address another aspect of the original request, specifically seeking papers related to QLoRA.\n","    4. Upon receiving the information from Arxiv, the agent identified that the biography of the lead author of the QLoRA paper was still missing, prompting it to issue another request to DuckDuckGo.\n","    5. With the latest feedback from DuckDuckGo, all pieces of information now collected, the agent prepared to deliver a comprehensive response to the user."]},{"cell_type":"markdown","metadata":{"id":"pV3XeFOT1Sar"},"source":["### Pre-processing for LangSmith"]},{"cell_type":"markdown","metadata":{"id":"wruQCuzewUuO"},"source":["To do a little bit more preprocessing, let's wrap our LangGraph agent in a simple chain."]},{"cell_type":"code","execution_count":18,"metadata":{"executionInfo":{"elapsed":4,"status":"ok","timestamp":1711193511347,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"oeXdQgbxwhTv"},"outputs":[],"source":["def convert_inputs(input_object):\n","  return {\"messages\" : [HumanMessage(content=input_object[\"question\"])]}\n","\n","def parse_output(input_state):\n","  return input_state[\"messages\"][-1].content\n","\n","agent_chain = convert_inputs | app | parse_output"]},{"cell_type":"code","execution_count":19,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":70},"executionInfo":{"elapsed":4004,"status":"ok","timestamp":1711193515348,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"orYxBZXSxJjZ","outputId":"e9d1e555-c7a9-4f05-80ae-d0d062bb500e"},"outputs":[{"data":{"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"},"text/plain":["\"RAG stands for Retrieval-augmented generation. It is an AI framework that improves the quality of language model-generated responses by grounding the model on external sources of knowledge to supplement the model's internal representation of information. RAG ensures that the model has access to current and reliable facts, enhancing the accuracy and reliability of generative AI models.\""]},"execution_count":19,"metadata":{},"output_type":"execute_result"}],"source":["agent_chain.invoke({\"question\" : \"What is RAG?\"})"]},{"cell_type":"markdown","metadata":{"id":"JQmrzYfrm1Dr"},"source":["# 🤝 Breakout Room #2"]},{"cell_type":"markdown","metadata":{"id":"f9UkCIqkpyZu"},"source":["## Task 1: Creating An Evaluation Dataset\n","\n","Just as we saw last week, we'll want to create a dataset to test our Agent's ability to answer questions.\n","\n","In order to do this - we'll want to provide some questions and some answers. Let's look at how we can create such a dataset below.\n","\n","```python\n","questions = [\n","    \"What optimizer is used in QLoRA?\",\n","    \"What data type was created in the QLoRA paper?\",\n","    \"What is a Retrieval Augmented Generation system?\",\n","    \"Who authored the QLoRA paper?\",\n","    \"What is the most popular deep learning framework?\",\n","    \"What significant improvements does the LoRA system make?\"\n","]\n","\n","answers = [\n","    {\"must_mention\" : [\"paged\", \"optimizer\"]},\n","    {\"must_mention\" : [\"NF4\", \"NormalFloat\"]},\n","    {\"must_mention\" : [\"ground\", \"context\"]},\n","    {\"must_mention\" : [\"Tim\", \"Dettmers\"]},\n","    {\"must_mention\" : [\"PyTorch\", \"TensorFlow\"]},\n","    {\"must_mention\" : [\"reduce\", \"parameters\"]},\n","]\n","```"]},{"cell_type":"markdown","metadata":{"id":"VfMXF2KAsQxs"},"source":["####🏗️ Activity #3:\n","\n","Please create a dataset in the above format with at least 5 questions."]},{"cell_type":"code","execution_count":21,"metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1711198814069,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"CbagRuJop83E"},"outputs":[],"source":["questions = [\n","    \"What novel layer type was introduced in the Transformer architecture?\",\n","    \"What data type was created in the QLoRA paper?\",\n","    \"What is a Retrieval Augmented Generation system?\",\n","    \"Who is recognized for pioneering work in deep learning and neural networks?\",\n","    \"Which benchmark dataset is commonly used for image classification tasks?\",\n","    \"What is the primary advantage of using dropout in neural networks?\"\n","]\n","\n","answers = [\n","    {\"must_mention\": [\"multi-head\", \"attention\"]},\n","    {\"must_mention\" : [\"NF4\", \"NormalFloat\"]},\n","    {\"must_mention\" : [\"ground\", \"context\"]},\n","    {\"must_mention\": [\"Yoshua\", \"Bengio\"]},\n","    {\"must_mention\": [\"ImageNet\", \"dataset\"]},\n","    {\"must_mention\": [\"reduce\", \"overfitting\"]},\n","]"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"H-4cWaW6VnJG"},"outputs":[],"source":["\"\"\"\n","\n","questions = [\n","    \"What optimizer is used in QLoRA?\",\n","    \"What data type was created in the QLoRA paper?\",\n","    \"What is a Retrieval Augmented Generation system?\",\n","    \"Who authored the QLoRA paper?\",\n","    \"What is the most popular deep learning framework?\",\n","    \"What significant improvements does the LoRA system make?\"\n","]\n","\n","answers = [\n","    {\"must_mention\" : [\"paged\", \"optimizer\"]},\n","    {\"must_mention\" : [\"NF4\", \"NormalFloat\"]},\n","    {\"must_mention\" : [\"ground\", \"context\"]},\n","    {\"must_mention\" : [\"Tim\", \"Dettmers\"]},\n","    {\"must_mention\" : [\"PyTorch\", \"TensorFlow\"]},\n","    {\"must_mention\" : [\"reduce\", \"parameters\"]},\n","]\n","\"\"\""]},{"cell_type":"markdown","metadata":{"id":"z7QVFuAmsh7L"},"source":["Now we can add our dataset to our LangSmith project using the following code which we saw last Thursday!"]},{"cell_type":"code","execution_count":22,"metadata":{"executionInfo":{"elapsed":1028,"status":"ok","timestamp":1711198854810,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"RLfrZrgSsn85"},"outputs":[],"source":["from langsmith import Client\n","\n","client = Client()\n","dataset_name = f\"Retrieval Augmented Generation - Evaluation Dataset - {uuid4().hex[0:8]}\"\n","\n","dataset = client.create_dataset(\n","    dataset_name=dataset_name,\n","    description=\"Questions about the QLoRA Paper to Evaluate RAG over the same paper.\"\n",")\n","\n","client.create_examples(\n","    inputs=[{\"question\" : q} for q in questions],\n","    outputs=answers,\n","    dataset_id=dataset.id,\n",")"]},{"cell_type":"markdown","metadata":{"id":"ciV73F9Q04w0"},"source":["#### ❓ Question #3:\n","\n","How are the correct answers associated with the questions?\n","\n","> NOTE: Feel free to indicate if this is problematic or not\n","\n","Ans: Correct answers are linked to their respective questions through a designated pair of \"must mention\" strings.\n","\n","However, this approach may not be correct because there are often numerous correct ways to address a question, and each of these correct answers can be expressed differently.\n","It's presumed that a valid response to any question should invariably contain this pair of strings. "]},{"cell_type":"markdown","metadata":{"id":"-lRTXUrTtP9Y"},"source":["## Task 2: Adding Evaluators\n","\n","Now we can add a custom evaluator to see if our responses contain the expected information.\n","\n","We'll be using a fairly naive exact-match process to determine if our response contains specific strings."]},{"cell_type":"code","execution_count":23,"metadata":{"executionInfo":{"elapsed":427,"status":"ok","timestamp":1711198866401,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"QrAUXMFftlAY"},"outputs":[],"source":["from langsmith.evaluation import EvaluationResult, run_evaluator\n","\n","@run_evaluator\n","def must_mention(run, example) -> EvaluationResult:\n","    prediction = run.outputs.get(\"output\") or \"\"\n","    required = example.outputs.get(\"must_mention\") or []\n","    score = all(phrase in prediction for phrase in required)\n","    return EvaluationResult(key=\"must_mention\", score=score)"]},{"cell_type":"markdown","metadata":{"id":"PNtHORUh0jZY"},"source":["#### ❓ Question #4:\n","\n","What are some ways you could improve this metric as-is?\n","\n","> NOTE: Alternatively you can suggest where gaps exist in this method.\n","\n","Ans. The \"must mention\" approach has its drawbacks, notably that certain questions can be correctly answered without explicitly including all the \"must_mention\" terms. This technique could unjustly disadvantage comprehensive answers that deliver the required insights through an analysis that omits the specified terms. It relies heavily on the hard coded must_mention key.\n","\n","To mitigate this issue, an effective solution could involve expanding the list of \"must mention\" phrases and deeming an answer correct if it incorporates at least one of these terms or use a combination of metrics that together provide a comprehensive evaluation of the model's performance."]},{"cell_type":"markdown","metadata":{"id":"mZ4DVSXl0BX5"},"source":["Now that we have created our custom evaluator - let's initialize our `RunEvalConfig` with it, and a few others:\n","\n","- `\"criteria\"` includes the default criteria which, in this case, means \"helpfulness\"\n","- `\"cot_qa\"` includes a criteria that bases whether or not the answer is correct by utilizing a Chain of Thought prompt and the provided context to determine if the response is correct or not."]},{"cell_type":"code","execution_count":24,"metadata":{"executionInfo":{"elapsed":390,"status":"ok","timestamp":1711198872866,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"sL4-XcjytWsu"},"outputs":[],"source":["from langchain.smith import RunEvalConfig, run_on_dataset\n","\n","eval_config = RunEvalConfig(\n","    custom_evaluators=[must_mention],\n","    evaluators=[\n","        \"criteria\",\n","        \"cot_qa\",\n","    ],\n",")"]},{"cell_type":"markdown","metadata":{"id":"r1RJr349zhv7"},"source":["Task 3: Evaluating\n","\n","All that is left to do is evaluate our agent's response!"]},{"cell_type":"code","execution_count":25,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":23280,"status":"ok","timestamp":1711198900727,"user":{"displayName":"Sherilyn","userId":"03312585092674113221"},"user_tz":-330},"id":"p5TeCUUkuGld","outputId":"585dc25f-5602-404d-fc06-dfd0c2769899"},"outputs":[{"name":"stdout","output_type":"stream","text":["View the evaluation results for project 'RAG Pipeline - Evaluation - c3c7bd10' at:\n","https://smith.langchain.com/o/200d3068-1fc0-5fb9-9780-f234a739f7fe/datasets/63dcfade-6b08-4358-af36-35020250bb09/compare?selectedSessions=483d6f04-95da-473c-bb45-66f948469658\n","\n","View all tests for Dataset Retrieval Augmented Generation - Evaluation Dataset - 62de3165 at:\n","https://smith.langchain.com/o/200d3068-1fc0-5fb9-9780-f234a739f7fe/datasets/63dcfade-6b08-4358-af36-35020250bb09\n","[------------------------------------------------->] 6/6\n"," Experiment Results:\n","        feedback.helpfulness  feedback.COT Contextual Accuracy feedback.must_mention error  execution_time                                run_id\n","count                   6.00                              6.00                     6     0            6.00                                     6\n","unique                   NaN                               NaN                     2     0             NaN                                     6\n","top                      NaN                               NaN                  True   NaN             NaN  df6f02ad-a721-4e0c-b5be-db6f67b08d44\n","freq                     NaN                               NaN                     4   NaN             NaN                                     1\n","mean                    1.00                              0.83                   NaN   NaN            3.13                                   NaN\n","std                     0.00                              0.41                   NaN   NaN            1.72                                   NaN\n","min                     1.00                              0.00                   NaN   NaN            1.43                                   NaN\n","25%                     1.00                              1.00                   NaN   NaN            1.85                                   NaN\n","50%                     1.00                              1.00                   NaN   NaN            2.85                                   NaN\n","75%                     1.00                              1.00                   NaN   NaN            3.78                                   NaN\n","max                     1.00                              1.00                   NaN   NaN            6.05                                   NaN\n"]},{"data":{"text/plain":["{'project_name': 'RAG Pipeline - Evaluation - c3c7bd10',\n"," 'results': {'60e5e9ba-6574-49c2-a536-c13b561a275c': {'input': {'question': 'What novel layer type was introduced in the Transformer architecture?'},\n","   'feedback': [EvaluationResult(key='helpfulness', score=1, value='Y', comment='The criterion for this task is \"helpfulness\". \\n\\nThe submission provides a clear and concise answer to the question asked, identifying the novel layer type introduced in the Transformer architecture as the \"self-attention\" mechanism. \\n\\nMoreover, the submission goes beyond just naming the layer type, it also explains what the self-attention mechanism does, which is to weigh the importance of different input tokens when making predictions. This additional information is insightful and can be very helpful for someone trying to understand the Transformer architecture.\\n\\nThe submission is also appropriate as it directly addresses the question and does not include any irrelevant or inappropriate content.\\n\\nBased on this analysis, the submission meets the criterion of being helpful, insightful, and appropriate.\\n\\nY', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('73805d52-d55d-490f-8055-ef15185e9d77'))}, source_run_id=None, target_run_id=None),\n","    EvaluationResult(key='COT Contextual Accuracy', score=0, value='INCORRECT', comment=\"The student's answer is incorrect. The context provided states that the novel layer type introduced in the Transformer architecture is 'multi-head' and 'attention'. The student's answer mentions 'self-attention' which is not mentioned in the context. Therefore, the student's answer is not factually accurate based on the context provided.\\nGRADE: INCORRECT\", correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('3ef39b9b-b85c-4192-a56a-7ee64f856dde'))}, source_run_id=None, target_run_id=None),\n","    EvaluationResult(key='must_mention', score=False, value=None, comment=None, correction=None, evaluator_info={}, source_run_id=None, target_run_id=None)],\n","   'execution_time': 1.429166,\n","   'run_id': 'df6f02ad-a721-4e0c-b5be-db6f67b08d44',\n","   'output': 'The novel layer type introduced in the Transformer architecture is the \"self-attention\" mechanism. This mechanism allows the model to weigh the importance of different input tokens when making predictions, enabling the model to capture long-range dependencies in the data more effectively.',\n","   'reference': {'must_mention': ['multi-head', 'attention']}},\n","  '55160640-1fb7-4770-b79d-ecb06f3cd5c2': {'input': {'question': 'What data type was created in the QLoRA paper?'},\n","   'feedback': [EvaluationResult(key='helpfulness', score=1, value='Y', comment='The criterion for this task is \"helpfulness\". \\n\\nThe submission provides a clear and direct answer to the question asked in the input. It not only names the data type created in the QLoRA paper, but also provides additional information about its characteristics. This additional information could be helpful for someone who is not familiar with the QLoRA paper or the \"4-bit NormalFloat (NF4)\" data type. \\n\\nTherefore, the submission is helpful, insightful, and appropriate. \\n\\nY', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('aede4004-4f49-4fc7-a05d-f4cc574f59e6'))}, source_run_id=None, target_run_id=None),\n","    EvaluationResult(key='COT Contextual Accuracy', score=1, value='CORRECT', comment='The student\\'s answer matches the context given. The student correctly identified the data type created in the QLoRA paper as \"4-bit NormalFloat (NF4)\". The additional information provided by the student about the data type being information theoretically optimal for normally distributed weights does not conflict with the context provided.\\nGRADE: CORRECT', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('48f3f9c0-b83c-494b-ae1d-e2c1f08cc737'))}, source_run_id=None, target_run_id=None),\n","    EvaluationResult(key='must_mention', score=True, value=None, comment=None, correction=None, evaluator_info={}, source_run_id=None, target_run_id=None)],\n","   'execution_time': 3.90371,\n","   'run_id': 'f3df4ab8-69b6-4e71-87d9-8d4a99a3fabe',\n","   'output': 'The QLoRA paper introduced a new data type called \"4-bit NormalFloat (NF4)\" which is information theoretically optimal for normally distributed weights.',\n","   'reference': {'must_mention': ['NF4', 'NormalFloat']}},\n","  'f1520357-a659-4173-ba15-276507c6c469': {'input': {'question': 'What is a Retrieval Augmented Generation system?'},\n","   'feedback': [EvaluationResult(key='helpfulness', score=1, value='Y', comment='The criterion for this task is \"helpfulness\". \\n\\nThe submission provides a detailed explanation of what a Retrieval Augmented Generation (RAG) system is. It explains the purpose of RAG, how it works, and its benefits. The submission also provides context by mentioning a specific example of a Large Language Model (LLM), ChatGPT. \\n\\nThe submission is insightful as it not only defines RAG but also explains the process of how it works. It mentions the use of user\\'s input to fetch relevant information from external sources and how this process enriches the context and content of the language model\\'s response.\\n\\nThe submission is appropriate as it directly answers the question asked in the input. It does not provide irrelevant or unnecessary information.\\n\\nBased on the above reasoning, the submission meets the criterion of being helpful, insightful, and appropriate.\\n\\nY', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('54ca3bf4-3f69-4a75-a788-76115ee0327f'))}, source_run_id=None, target_run_id=None),\n","    EvaluationResult(key='COT Contextual Accuracy', score=1, value='CORRECT', comment=\"The student's answer provides a detailed explanation of what a Retrieval Augmented Generation (RAG) system is. The student correctly identifies that RAG is an architecture that enhances the capabilities of a Large Language Model (LLM) by incorporating an information retrieval system. The student also correctly explains that this system allows for the retrieval of real-time, external data to enrich the context and content of the language model's response. The student's answer aligns with the context provided and does not contain any conflicting statements. \\nGRADE: CORRECT\", correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('dfebd24f-c691-4b85-b10b-ee428ec746fe'))}, source_run_id=None, target_run_id=None),\n","    EvaluationResult(key='must_mention', score=True, value=None, comment=None, correction=None, evaluator_info={}, source_run_id=None, target_run_id=None)],\n","   'execution_time': 6.05205,\n","   'run_id': '1c325e9b-6b46-4163-ad1b-bdc968184447',\n","   'output': \"Retrieval Augmented Generation (RAG) is an architecture that augments the capabilities of a Large Language Model (LLM) like ChatGPT by adding an information retrieval system that provides grounding data. Adding an information retrieval system gives you control over grounding data used by an LLM when it formulates a response. Retrieval Augmented Generation is a technique that enhances traditional language model responses by incorporating real-time, external data retrieval. It starts with the user's input, which is then used to fetch relevant information from various external sources. This process enriches the context and content of the language model's response. RAG is a framework for improving the quality of LLM-generated responses by grounding the model on external sources of knowledge to supplement the LLM's internal representation of information.\",\n","   'reference': {'must_mention': ['ground', 'context']}},\n","  '80e73986-e9ad-4a32-9800-77a01fc8355e': {'input': {'question': 'Who is recognized for pioneering work in deep learning and neural networks?'},\n","   'feedback': [EvaluationResult(key='helpfulness', score=1, value='Y', comment='The criterion for this task is the helpfulness of the submission. \\n\\nThe submission provides a clear and direct answer to the question asked in the input. It identifies Geoffrey Hinton, Yann LeCun, and Yoshua Bengio as the individuals recognized for pioneering work in deep learning and neural networks. \\n\\nIn addition to answering the question, the submission also provides additional information that could be helpful to the reader. It mentions that these individuals are often referred to as the \"Godfathers of AI\" and that they have made significant contributions to the field of artificial intelligence. \\n\\nThis additional information is relevant and could be useful to someone who is not familiar with these individuals or their work. \\n\\nTherefore, the submission is helpful, insightful, and appropriate, meeting the given criterion. \\n\\nY', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('16f8faef-3e77-47e5-9e92-7233e2c6b19e'))}, source_run_id=None, target_run_id=None),\n","    EvaluationResult(key='COT Contextual Accuracy', score=1, value='CORRECT', comment=\"The context provided only mentions Yoshua Bengio as the person recognized for pioneering work in deep learning and neural networks. However, the student's answer includes Geoffrey Hinton and Yann LeCun as well. While it is true that all three individuals have made significant contributions to the field of artificial intelligence, the student's answer contains more information than what is provided in the context. However, the additional information does not conflict with the context. Therefore, the student's answer is correct.\\nGRADE: CORRECT\", correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('79cfc30f-870c-4dc1-9e4d-869b81fb9f31'))}, source_run_id=None, target_run_id=None),\n","    EvaluationResult(key='must_mention', score=True, value=None, comment=None, correction=None, evaluator_info={}, source_run_id=None, target_run_id=None)],\n","   'execution_time': 2.304315,\n","   'run_id': '80bb9461-9407-4b56-8120-ceb979e55d76',\n","   'output': 'Geoffrey Hinton, Yann LeCun, and Yoshua Bengio are recognized for their pioneering work in deep learning and neural networks. They are often referred to as the \"Godfathers of AI\" and have made significant contributions to the field of artificial intelligence.',\n","   'reference': {'must_mention': ['Yoshua', 'Bengio']}},\n","  '7690e31f-baa6-4205-b7c2-de3d4e9e3eb3': {'input': {'question': 'Which benchmark dataset is commonly used for image classification tasks?'},\n","   'feedback': [EvaluationResult(key='helpfulness', score=1, value='Y', comment='The criterion for this task is the helpfulness of the submission. \\n\\nThe submission provides a clear and direct answer to the question, identifying the ImageNet dataset as the commonly used benchmark for image classification tasks. \\n\\nIn addition to providing the direct answer, the submission also provides additional information about the ImageNet dataset, such as its scale and the fact that it is used to train and evaluate deep learning models. This additional information is relevant and could be helpful to someone who is not familiar with the ImageNet dataset.\\n\\nTherefore, the submission is helpful, insightful, and appropriate, meeting the given criterion.\\n\\nY', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('fbda0008-04dc-4a1e-935b-54aabbd1007d'))}, source_run_id=None, target_run_id=None),\n","    EvaluationResult(key='COT Contextual Accuracy', score=1, value='CORRECT', comment=\"The student's answer correctly identifies ImageNet as the benchmark dataset commonly used for image classification tasks. The additional information provided about the size and use of the ImageNet dataset is accurate and does not conflict with the context provided. Therefore, the student's answer is correct.\\nGRADE: CORRECT\", correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('3e683f54-2bae-4b7e-b265-d92f40077407'))}, source_run_id=None, target_run_id=None),\n","    EvaluationResult(key='must_mention', score=True, value=None, comment=None, correction=None, evaluator_info={}, source_run_id=None, target_run_id=None)],\n","   'execution_time': 1.69967,\n","   'run_id': '3335e421-168f-45b4-8d25-96d09f3e9825',\n","   'output': 'The benchmark dataset commonly used for image classification tasks is the ImageNet dataset. It is a large-scale dataset with millions of labeled images across thousands of categories. Many deep learning models and algorithms are trained and evaluated on the ImageNet dataset to benchmark their performance in image classification tasks.',\n","   'reference': {'must_mention': ['ImageNet', 'dataset']}},\n","  'b3fb2632-f7cd-4b27-af2a-96981c944af7': {'input': {'question': 'What is the primary advantage of using dropout in neural networks?'},\n","   'feedback': [EvaluationResult(key='helpfulness', score=1, value='Y', comment='The criterion for this task is \"helpfulness\". \\n\\nThe submission provides a detailed explanation of the primary advantage of using dropout in neural networks. It explains the concept of overfitting and how dropout helps to prevent it. The submission also explains how dropout works by randomly dropping out a proportion of the neurons in the neural network during training. This explanation is helpful for understanding the concept.\\n\\nThe submission is insightful as it goes beyond just stating that dropout prevents overfitting. It explains why overfitting is a problem, how dropout prevents overfitting, and the benefits of using dropout, such as improving the model\\'s performance on unseen data and enhancing its generalization capabilities.\\n\\nThe submission is appropriate as it directly answers the question and provides relevant information.\\n\\nBased on the above reasoning, the submission meets the criterion of being helpful, insightful, and appropriate.\\n\\nY', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('5d416acf-983d-4206-b90c-d4e84e2ed6c7'))}, source_run_id=None, target_run_id=None),\n","    EvaluationResult(key='COT Contextual Accuracy', score=1, value='CORRECT', comment=\"The student's answer is in line with the context provided. The context states that the primary advantage of using dropout in neural networks is to 'reduce overfitting'. The student's answer elaborates on this point by explaining what overfitting is, how dropout helps prevent it, and the benefits of using dropout. The student's answer does not contradict the context in any way.\\nGRADE: CORRECT\", correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('c3caabb2-f1f7-4a46-ad5d-98f57e1028db'))}, source_run_id=None, target_run_id=None),\n","    EvaluationResult(key='must_mention', score=False, value=None, comment=None, correction=None, evaluator_info={}, source_run_id=None, target_run_id=None)],\n","   'execution_time': 3.402673,\n","   'run_id': '937f2de7-1942-47f1-a531-69591e042554',\n","   'output': \"The primary advantage of using dropout in neural networks is that it helps prevent overfitting. Overfitting occurs when a model learns the details and noise in the training data to the extent that it negatively impacts the model's performance on unseen data. Dropout helps prevent overfitting by randomly dropping out (setting to zero) a proportion of the neurons in the neural network during training. This prevents the network from relying too much on any individual neuron and encourages the network to learn more robust and generalizable features. As a result, dropout can improve the model's performance on unseen data and enhance its generalization capabilities.\",\n","   'reference': {'must_mention': ['reduce', 'overfitting']}}},\n"," 'aggregate_metrics': None}"]},"execution_count":25,"metadata":{},"output_type":"execute_result"}],"source":["client.run_on_dataset(\n","    dataset_name=dataset_name,\n","    llm_or_chain_factory=agent_chain,\n","    evaluation=eval_config,\n","    verbose=True,\n","    project_name=f\"RAG Pipeline - Evaluation - {uuid4().hex[0:8]}\",\n","    project_metadata={\"version\": \"1.0.0\"},\n",")"]}],"metadata":{"colab":{"provenance":[]},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":0}