notebooks/llm-agent-react/llm-agent-react-langchain.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b3c04f1a",
   "metadata": {},
   "source": [
    "# Create ReAct Agent using OpenVINO and LangChain\n",
    "\n",
    "LLM are limited to the knowledge on which they have been trained and the additional knowledge provided as context, as a result, if a useful piece of information is missing the provided knowledge, the model cannot “go around” and try to find it in other sources. This is the reason why we need to introduce the concept of Agents.\n",
    "\n",
    "The core idea of agents is to use a language model to choose a sequence of actions to take. In agents, a language model is used as a reasoning engine to determine which actions to take and in which order. Agents can be seen as applications powered by LLMs and integrated with a set of tools like search engines, databases, websites, and so on. Within an agent, the LLM is the reasoning engine that, based on the user input, is able to plan and execute a set of actions that are needed to fulfill the request.\n",
    "\n",
    "![agent](https://github.com/openvinotoolkit/openvino_notebooks/assets/91237924/22fa5396-8381-400f-a78f-97e25d57d807)\n",
    "\n",
    "[LangChain](https://python.langchain.com/docs/get_started/introduction) is a framework for developing applications powered by language models. LangChain comes with a number of built-in agents that are optimized for different use cases.\n",
    "\n",
    "This notebook explores how to create an AI Agent step by step using OpenVINO and LangChain.\n",
    "\n",
    "#### Table of contents:\n",
    "\n",
    "- [Prerequisites](#Prerequisites)\n",
    "- [Create tools](#Create-tools)\n",
    "- [Create prompt template](#Create-prompt-template)\n",
    "- [Create LLM](#Create-LLM)\n",
    "  - [Download model](#Select-model)\n",
    "  - [Select inference device for LLM](#Select-inference-device-for-LLM)\n",
    "- [Create agent](#Create-agent)\n",
    "- [Run the agent](#Run-agent)\n",
    "- [Interactive Demo](#Interactive-Demo)\n",
    "  - [Use built-in tool](#Use-built-in-tool)\n",
    "  - [Create customized tools](#Create-customized-tools)\n",
    "  - [Create AI agent demo with Gradio UI](#Create-AI-agent-demo-with-Gradio-UI)\n",
    "\n",
    "### Installation Instructions\n",
    "\n",
    "This is a self-contained example that relies solely on its own code.\n",
    "\n",
    "We recommend  running the notebook in a virtual environment. You only need a Jupyter server to start.\n",
    "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide).\n",
    "\n",
    "<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=5b5a4db0-7875-4bfb-bdbd-01698b5b1a77&file=notebooks/llm-agent-react/llm-agent-react-langchain.ipynb\" />\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f7bb0a67",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "47d43de7-9946-482d-84cb-222294c1cda8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from pathlib import Path\n",
    "\n",
    "r = requests.get(\n",
    "    url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\",\n",
    ")\n",
    "open(\"notebook_utils.py\", \"w\").write(r.text)\n",
    "\n",
    "if not Path(\"cmd_helper.py\").exists():\n",
    "    r = requests.get(url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/cmd_helper.py\")\n",
    "    open(\"cmd_helper.py\", \"w\", encoding=\"utf-8\").write(r.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b772846d-d559-4ad9-bb7a-d2164732964a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"GIT_CLONE_PROTECTION_ACTIVE\"] = \"false\"\n",
    "\n",
    "%pip install -Uq pip\n",
    "%pip uninstall -q -y optimum optimum-intel\n",
    "%pip install --pre -Uq \"openvino>=2024.5.0\" openvino-tokenizers[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly\n",
    "%pip install -q --extra-index-url https://download.pytorch.org/whl/cpu \"transformers>=4.38.1\" \"langchain>=0.2.3\" \"langchain-huggingface>=0.1.2\" \"langchain-community>=0.2.4\" \"Wikipedia\" \\\n",
    "\"torch>=2.1\" \\\n",
    "\"datasets\" \\\n",
    "\"accelerate\" \\\n",
    "\"pydantic<2.10.0\" \\\n",
    "\"gradio>=4.19\" \\\n",
    "\"huggingface-hub>=0.26.5\"\n",
    "%pip install -q \"git+https://github.com/huggingface/optimum-intel.git\" \\\n",
    "\"git+https://github.com/openvinotoolkit/nncf.git\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c722d565",
   "metadata": {},
   "source": [
    "## Create a tools\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "First, we need to create some tools to call. In this example, we will create 3 custom functions to do basic calculation. For [more information](https://python.langchain.com/docs/modules/tools/) on creating custom tools.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e8bfe609-1823-4df7-9a68-f210a58a0d38",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.tools import tool\n",
    "\n",
    "\n",
    "@tool\n",
    "def multiply(first_int: int, second_int: int) -> int:\n",
    "    \"\"\"Multiply two integers together.\"\"\"\n",
    "    return first_int * second_int\n",
    "\n",
    "\n",
    "@tool\n",
    "def add(first_int: int, second_int: int) -> int:\n",
    "    \"Add two integers.\"\n",
    "    return first_int + second_int\n",
    "\n",
    "\n",
    "@tool\n",
    "def exponentiate(base: int, exponent: int) -> int:\n",
    "    \"Exponentiate the base to the exponent power.\"\n",
    "    return base**exponent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5ea4ce13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name of `multiply` tool: multiply\n",
      "description of `multiply` tool: Multiply two integers together.\n"
     ]
    }
   ],
   "source": [
    "print(f\"name of `multiply` tool: {multiply.name}\")\n",
    "print(f\"description of `multiply` tool: {multiply.description}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6b476967",
   "metadata": {},
   "source": [
    "Tools are interfaces that an agent, chain, or LLM can use to interact with the world. They combine a few things:\n",
    "\n",
    "1. The name of the tool\n",
    "2. A description of what the tool is\n",
    "3. JSON schema of what the inputs to the tool are\n",
    "4. The function to call\n",
    "5. Whether the result of a tool should be returned directly to the user\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d5f1ac30",
   "metadata": {},
   "source": [
    "Now that we have created all of them, and we can create a list of tools that we will use downstream.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a5c41830",
   "metadata": {},
   "outputs": [],
   "source": [
    "tools = [multiply, add, exponentiate]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dba80270",
   "metadata": {},
   "source": [
    "## Create prompt template\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "A prompt for a language model is a set of instructions or input provided by a user to guide the model's response, helping it understand the context and generate relevant and coherent language-based output, such as answering questions, completing sentences, or engaging in a conversation.\n",
    "\n",
    "Different agents have different prompting styles for reasoning. In this example, we will use [ReAct agent](https://react-lm.github.io/) with its typical prompt template. For a full list of built-in agents see [agent types](https://python.langchain.com/docs/modules/agents/agent_types/).\n",
    "\n",
    "![react](https://github.com/openvinotoolkit/openvino_notebooks/assets/91237924/a83bdf7f-bb9d-4b1f-9a0a-3fe4a76ba1ae)\n",
    "\n",
    "A ReAct prompt consists of few-shot task-solving trajectories, with human-written text reasoning traces and actions, as well as environment observations in response to actions. ReAct prompting is intuitive and flexible to design, and achieves state-of-the-art few-shot performances across a variety of tasks, from question answering to online shopping!\n",
    "\n",
    "In an prompt template for agent, `input` is user's query and `agent_scratchpad` should be a sequence of messages that contains the previous agent tool invocations and the corresponding tool outputs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "107f040a-e859-475c-9422-f980ac593fcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "PREFIX = \"\"\"Respond to the human as helpfully and accurately as possible. You have access to the following tools:\"\"\"\n",
    "\n",
    "FORMAT_INSTRUCTIONS = \"\"\"Use a json blob to specify a tool by providing an action key (tool name) and an action_input key (tool input).\n",
    "\n",
    "Valid \"action\" values: \"Final Answer\" or {tool_names}\n",
    "\n",
    "Provide only ONE action per $JSON_BLOB, as shown:\n",
    "\n",
    "```\n",
    "{{{{\n",
    "  \"action\": $TOOL_NAME,\n",
    "  \"action_input\": $INPUT\n",
    "}}}}\n",
    "```\n",
    "\n",
    "Follow this format:\n",
    "\n",
    "Question: input question to answer\n",
    "Thought: consider previous and subsequent steps\n",
    "Action:\n",
    "```\n",
    "$JSON_BLOB\n",
    "```\n",
    "Observation: action result\n",
    "... (repeat Thought/Action/Observation N times)\n",
    "Thought: I know what to respond\n",
    "Action:\n",
    "```\n",
    "{{{{\n",
    "  \"action\": \"Final Answer\",\n",
    "  \"action_input\": \"Final response to human\"\n",
    "}}}}\n",
    "```\"\"\"\n",
    "\n",
    "SUFFIX = \"\"\"Begin! Reminder to ALWAYS respond with a valid json blob of a single action. Use tools if necessary. Respond directly if appropriate. Format is Action:```$JSON_BLOB```then Observation:.\n",
    "Thought:\"\"\"\n",
    "\n",
    "HUMAN_MESSAGE_TEMPLATE = \"{input}\\n\\n{agent_scratchpad}\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "259e1f0d",
   "metadata": {},
   "source": [
    "## Create LLM\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Large Language Models (LLMs) are a core component of LangChain. LangChain does not serve its own LLMs, but rather provides a standard interface for interacting with many different LLMs. In this example, we select following models as LLM in agent pipeline.\n",
    "\n",
    "\n",
    "* **qwen2.5-3b-instruct/qwen2.5-7b-instruct/qwen2.5-14b-instruct** - Qwen2.5 is the latest series of Qwen large language models. Comparing with Qwen2, Qwen2.5 series brings significant improvements in coding, mathematics and general knowledge skills. Additionally, it brings long-context and multiple languages support including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more. \n",
    "For more details, please refer to [model_card](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct), [blog](https://qwenlm.github.io/blog/qwen2.5/), [GitHub](https://github.com/QwenLM/Qwen2.5), and [Documentation](https://qwen.readthedocs.io/en/latest/).\n",
    "* **llama-3.1-8b-instruct** - The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks. More details about model can be found in [Meta blog post](https://ai.meta.com/blog/meta-llama-3-1/), [model website](https://llama.meta.com) and [model card](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct).\n",
    ">**Note**: run model with demo, you will need to accept license agreement. \n",
    ">You must be a registered user in 🤗 Hugging Face Hub. Please visit [HuggingFace model card](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct), carefully read terms of usage and click accept button.  You will need to use an access token for the code below to run. For more information on access tokens, refer to [this section of the documentation](https://huggingface.co/docs/hub/security-tokens).\n",
    ">You can login on Hugging Face Hub in notebook environment, using following code:\n",
    " \n",
    "```python\n",
    "    ## login to huggingfacehub to get access to pretrained model \n",
    "\n",
    "    from huggingface_hub import notebook_login, whoami\n",
    "\n",
    "    try:\n",
    "        whoami()\n",
    "        print('Authorization token already provided')\n",
    "    except OSError:\n",
    "        notebook_login()\n",
    "```\n",
    "\n",
    "### Download model\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "To run LLM locally, we have to download the model in the first step. It is possible to [export your model](https://github.com/huggingface/optimum-intel?tab=readme-ov-file#export) to the OpenVINO IR format with the CLI, and load the model from local folder.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "981df8fe-cfcf-455a-919e-dda36f3b5dfb",
   "metadata": {
    "test_replace": {
     "Qwen/Qwen2.5-7B-Instruct": "Qwen/Qwen2.5-3B-Instruct"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "29a31455ce2d48c4b6583eea7af618da",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Model:', options=('Qwen/Qwen2.5-7B-Instruct', 'Qwen/Qwen2.5-3B-Instruct', 'Qwen/qwen2.5-…"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import ipywidgets as widgets\n",
    "\n",
    "llm_model_ids = [\"Qwen/Qwen2.5-7B-Instruct\", \"Qwen/Qwen2.5-3B-Instruct\", \"Qwen/qwen2.5-14b-instruct\", \"meta-llama/Meta-Llama-3.1-8B-Instruct\"]\n",
    "\n",
    "llm_model_id = widgets.Dropdown(\n",
    "    options=llm_model_ids,\n",
    "    value=llm_model_ids[0],\n",
    "    description=\"Model:\",\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "llm_model_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "6d90a485-2823-47c3-958f-6d2d09a06ecf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Export command:**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "`optimum-cli export openvino --model Qwen/Qwen2.5-7B-Instruct Qwen2.5-7B-Instruct --task text-generation-with-past --weight-format int4 --group-size 128 --ratio 1.0 --sym`"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading shards: 100%|██████████| 4/4 [06:03<00:00, 90.89s/it]\n",
      "Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.20it/s]\n",
      "We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)\n",
      "/home2/ethan/intel/openvino_notebooks/openvino_venv/lib/python3.10/site-packages/optimum/exporters/openvino/model_patcher.py:506: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if sequence_length != 1:\n",
      "/home2/ethan/intel/openvino_notebooks/openvino_venv/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py:165: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if seq_len > self.max_seq_len_cached:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:Statistics of the bitwidth distribution:\n",
      "┍━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑\n",
      "│ Weight compression mode   │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │\n",
      "┝━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥\n",
      "│ int8_asym                 │ 14% (2 / 198)               │ 0% (0 / 196)                           │\n",
      "├───────────────────────────┼─────────────────────────────┼────────────────────────────────────────┤\n",
      "│ int4_sym                  │ 86% (196 / 198)             │ 100% (196 / 196)                       │\n",
      "┕━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙\n",
      "\u001b[2KApplying Weight Compression \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m • \u001b[38;2;0;104;181m0:04:25\u001b[0m • \u001b[38;2;0;104;181m0:00:00\u001b[0m;0;104;181m0:00:01\u001b[0m181m0:00:11\u001b[0m\n",
      "\u001b[?25h"
     ]
    }
   ],
   "source": [
    "from cmd_helper import optimum_cli\n",
    "\n",
    "llm_model_path = llm_model_id.value.split(\"/\")[-1]\n",
    "repo_name = llm_model_id.value.split(\"/\")[0]\n",
    "\n",
    "if not Path(llm_model_path).exists():\n",
    "    optimum_cli(\n",
    "        llm_model_id.value,\n",
    "        llm_model_path,\n",
    "        additional_args={\"task\": \"text-generation-with-past\", \"weight-format\": \"int4\", \"group-size\": \"128\", \"ratio\": \"1.0\", \"sym\": \"\"},\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6cfdbbae",
   "metadata": {},
   "source": [
    "### Select inference device for LLM\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "a1ea3bdb-f97c-4374-880a-2b62abb5baaa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d1af506db43e406dabc6bbb8d00d811a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Device:', options=('CPU', 'GPU', 'AUTO'), value='CPU')"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from notebook_utils import device_widget\n",
    "\n",
    "device = device_widget(\"CPU\", exclude=[\"NPU\"])\n",
    "\n",
    "device"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "77244c52",
   "metadata": {},
   "source": [
    "OpenVINO models can be run locally through the `HuggingFacePipeline` class in LangChain. To deploy a model with OpenVINO, you can specify the `backend=\"openvino\"` parameter to trigger OpenVINO as backend inference framework. For [more information](https://python.langchain.com/docs/integrations/llms/openvino/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "abfaab28-fd5b-46cd-88ad-b60ea5a3cdd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_huggingface import HuggingFacePipeline\n",
    "from transformers.generation.stopping_criteria import StoppingCriteriaList, StoppingCriteria\n",
    "\n",
    "import openvino.properties as props\n",
    "import openvino.properties.hint as hints\n",
    "import openvino.properties.streams as streams\n",
    "\n",
    "\n",
    "class StopSequenceCriteria(StoppingCriteria):\n",
    "    \"\"\"\n",
    "    This class can be used to stop generation whenever a sequence of tokens is encountered.\n",
    "\n",
    "    Args:\n",
    "        stop_sequences (`str` or `List[str]`):\n",
    "            The sequence (or list of sequences) on which to stop execution.\n",
    "        tokenizer:\n",
    "            The tokenizer used to decode the model outputs.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, stop_sequences, tokenizer):\n",
    "        if isinstance(stop_sequences, str):\n",
    "            stop_sequences = [stop_sequences]\n",
    "        self.stop_sequences = stop_sequences\n",
    "        self.tokenizer = tokenizer\n",
    "\n",
    "    def __call__(self, input_ids, scores, **kwargs) -> bool:\n",
    "        decoded_output = self.tokenizer.decode(input_ids.tolist()[0])\n",
    "        return any(decoded_output.endswith(stop_sequence) for stop_sequence in self.stop_sequences)\n",
    "\n",
    "\n",
    "ov_config = {hints.performance_mode(): hints.PerformanceMode.LATENCY, streams.num(): \"1\", props.cache_dir(): \"\"}\n",
    "stop_tokens = [\"Observation:\", \"Observation:\\n\"]\n",
    "\n",
    "ov_llm = HuggingFacePipeline.from_model_id(\n",
    "    model_id=llm_model_path,\n",
    "    task=\"text-generation\",\n",
    "    backend=\"openvino\",\n",
    "    model_kwargs={\n",
    "        \"device\": device.value,\n",
    "        \"ov_config\": ov_config,\n",
    "        \"trust_remote_code\": True,\n",
    "    },\n",
    "    pipeline_kwargs={\"max_new_tokens\": 2048},\n",
    ")\n",
    "\n",
    "tokenizer = ov_llm.pipeline.tokenizer\n",
    "ov_llm.pipeline._forward_params[\"stopping_criteria\"] = StoppingCriteriaList([StopSequenceCriteria(stop_tokens, tokenizer)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "f5cae381-417f-4984-8873-9a6ad46c4eed",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_huggingface import ChatHuggingFace\n",
    "\n",
    "ov_chat = ChatHuggingFace(llm=ov_llm, verbose=True)\n",
    "ov_chat = ov_chat.bind(skip_prompt=True, stop=[\"Observation:\", \"Observation:\\n\"])\n",
    "\n",
    "if llm_model_id.value == \"meta-llama/Meta-Llama-3.1-8B-Instruct\":\n",
    "    ov_chat.llm.pipeline.tokenizer.pad_token_id = ov_chat.llm.pipeline.tokenizer.eos_token_id"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d70905e2",
   "metadata": {},
   "source": [
    "You can get additional inference speed improvement with [Dynamic Quantization of activations and KV-cache quantization on CPU](https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide/llm-inference-hf.html#enabling-openvino-runtime-optimizations). These options can be enabled with `ov_config` as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "17c2d285-e8bb-4da1-baa6-682a7851d333",
   "metadata": {},
   "outputs": [],
   "source": [
    "ov_config = {\n",
    "    \"KV_CACHE_PRECISION\": \"u8\",\n",
    "    \"DYNAMIC_QUANTIZATION_GROUP_SIZE\": \"32\",\n",
    "    hints.performance_mode(): hints.PerformanceMode.LATENCY,\n",
    "    streams.num(): \"1\",\n",
    "    props.cache_dir(): \"\",\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "52a9a190",
   "metadata": {},
   "source": [
    "## Create agent\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Now that we have defined the tools, prompt template and LLM, we can create the agent_executor.\n",
    "\n",
    "The agent executor is the runtime for an agent. This is what actually calls the agent, executes the actions it chooses, passes the action outputs back to the agent, and repeats.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "4799540b-eee0-491f-a5b6-5bae68c22af9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.agents import AgentExecutor, StructuredChatAgent\n",
    "\n",
    "agent = StructuredChatAgent.from_llm_and_tools(\n",
    "    ov_chat,\n",
    "    tools,\n",
    "    prefix=PREFIX,\n",
    "    suffix=SUFFIX,\n",
    "    human_message_template=HUMAN_MESSAGE_TEMPLATE,\n",
    "    format_instructions=FORMAT_INSTRUCTIONS,\n",
    ")\n",
    "agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "038e76d1",
   "metadata": {},
   "source": [
    "## Run the agent\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "We can now run the agent with a math query. Before getting the final answer, a agent executor will also produce intermediate steps of reasoning and actions. The format of these messages will follow your prompt template.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "eebc8f67-8107-4a6b-90bf-ea9256c64ee5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
      "\u001b[32;1m\u001b[1;3mThought: First, I need to calculate 3 raised to the fifth power. Then, I will find the sum of twelve and three. After that, I will multiply the first result by the second result, and finally, I will square the entire result.\n",
      "\n",
      "Action:\n",
      "```\n",
      "{\n",
      "  \"action\": \"exponentiate\",\n",
      "  \"action_input\": {\n",
      "    \"base\": 3,\n",
      "    \"exponent\": 5\n",
      "  }\n",
      "}\n",
      "```\n",
      "Observation:\u001b[0m\n",
      "Observation: \u001b[38;5;200m\u001b[1;3m243\u001b[0m\n",
      "Thought:\u001b[32;1m\u001b[1;3mNext, I need to find the sum of twelve and three.\n",
      "\n",
      "Action:\n",
      "```\n",
      "{\n",
      "  \"action\": \"add\",\n",
      "  \"action_input\": {\n",
      "    \"first_int\": 12,\n",
      "    \"second_int\": 3\n",
      "  }\n",
      "}\n",
      "```\n",
      "Observation:\u001b[0m\n",
      "Observation: \u001b[33;1m\u001b[1;3m15\u001b[0m\n",
      "Thought:\u001b[32;1m\u001b[1;3mNow, I will multiply the result of \\(3^5\\) (which is 243) by the sum of twelve and three (which is 15). After that, I will square the entire result.\n",
      "\n",
      "Action:\n",
      "```\n",
      "{\n",
      "  \"action\": \"multiply\",\n",
      "  \"action_input\": {\n",
      "    \"first_int\": 243,\n",
      "    \"second_int\": 15\n",
      "  }\n",
      "}\n",
      "```\n",
      "Observation:\u001b[0m\n",
      "Observation: \u001b[36;1m\u001b[1;3m3645\u001b[0m\n",
      "Thought:\u001b[32;1m\u001b[1;3mThought: Now I need to square the result of the multiplication, which is 3645.\n",
      "\n",
      "Action:\n",
      "```\n",
      "{\n",
      "  \"action\": \"exponentiate\",\n",
      "  \"action_input\": {\n",
      "    \"base\": 3645,\n",
      "    \"exponent\": 2\n",
      "  }\n",
      "}\n",
      "```\n",
      "Observation:\u001b[0m\n",
      "Observation: \u001b[38;5;200m\u001b[1;3m13286025\u001b[0m\n",
      "Thought:\u001b[32;1m\u001b[1;3mThought: I know what to respond\n",
      "\n",
      "Action:\n",
      "```\n",
      "{\n",
      "  \"action\": \"Final Answer\",\n",
      "  \"action_input\": \"The final result is 13286025.\"\n",
      "}\n",
      "```\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'input': 'Take 3 to the fifth power and multiply that by the sum of twelve and three, then square the whole result',\n",
       " 'output': 'The final result is 13286025.'}"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agent_executor.invoke({\"input\": \"Take 3 to the fifth power and multiply that by the sum of twelve and three, then square the whole result\"})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "688ced57",
   "metadata": {},
   "source": [
    "## Interactive Demo\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Let's create a interactive agent using [Gradio](https://www.gradio.app/).\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ce623516",
   "metadata": {},
   "source": [
    "### Use built-in tools\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "LangChain has provided a list of all [built-in tools](https://python.langchain.com/docs/integrations/tools/). In this example, we will use `Wikipedia` python package to query key words generated by agent.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "1f64b67c-1259-4fe6-bfc3-af317bfe04f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.tools import WikipediaQueryRun\n",
    "from langchain_community.utilities import WikipediaAPIWrapper\n",
    "from langchain_core.callbacks import CallbackManagerForToolRun\n",
    "from typing import Optional\n",
    "\n",
    "from pydantic import BaseModel, Field\n",
    "\n",
    "\n",
    "class WikipediaQueryRunWrapper(WikipediaQueryRun):\n",
    "    def _run(\n",
    "        self,\n",
    "        text: str,\n",
    "        run_manager: Optional[CallbackManagerForToolRun] = None,\n",
    "    ) -> str:\n",
    "        \"\"\"Use the Wikipedia tool.\"\"\"\n",
    "        return self.api_wrapper.run(text)\n",
    "\n",
    "\n",
    "api_wrapper = WikipediaAPIWrapper(top_k_results=2, doc_content_chars_max=1000)\n",
    "\n",
    "\n",
    "class WikiInputs(BaseModel):\n",
    "    \"\"\"inputs to the wikipedia tool.\"\"\"\n",
    "\n",
    "    text: str = Field(description=\"query to look up on wikipedia.\")\n",
    "\n",
    "\n",
    "wikipedia = WikipediaQueryRunWrapper(\n",
    "    description=\"A wrapper around Wikipedia. Useful for when you need to answer general questions about people, places, companies, facts, historical events, or other subjects. Input should be a search query.\",\n",
    "    args_schema=WikiInputs,\n",
    "    api_wrapper=api_wrapper,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "ea649d2f-ab4b-4228-8f32-75b01a98d9a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Page: OpenVINO\\nSummary: OpenVINO is an open-source software toolkit for optimizing and deploying deep learning models. It enables programmers to develop scalable and efficient AI solutions with relatively few lines of code. It supports several popular model formats and categories, such as large language models, computer vision, and generative AI.\\nActively developed by Intel, it prioritizes high-performance inference on Intel hardware but also supports ARM/ARM64 processors and encourages contributors to add new devices to the portfolio.\\nBased in C++, it offers the following APIs: C/C++, Python, and Node.js (an early preview).\\nOpenVINO is cross-platform and free for use under Apache License 2.0.\\n\\nPage: Audacity (audio editor)\\nSummary: Audacity is a free and open-source digital audio editor and recording application software, available for Windows, macOS, Linux, and other Unix-like operating systems. \\nAs of December 6, 2022, Audacity is the most popular download at FossHub, with over 114.'"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wikipedia.invoke({\"text\": \"OpenVINO\"})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5437e2dc-38cc-4052-bf7c-8d96b0852294",
   "metadata": {},
   "source": [
    "### Create customized tools\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "In this examples, we will create 2 customized tools for `image generation` and `weather qurey`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "318acd7b-d022-4bdf-b65c-b4fdcc015048",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{image_url: \"https://image.pollinations.ai/prompt/a%20cat\"}'"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import urllib.parse\n",
    "import json5\n",
    "\n",
    "\n",
    "@tool\n",
    "def painting(prompt: str) -> str:\n",
    "    \"\"\"\n",
    "    AI painting (image generation) service, input text description, and return the image URL drawn based on text information.\n",
    "    \"\"\"\n",
    "    prompt = urllib.parse.quote(prompt)\n",
    "    return json5.dumps({\"image_url\": f\"https://image.pollinations.ai/prompt/{prompt}\"}, ensure_ascii=False)\n",
    "\n",
    "\n",
    "painting.invoke({\"prompt\": \"a cat\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "96e9f671-1680-402c-a799-0e45b731f63e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"{'current_condition': {'temp_C': '0', 'FeelsLikeC': '-4', 'humidity': '86', 'weatherDesc': [{'value': 'Clear'}], 'observation_time': '12:16 AM'}}\""
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@tool\n",
    "def weather(\n",
    "    city_name: str,\n",
    ") -> str:\n",
    "    \"\"\"\n",
    "    Get the current weather for `city_name`\n",
    "    \"\"\"\n",
    "\n",
    "    if not isinstance(city_name, str):\n",
    "        raise TypeError(\"City name must be a string\")\n",
    "\n",
    "    key_selection = {\n",
    "        \"current_condition\": [\n",
    "            \"temp_C\",\n",
    "            \"FeelsLikeC\",\n",
    "            \"humidity\",\n",
    "            \"weatherDesc\",\n",
    "            \"observation_time\",\n",
    "        ],\n",
    "    }\n",
    "    import requests\n",
    "\n",
    "    resp = requests.get(f\"https://wttr.in/{city_name}?format=j1\")\n",
    "    resp.raise_for_status()\n",
    "    resp = resp.json()\n",
    "    ret = {k: {_v: resp[k][0][_v] for _v in v} for k, v in key_selection.items()}\n",
    "\n",
    "    return str(ret)\n",
    "\n",
    "\n",
    "weather.invoke({\"city_name\": \"London\"})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "266d55d3-96c6-4275-8e05-d2ab645b3d32",
   "metadata": {},
   "source": [
    "### Create AI agent demo with Gradio UI\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "ce703b35-6396-409c-bf4f-d4bc4dd584b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "tools = [wikipedia, painting, weather]\n",
    "\n",
    "agent = StructuredChatAgent.from_llm_and_tools(\n",
    "    ov_chat,\n",
    "    tools,\n",
    "    prefix=PREFIX,\n",
    "    suffix=SUFFIX,\n",
    "    human_message_template=HUMAN_MESSAGE_TEMPLATE,\n",
    "    format_instructions=FORMAT_INSTRUCTIONS,\n",
    ")\n",
    "agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "d6b24449",
   "metadata": {},
   "outputs": [],
   "source": [
    "def partial_text_processor(partial_text, new_text):\n",
    "    \"\"\"\n",
    "    helper for updating partially generated answer, used by default\n",
    "\n",
    "    Params:\n",
    "      partial_text: text buffer for storing previosly generated text\n",
    "      new_text: text update for the current step\n",
    "    Returns:\n",
    "      updated text string\n",
    "\n",
    "    \"\"\"\n",
    "    partial_text += new_text\n",
    "    return partial_text\n",
    "\n",
    "\n",
    "def run_chatbot(history):\n",
    "    \"\"\"\n",
    "    callback function for running chatbot on submit button click\n",
    "\n",
    "    Params:\n",
    "      history: conversation history\n",
    "\n",
    "    \"\"\"\n",
    "    partial_text = \"\"\n",
    "\n",
    "    for new_text in agent_executor.stream(\n",
    "        {\"input\": history[-1][0]},\n",
    "    ):\n",
    "        if \"output\" in new_text.keys():\n",
    "            partial_text = partial_text_processor(partial_text, new_text[\"output\"])\n",
    "            history[-1][1] = partial_text\n",
    "            yield history\n",
    "\n",
    "\n",
    "def request_cancel():\n",
    "    ov_chat.llm.pipeline.model.request.cancel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff5a60fb-e13e-4eaf-a2f3-999e4e3109bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not Path(\"gradio_helper.py\").exists():\n",
    "    r = requests.get(url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/llm-agent-react/gradio_helper.py\")\n",
    "    open(\"gradio_helper.py\", \"w\").write(r.text)\n",
    "\n",
    "from gradio_helper import make_demo\n",
    "\n",
    "examples = [\n",
    "    [\"Based on current weather in London, show me a picture of Big Ben through its URL\"],\n",
    "    [\"What is OpenVINO ?\"],\n",
    "    [\"Create an image of pink cat and return its URL\"],\n",
    "    [\"How many people live in Canada ?\"],\n",
    "    [\"What is the weather like in New York now ?\"],\n",
    "]\n",
    "\n",
    "demo = make_demo(run_fn=run_chatbot, stop_fn=request_cancel, examples=examples)\n",
    "\n",
    "try:\n",
    "    demo.launch()\n",
    "except Exception:\n",
    "    demo.launch(share=True)\n",
    "# If you are launching remotely, specify server_name and server_port\n",
    "# EXAMPLE: `demo.launch(server_name='your server name', server_port='server port in int')`\n",
    "# To learn more please refer to the Gradio docs: https://gradio.app/docs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "815bc496-8bd9-45b7-8650-b5e0741490a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# please uncomment and run this cell for stopping gradio interface\n",
    "# demo.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "openvino_notebooks": {
   "imageUrl": "https://github.com/openvinotoolkit/openvino_notebooks/assets/91237924/2abb2389-e612-4599-82c6-64cdac259120",
   "tags": {
    "categories": [
     "Model Demos",
     "AI Trends"
    ],
    "libraries": [],
    "other": [
     "LLM"
    ],
    "tasks": [
     "Text Generation"
    ]
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}