4-Evaluating LLMs/4_2_tracing_medical_agent.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/peremartra/Large-Language-Model-Notebooks-Course/blob/main/4-Evaluating%20LLMs/4_2_tracing_medical_agent.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div>\n",
        "    <h1>Large Language Models Projects</a></h1>\n",
        "    <h3>Apply and Implement Strategies for Large Language Models</h3>\n",
        "    <h2>4.2-Tracing and Evaluating LLMs with LangSmith.\n",
        "  </h2>\n",
        "    <h3>Tracing a Medical Agent with LangSmith</h3>\n",
        "</div>\n",
        "\n",
        "by [Pere Martra](https://www.linkedin.com/in/pere-martra/)\n",
        "______________\n",
        "Models: OpenAI\n",
        "\n",
        "Colab Environment: CPU-High RAM.\n",
        "\n",
        "Keys:\n",
        "* Tracing LLMs\n",
        "* LangSmith\n",
        "* Agents.\n",
        "______________\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "f3cvnSVemTSU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this notebook, we will explore how to trace the different calls that occur in a LangChain Agent using LangSmith. We will use a familiar agent, employed in the LangChain section of the course, where a RAG system with medical information was created.\n",
        "\n",
        "So, not only will we observe the traces of the agent, but we will also examine the traces of the retriever. Additionally, we'll inspect the query sent to the vectorial database and the returned results.\n",
        "__________"
      ],
      "metadata": {
        "id": "knN-MUbEIMI8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Installing libraries & Loading Dataset"
      ],
      "metadata": {
        "id": "KNW-LniD2I2o"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pva9ehKXUpU2",
        "outputId": "7f8000eb-76f4-4e18-9ac3-1102362091c6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.3/2.3 MB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.3/49.3 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "!pip install -q langchain==0.3.0\n",
        "!pip install -q langchain-openai==0.2.0\n",
        "!pip install -q langchainhub==0.1.21\n",
        "!pip install -q datasets==3.0.0\n",
        "!pip install -q chromadb==0.5.5\n",
        "!pip install -q langchain-community==0.3.0"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will download the dataset from the Hugging Face datasets library. It's a dataset with information about diseases."
      ],
      "metadata": {
        "id": "KEwdHiGFkK_S"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "laSDMjqQXuj-"
      },
      "outputs": [],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "data = load_dataset(\"keivalya/MedQuad-MedicalQnADataset\", split='train')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        },
        "id": "JnWZTcJiXzor",
        "outputId": "36b061d7-8cd9-4db9-8067-a45d88a66a75"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "             qtype                                           Question  \\\n",
              "0   susceptibility  Who is at risk for Lymphocytic Choriomeningiti...   \n",
              "1         symptoms  What are the symptoms of Lymphocytic Choriomen...   \n",
              "2   susceptibility  Who is at risk for Lymphocytic Choriomeningiti...   \n",
              "3  exams and tests  How to diagnose Lymphocytic Choriomeningitis (...   \n",
              "4        treatment  What are the treatments for Lymphocytic Chorio...   \n",
              "5       prevention  How to prevent Lymphocytic Choriomeningitis (L...   \n",
              "6      information          What is (are) Parasites - Cysticercosis ?   \n",
              "7   susceptibility    Who is at risk for Parasites - Cysticercosis? ?   \n",
              "8  exams and tests        How to diagnose Parasites - Cysticercosis ?   \n",
              "9        treatment  What are the treatments for Parasites - Cystic...   \n",
              "\n",
              "                                              Answer  \n",
              "0  LCMV infections can occur after exposure to fr...  \n",
              "1  LCMV is most commonly recognized as causing ne...  \n",
              "2  Individuals of all ages who come into contact ...  \n",
              "3  During the first phase of the disease, the mos...  \n",
              "4  Aseptic meningitis, encephalitis, or meningoen...  \n",
              "5  LCMV infection can be prevented by avoiding co...  \n",
              "6  Cysticercosis is an infection caused by the la...  \n",
              "7  Cysticercosis is an infection caused by the la...  \n",
              "8  If you think that you may have cysticercosis, ...  \n",
              "9  Some people with cysticercosis do not need to ...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-765bd9f8-0614-42c2-8e69-e343050404ed\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>qtype</th>\n",
              "      <th>Question</th>\n",
              "      <th>Answer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>susceptibility</td>\n",
              "      <td>Who is at risk for Lymphocytic Choriomeningiti...</td>\n",
              "      <td>LCMV infections can occur after exposure to fr...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>symptoms</td>\n",
              "      <td>What are the symptoms of Lymphocytic Choriomen...</td>\n",
              "      <td>LCMV is most commonly recognized as causing ne...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>susceptibility</td>\n",
              "      <td>Who is at risk for Lymphocytic Choriomeningiti...</td>\n",
              "      <td>Individuals of all ages who come into contact ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>exams and tests</td>\n",
              "      <td>How to diagnose Lymphocytic Choriomeningitis (...</td>\n",
              "      <td>During the first phase of the disease, the mos...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>treatment</td>\n",
              "      <td>What are the treatments for Lymphocytic Chorio...</td>\n",
              "      <td>Aseptic meningitis, encephalitis, or meningoen...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>prevention</td>\n",
              "      <td>How to prevent Lymphocytic Choriomeningitis (L...</td>\n",
              "      <td>LCMV infection can be prevented by avoiding co...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>information</td>\n",
              "      <td>What is (are) Parasites - Cysticercosis ?</td>\n",
              "      <td>Cysticercosis is an infection caused by the la...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>susceptibility</td>\n",
              "      <td>Who is at risk for Parasites - Cysticercosis? ?</td>\n",
              "      <td>Cysticercosis is an infection caused by the la...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>exams and tests</td>\n",
              "      <td>How to diagnose Parasites - Cysticercosis ?</td>\n",
              "      <td>If you think that you may have cysticercosis, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>treatment</td>\n",
              "      <td>What are the treatments for Parasites - Cystic...</td>\n",
              "      <td>Some people with cysticercosis do not need to ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-765bd9f8-0614-42c2-8e69-e343050404ed')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-765bd9f8-0614-42c2-8e69-e343050404ed button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-765bd9f8-0614-42c2-8e69-e343050404ed');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-1fc1f0f0-8f41-4b3c-a38a-87348555bece\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-1fc1f0f0-8f41-4b3c-a38a-87348555bece')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-1fc1f0f0-8f41-4b3c-a38a-87348555bece button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "data",
              "summary": "{\n  \"name\": \"data\",\n  \"rows\": 16407,\n  \"fields\": [\n    {\n      \"column\": \"qtype\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 16,\n        \"samples\": [\n          \"susceptibility\",\n          \"symptoms\",\n          \"information\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Question\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 14979,\n        \"samples\": [\n          \"What are the symptoms of Danon disease ?\",\n          \"What is (are) Dowling-Degos disease ?\",\n          \"What are the genetic changes related to Pearson marrow-pancreas syndrome ?\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Answer\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 15817,\n        \"samples\": [\n          \"These resources address the diagnosis or management of glycogen storage disease type III:  - Gene Review: Gene Review: Glycogen Storage Disease Type III  - Genetic Testing Registry: Glycogen storage disease type III   These resources from MedlinePlus offer information about the diagnosis and management of various health conditions:  - Diagnostic Tests  - Drug Therapy  - Surgery and Rehabilitation  - Genetic Counseling   - Palliative Care\",\n          \"Diagnostic Challenges\\n  \\nFor doctors, diagnosing chronic fatigue syndrome (CFS) can be complicated by a number of factors:\\n  \\n   - There's no lab test or biomarker for CFS.\\n   - Fatigue and other symptoms of CFS are common to many illnesses.\\n   - For some CFS patients, it may not be obvious to doctors that they are ill.\\n   - The illness has a pattern of remission and relapse.\\n   - Symptoms vary from person to person in type, number, and severity.\\n  \\n  \\nThese factors have contributed to a low diagnosis rate. Of the one to four million Americans who have CFS, less than 20% have been diagnosed.\\n  Exams and Screening Tests for CFS\\n  \\nBecause there is no blood test, brain scan, or other lab test to diagnose CFS, the doctor should first rule out other possible causes.\\n  \\nIf a patient has had 6 or more consecutive months of severe fatigue that is reported to be unrelieved by sufficient bed rest and that is accompanied by nonspecific symptoms, including flu-like symptoms, generalized pain, and memory problems, the doctor should consider the possibility that the patient may have CFS. Further exams and tests are needed before a diagnosis can be made:\\n  \\n   - A detailed medical history will be needed and should include a review of medications that could be causing the fatigue and symptoms\\n   - A thorough physical and mental status examination will also be needed\\n   - A battery of laboratory screening tests will be needed to help identify or rule out other possible causes of the symptoms that could be treated\\n   - The doctor may also order additional tests to follow up on results of the initial screening tests\\n  \\n  \\nA CFS diagnosis requires that the patient has been fatigued for 6 months or more and has 4 of the 8 symptoms for CFS for 6 months or more. If, however, the patient has been fatigued for 6 months or more but does not have four of the eight symptoms, the diagnosis may be idiopathic fatigue.\\n  \\nThe complete process for diagnosing CFS can be found here.\\n  \\nAdditional information for healthcare professionals on use of tests can be found here.\",\n          \"Eating, diet, and nutrition have not been shown to play a role in causing or preventing simple kidney cysts.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "data = data.to_pandas()\n",
        "data.head(10)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#uncoment this line if you want to limit the size of the data.\n",
        "data = data[0:100]"
      ],
      "metadata": {
        "id": "hf7RQa6B5xpx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "As you can see, the medical information in the dataset is well-organized, and to someone like me, who is not an expert in the field, it appears to be quite valuable. This information could be a useful addition to any general medicine book to support primary care doctors."
      ],
      "metadata": {
        "id": "4FhUslovtiqn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Load the langchain libraries to load the document."
      ],
      "metadata": {
        "id": "2sLkrHF6lWhM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain.document_loaders import DataFrameLoader\n",
        "from langchain.vectorstores import Chroma"
      ],
      "metadata": {
        "id": "cCBAlIb596wZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The Document is in the Answer column, and the others columns are Metadata."
      ],
      "metadata": {
        "id": "nGCkVX6xldOR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_loader = DataFrameLoader(data, page_content_column=\"Answer\")\n"
      ],
      "metadata": {
        "id": "JZX8SaTe99Uf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_document = df_loader.load()\n",
        "display(df_document[:2])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 451
        },
        "id": "HDWnTqRY-IDr",
        "outputId": "b71a6cdd-9d34-46fd-f1ea-d138ccdd9387"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "[Document(metadata={'qtype': 'susceptibility', 'Question': 'Who is at risk for Lymphocytic Choriomeningitis (LCM)? ?'}, page_content='LCMV infections can occur after exposure to fresh urine, droppings, saliva, or nesting materials from infected rodents.  Transmission may also occur when these materials are directly introduced into broken skin, the nose, the eyes, or the mouth, or presumably, via the bite of an infected rodent. Person-to-person transmission has not been reported, with the exception of vertical transmission from infected mother to fetus, and rarely, through organ transplantation.'),\n",
              " Document(metadata={'qtype': 'symptoms', 'Question': 'What are the symptoms of Lymphocytic Choriomeningitis (LCM) ?'}, page_content='LCMV is most commonly recognized as causing neurological disease, as its name implies, though infection without symptoms or mild febrile illnesses are more common clinical manifestations. \\n                \\nFor infected persons who do become ill, onset of symptoms usually occurs 8-13 days after exposure to the virus as part of a biphasic febrile illness. This initial phase, which may last as long as a week, typically begins with any or all of the following symptoms: fever, malaise, lack of appetite, muscle aches, headache, nausea, and vomiting. Other symptoms appearing less frequently include sore throat, cough, joint pain, chest pain, testicular pain, and parotid (salivary gland) pain. \\n                \\nFollowing a few days of recovery, a second phase of illness may occur. Symptoms may consist of meningitis (fever, headache, stiff neck, etc.), encephalitis (drowsiness, confusion, sensory disturbances, and/or motor abnormalities, such as paralysis), or meningoencephalitis (inflammation of both the brain and meninges). LCMV has also been known to cause acute hydrocephalus (increased fluid on the brain), which often requires surgical shunting to relieve increased intracranial pressure. In rare instances, infection results in myelitis (inflammation of the spinal cord) and presents with symptoms such as muscle weakness, paralysis, or changes in body sensation. An association between LCMV infection and myocarditis (inflammation of the heart muscles) has been suggested. \\n                \\nPrevious observations show that most patients who develop aseptic meningitis or encephalitis due to LCMV survive. No chronic infection has been described in humans, and after the acute phase of illness, the virus is cleared from the body. However, as in all infections of the central nervous system, particularly encephalitis, temporary or permanent neurological damage is possible. Nerve deafness and arthritis have been reported. \\n                \\nWomen who become infected with LCMV during pregnancy may pass the infection on to the fetus. Infections occurring during the first trimester may result in fetal death and pregnancy termination, while in the second and third trimesters, birth defects can develop. Infants infected In utero can have many serious and permanent birth defects, including vision problems, mental retardation, and hydrocephaly (water on the brain). Pregnant women may recall a flu-like illness during pregnancy, or may not recall any illness. \\n                \\nLCM is usually not fatal. In general, mortality is less than 1%.')]"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can chunk the documents. The size to which we want to split the document is a design decision. The larger it is, the larger the prompt will be, and the slower the Model's response process.\n",
        "\n",
        "We also need to consider the maximum prompt size and ensure that the document does not exceed it."
      ],
      "metadata": {
        "id": "zPuTbSsXl3uF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain.text_splitter import CharacterTextSplitter"
      ],
      "metadata": {
        "id": "8wogWV1_-lxh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text_splitter = CharacterTextSplitter(chunk_size=1250,\n",
        "                                      separator=\"\\n\",\n",
        "                                      chunk_overlap=100)\n",
        "texts = text_splitter.split_documents(df_document)\n"
      ],
      "metadata": {
        "id": "kg7SRkqO-f9x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "These warnings we see are because it can't perform the partition of the required size. This is because it waits for a page break to divide the text and does so when possible."
      ],
      "metadata": {
        "id": "y0dn9EfrKoeW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "first_doc = texts[1]\n",
        "print(first_doc.page_content)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XaLWV_TzASSB",
        "outputId": "51ec4976-8e4f-4327-8b7f-9c416f43cb7e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LCMV is most commonly recognized as causing neurological disease, as its name implies, though infection without symptoms or mild febrile illnesses are more common clinical manifestations. \n",
            "                \n",
            "For infected persons who do become ill, onset of symptoms usually occurs 8-13 days after exposure to the virus as part of a biphasic febrile illness. This initial phase, which may last as long as a week, typically begins with any or all of the following symptoms: fever, malaise, lack of appetite, muscle aches, headache, nausea, and vomiting. Other symptoms appearing less frequently include sore throat, cough, joint pain, chest pain, testicular pain, and parotid (salivary gland) pain.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B2_Pt7N6Zg2X"
      },
      "source": [
        "### Initialize the Embedding Model and Vector DB"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from getpass import getpass\n",
        "import os\n",
        "if not 'OPENAI_API_KEY' in os.environ:\n",
        "  os.environ[\"OPENAI_API_KEY\"] = getpass(\"OpenAI API Key: \")"
      ],
      "metadata": {
        "id": "F_Dn06xGwjKP",
        "outputId": "31bffd76-ccfc-42a1-9926-676e81b8754c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "OpenAI API Key: ··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Obtain Your LangChain API Key from your Personal->Settings Area in LangSmith panel.\n",
        "\n",
        "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/langsmith_API_KEY.jpg?raw=true)"
      ],
      "metadata": {
        "id": "9NmsRgJ8rPoA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "if not 'LANGCHAIN_API_KEY' in os.environ:\n",
        "  os.environ[\"LANGCHAIN_API_KEY\"] = getpass(\"LangChain API Key: \")"
      ],
      "metadata": {
        "id": "xjuVU5rIrSRl",
        "outputId": "70352923-15bf-4db3-d2a9-6020c0be55de",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "LangChain API Key: ··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
        "os.environ[\"LANGCHAIN_ENDPOINT\"]=\"https://api.smith.langchain.com\"\n",
        "os.environ[\"LANGCHAIN_PROJECT\"]=\"langsmith_test3\""
      ],
      "metadata": {
        "id": "jjZjm0JmrVsY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We load the text-embedding-ada-002 model from OpenAI.\n"
      ],
      "metadata": {
        "id": "n8ROb8oMnRLD"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U57x2_87YSpb"
      },
      "outputs": [],
      "source": [
        "from langchain_openai import OpenAIEmbeddings\n",
        "\n",
        "model_name = 'text-embedding-ada-002'\n",
        "\n",
        "embed = OpenAIEmbeddings(\n",
        "    model=model_name\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The execution of this cell may take 3 to 5 minutes. If you want it to be faster, you can reduce the number of records in the dataset."
      ],
      "metadata": {
        "id": "cgTCwF7UMyNW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "directory_cdb = '/content/drive/MyDrive/chromadb'\n",
        "chroma_db = Chroma.from_documents(\n",
        "    df_document, embed, persist_directory=directory_cdb\n",
        ")"
      ],
      "metadata": {
        "id": "SEhQMQ8eCMj8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We are going to create three objects.\n",
        "\n",
        "* The language model, which can be any of those from OpenAI, the most common being gpt-3.5.\n",
        "* The memory, responsible for keeping the prompt with all the necessary history.\n",
        "* The retrieval, used to obtain information stored in ChromaDB."
      ],
      "metadata": {
        "id": "KhjV-T8GoarF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model=\"gpt-4o\"\n",
        "#model=\"gpt-3.5-turbo\""
      ],
      "metadata": {
        "id": "VksVhCjUnf8b"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zMRs9Klic5-Y"
      },
      "outputs": [],
      "source": [
        "from langchain.chat_models import ChatOpenAI\n",
        "from langchain_openai import OpenAI\n",
        "from langchain.chains.conversation.memory import ConversationBufferWindowMemory\n",
        "from langchain.chains import RetrievalQA\n",
        "\n",
        "llm=OpenAI(temperature=0.0)\n",
        "\n",
        "conversational_memory = ConversationBufferWindowMemory(\n",
        "    memory_key='chat_history',\n",
        "    k=4, #Number of messages stored in memory\n",
        "    return_messages=True #Must return the messages in the response.\n",
        ")\n",
        "\n",
        "qa = RetrievalQA.from_chain_type(\n",
        "    llm=llm,\n",
        "    #chain_type=\"stuff\",\n",
        "    retriever=chroma_db.as_retriever()\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can try the isolated Retrieval to see if the information it returns is relevant.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "ergrieE4o8qu"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 107
        },
        "id": "LaYSq0V-dxHw",
        "outputId": "4a66e515-6323-4906-9410-405fe8e1cf9a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-25-e02068ce0ffd>:1: LangChainDeprecationWarning: The method `Chain.run` was deprecated in langchain 0.1.0 and will be removed in 1.0. Use invoke instead.\n",
            "  qa.run(\"What is the main symptom of LCM?\")\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "' The main symptom of LCM is a biphasic febrile illness, which includes symptoms such as fever, malaise, lack of appetite, muscle aches, headache, nausea, and vomiting.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 25
        }
      ],
      "source": [
        "qa.run(\"What is the main symptom of LCM?\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "When observing the Retriever on Langsmith is possible to see the input and the documents returned by it:\n",
        "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_4_Retriever_1.jpg?raw=true)\n",
        "\n",
        "In the image below you can observe the call to the Model where the full prompt and the response are displayed.\n",
        "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_4_Retriever_2.jpg?raw=true)\n",
        "\n",
        "\n",
        "## Creating the Agent."
      ],
      "metadata": {
        "id": "pf9MXPeipEBO"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FwCYrS4duqBW"
      },
      "outputs": [],
      "source": [
        "from langchain.agents import Tool\n",
        "\n",
        "#Defining the list of tool objects to be used by LangChain.\n",
        "tools = [\n",
        "    Tool(\n",
        "        name='Medical KB',\n",
        "        func=qa.run,\n",
        "        description=(\n",
        "            \"\"\"use this tool when answering medical knowledge queries to get\n",
        "            more information about the topic\"\"\"\n",
        "        )\n",
        "    )\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JaKTzPUEvOoy",
        "outputId": "f2fceb76-32bd-4f2d-b562-ee70bded7034",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/langsmith/client.py:5301: LangChainBetaWarning: The function `loads` is in beta. It is actively being worked on, so the API may change.\n",
            "  prompt = loads(json.dumps(prompt_object.manifest))\n"
          ]
        }
      ],
      "source": [
        "from langchain.agents import create_react_agent\n",
        "from langchain import hub\n",
        "\n",
        "prompt = hub.pull(\"hwchase17/react-chat\")\n",
        "agent = create_react_agent(\n",
        "    tools=tools,\n",
        "    llm=llm,\n",
        "    prompt=prompt,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create an agent executor by passing in the agent and tools\n",
        "from langchain.agents import AgentExecutor\n",
        "agent_executor2 = AgentExecutor(agent=agent,\n",
        "                               tools=tools,\n",
        "                               verbose=True,\n",
        "                               memory=conversational_memory,\n",
        "                               max_iterations=30,\n",
        "                               max_execution_time=600,\n",
        "                               handle_parsing_errors=True\n",
        "                               )"
      ],
      "metadata": {
        "id": "5sjTL1EU1vNW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IlxUBWKcvzeP"
      },
      "source": [
        "### Using the Conversational Agent"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZZapCP4Pv2kz"
      },
      "source": [
        "To make queries we simply call the `agent` directly.\n",
        "\n",
        "First i will try a request not related to the Medical field."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "85vipqC02deV",
        "outputId": "c1383dcd-28be-4a1a-def4-eead285d44a6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:langchain_core.callbacks.manager:Error in StdOutCallbackHandler.on_chain_start callback: AttributeError(\"'NoneType' object has no attribute 'get'\")\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[32;1m\u001b[1;3mThought: Do I need to use a tool? No\n",
            "Final Answer: 2 multiplied by 2 is 4.\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'input': 'What is 2 multiplied by 2?',\n",
              " 'chat_history': [],\n",
              " 'output': '2 multiplied by 2 is 4.'}"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ],
      "source": [
        "agent_executor2.invoke({\"input\": \"What is 2 multiplied by 2?\"})"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The initial call to the Agent happens with just one call to OpenAI. One piece of information available in LangSmith is the entire prompt. I'll copy it just below the image, so you can see.\n",
        "\n",
        "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_4_1AE_1.jpg?raw=true)\n"
      ],
      "metadata": {
        "id": "LaTqyvBgG_wr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "```\n",
        "Assistant is a large language model trained by OpenAI.\n",
        "\n",
        "Assistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\n",
        "\n",
        "Assistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.\n",
        "\n",
        "Overall, Assistant is a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist.\n",
        "\n",
        "TOOLS:\n",
        "------\n",
        "\n",
        "Assistant has access to the following tools:\n",
        "\n",
        "Medical KB: use this tool when answering medical knowledge queries to get\n",
        "            more information about the topic\n",
        "\n",
        "To use a tool, please use the following format:\n",
        "\n",
        "\n",
        "Thought: Do I need to use a tool? Yes\n",
        "Action: the action to take, should be one of [Medical KB]\n",
        "Action Input: the input to the action\n",
        "Observation: the result of the action\n",
        "\n",
        "\n",
        "When you have a response to say to the Human, or if you do not need to use a tool, you MUST use the format:\n",
        "\n",
        "\n",
        "Thought: Do I need to use a tool? No\n",
        "Final Answer: [your response here]\n",
        "\n",
        "\n",
        "Begin!\n",
        "\n",
        "Previous conversation history:\n",
        "[]\n",
        "\n",
        "New input: What is 2 multiplied by 2?\n",
        "```"
      ],
      "metadata": {
        "id": "7LpYqjJuFKlw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Perfect, the model has responded without accessing the configured knowledge database.\n",
        "\n",
        "Now I will try with a question that is also not related to health."
      ],
      "metadata": {
        "id": "8suStMR7G11e"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "agent_executor2.memory.clear()"
      ],
      "metadata": {
        "id": "Wtwgfuy158LV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RJoAhy76vzAB",
        "outputId": "d2927d47-4f97-44dc-c7c3-dd05b220c1dc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:langchain_core.callbacks.manager:Error in StdOutCallbackHandler.on_chain_start callback: AttributeError(\"'NoneType' object has no attribute 'get'\")\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[32;1m\u001b[1;3m\n",
            "Thought: Do I need to use a tool? Yes\n",
            "Action: Medical KB\n",
            "Action Input: Botulism\u001b[0m\u001b[36;1m\u001b[1;3m Botulism is a rare but serious paralytic illness caused by a nerve toxin produced by certain bacteria. It can be contracted through contaminated food, wounds, or ingestion of bacterial spores. Symptoms include muscle paralysis, difficulty swallowing, and respiratory failure. Treatment includes antitoxin, supportive care, and removal of contaminated food or wound.\u001b[0m\u001b[32;1m\u001b[1;3mDo I need to use a tool? No\n",
            "Final Answer: To confirm the diagnosis, you should perform a physical exam and order laboratory tests, such as a stool or blood test, to detect the presence of the bacteria or its toxin. You may also need to consult with a specialist, such as an infectious disease doctor, for further evaluation and treatment.\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'input': 'I have a patient that can have Botulism,\\nhow can I confirm the diagnosis?',\n",
              " 'chat_history': [],\n",
              " 'output': 'To confirm the diagnosis, you should perform a physical exam and order laboratory tests, such as a stool or blood test, to detect the presence of the bacteria or its toxin. You may also need to consult with a specialist, such as an infectious disease doctor, for further evaluation and treatment.'}"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ],
      "source": [
        "agent_executor2.invoke({\"input\": \"\"\"I have a patient that can have Botulism,\n",
        "how can I confirm the diagnosis?\"\"\"})"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Perfect, the most important thing for us is that it has been able to identify that it should go to the medical database to search for information about the symptoms.\n",
        "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_4_1AE_3.jpg?raw=true)"
      ],
      "metadata": {
        "id": "8lsaoF8nJNnR"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mQeicHTj2pmY",
        "outputId": "092a21c6-53d1-4c06-9cb0-9820c6198dbe"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:langchain_core.callbacks.manager:Error in StdOutCallbackHandler.on_chain_start callback: AttributeError(\"'NoneType' object has no attribute 'get'\")\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[32;1m\u001b[1;3m\n",
            "Thought: Do I need to use a tool? Yes\n",
            "Action: Medical KB\n",
            "Action Input: Botulism\u001b[0m\u001b[36;1m\u001b[1;3m Botulism is a rare but serious paralytic illness caused by a nerve toxin produced by certain bacteria. It can be contracted through contaminated food, wounds, or ingestion of bacterial spores. Symptoms include muscle paralysis, difficulty swallowing, and respiratory failure. Treatment includes antitoxin, supportive care, and removal of the source of the toxin.\u001b[0m\u001b[32;1m\u001b[1;3mDo I need to use a tool? No\n",
            "Final Answer: Yes, botulism is an important illness that requires prompt diagnosis and treatment. If you suspect a patient may have botulism, it is important to consult with a specialist and perform necessary tests to confirm the diagnosis.\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'input': 'Is this an important illness?',\n",
              " 'chat_history': [HumanMessage(content='I have a patient that can have Botulism,\\nhow can I confirm the diagnosis?', additional_kwargs={}, response_metadata={}),\n",
              "  AIMessage(content='To confirm the diagnosis, you should perform a physical exam and order laboratory tests, such as a stool or blood test, to detect the presence of the bacteria or its toxin. You may also need to consult with a specialist, such as an infectious disease doctor, for further evaluation and treatment.', additional_kwargs={}, response_metadata={})],\n",
              " 'output': 'Yes, botulism is an important illness that requires prompt diagnosis and treatment. If you suspect a patient may have botulism, it is important to consult with a specialist and perform necessary tests to confirm the diagnosis.'}"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ],
      "source": [
        "agent_executor2.invoke({\"input\": \"Is this an important illness?\"})"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "On the left side of the image, you can see the various calls made by the Agent, which took 2.5 seconds to execute and consumed 2000 tokens.\n",
        "\n",
        "I will try to describe what happens in each call.\n",
        "\n",
        "**OpenAI**: he complete prompt, including the template from Hugging Face, and the user's question is passed to the OpenAI Model. It responds with the following step. Its answer is:\n",
        "\n",
        "*Thought: Do I need to use a tool?*\n",
        "\n",
        "*Yes  Action: Medical KB*  \n",
        "\n",
        "*Action Input: Botulism*\n",
        "\n",
        "**Medical KB**: The Agent utilizes the configured tool, passing only a single word as a parameter: \"Botulism.\"\n",
        "\n",
        "**Medical KB.Retriever:** The retriever returns four documents extracted from the Vectorial Database.\n",
        "\n",
        "**Medical KB.OpenAI:** The prompt is constructed with the information from the vectorial database. Even though I won't include the paste, I manage to identify that the four returned documents are actually two but duplicated. This, in a real project, could have helped me detect some issue, perhaps I have duplicates in the dataset, or maybe I loaded them twice. In any case, I'm consuming many more tokens than necessary. The model returns a response created considering the information contained in the prompt.\n",
        "\n",
        "**OpenAI:** In this final call to the model, it decides that the received response is what the user needs. Therefore, it marks it as correct and returns it to the user.\n",
        "\n",
        "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_4_1AE_3.jpg?raw=true)"
      ],
      "metadata": {
        "id": "_uNIh-zXKefJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "And the memory works perfectly. We can maintain a conversation, taking into account that the model knows the previous questions and answers.\n",
        "\n",
        "\n",
        "# Conclusions.\n",
        "LangSmith is an incredibly useful tool for tracing and storing all the information generated when making calls to LangChain.\n",
        "\n",
        "The experiment has been a small success. The Vectorial database has been configured and filled with information from the dataset. A LangChain agent has been created, and it has been able to retrieve information from the database only when necessary. Don't forget that our ChatBot has memory.\n",
        "\n",
        "\n",
        "\n",
        "And you have all the information in a project stored in LangSmith!!!!\n",
        "![My Image](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_4_1AE_Final.jpg?raw=true)\n",
        "\n"
      ],
      "metadata": {
        "id": "gdyhyd6nJnwN"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ykg5TYA033yR"
      },
      "source": [
        "---"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "machine_shape": "hm",
      "gpuType": "V100",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}