SimpleIndexDemo.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9c48213d-6e6a-4c10-838a-2a7c710c3a05",
   "metadata": {},
   "source": [
    "# Simple Vector Store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "aadf16de",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import openai\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv()\n",
    "\n",
    "# Access your environment variables\n",
    "api_key = os.getenv(\"OPENAI_API_KEY\")\n",
    "\n",
    "openai.api_key = os.environ[\"OPENAI_API_KEY\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "50d3b817-b70e-4667-be4f-d3a0fe4bd119",
   "metadata": {},
   "source": [
    "#### Load documents, build the VectorStoreIndex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "690a6918-7c75-4f95-9ccc-d2c4a1fe00d7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:numexpr.utils:Note: NumExpr detected 16 cores but \"NUMEXPR_MAX_THREADS\" not set, so enforcing safe limit of 8.\n",
      "Note: NumExpr detected 16 cores but \"NUMEXPR_MAX_THREADS\" not set, so enforcing safe limit of 8.\n",
      "INFO:numexpr.utils:NumExpr defaulting to 8 threads.\n",
      "NumExpr defaulting to 8 threads.\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "import sys\n",
    "\n",
    "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
    "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))\n",
    "\n",
    "from llama_index import (\n",
    "    VectorStoreIndex,\n",
    "    SimpleDirectoryReader,\n",
    "    load_index_from_storage,\n",
    "    StorageContext,\n",
    ")\n",
    "from IPython.display import Markdown, display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "03d1691e-544b-454f-825b-5ee12f7faa8a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# load documents\n",
    "documents = SimpleDirectoryReader(\n",
    "    \"DocScraper/llama_index\"\n",
    ").load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ad144ee7-96da-4dd6-be00-fd6cf0c78e58",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "index = VectorStoreIndex.from_documents(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2bbccf1d-ac39-427c-b3a3-f8e9d1d12348",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# save index to disk\n",
    "index.set_index_id(\"llama_index\")\n",
    "index.storage_context.persist(\"./storage\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "197ca78e-1310-474d-91e3-877c3636b901",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:llama_index.indices.loading:Loading indices with ids: ['llama_index']\n",
      "Loading indices with ids: ['llama_index']\n"
     ]
    }
   ],
   "source": [
    "# rebuild storage context\n",
    "storage_context = StorageContext.from_defaults(persist_dir=\"storage\")\n",
    "# load index\n",
    "index = load_index_from_storage(storage_context, index_id=\"llama_index\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b6caf93b-6345-4c65-a346-a95b0f1746c4",
   "metadata": {},
   "source": [
    "#### Query Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "85466fdf-93f3-4cb1-a5f9-0056a8245a6f",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# set Logging to DEBUG for more detailed outputs\n",
    "query_engine = index.as_query_engine(response_mode=\"tree_summarize\")\n",
    "response = query_engine.query(\"What is an index?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bdda1b2c-ae46-47cf-91d7-3153e8d0473b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<b>An index is a data structure that organizes and stores data in a way that enables efficient retrieval of information. In the context provided, there are different types of indexes mentioned, such as the summary index, vector store index, tree index, and keyword table index. Each index has its own way of storing and querying data. For example, the summary index stores nodes as a sequential chain, the vector store index stores nodes and corresponding embeddings, the tree index builds a hierarchical tree from nodes, and the keyword table index maps keywords to corresponding nodes. These indexes are used in the context of a system called LlamaIndex, which involves parsing and chunking text into nodes and synthesizing responses based on retrieved nodes.</b>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(f\"<b>{response}</b>\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c80abba3-d338-42fd-9df3-b4e5ceb01cdf",
   "metadata": {},
   "source": [
    "**Query Index with SVM/Linear Regression**\n",
    "\n",
    "Use Karpathy's [SVM-based](https://twitter.com/karpathy/status/1647025230546886658?s=20) approach. Set query as positive example, all other datapoints as negative examples, and then fit a hyperplane."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "35e029e6-467b-4533-b566-a1568cc5f361",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query mode: svm\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "<b>The author wrote short stories and also started programming on an IBM 1401 computer in 9th grade. They later got their own microcomputer, a TRS-80, and wrote simple games, a rocket prediction program, and a word processor.</b>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query mode: linear_regression\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "<b>The author worked on writing and programming growing up. They wrote short stories and also started programming on an IBM 1401 computer in 9th grade using an early version of Fortran. Later, they got a microcomputer and wrote simple games, a rocket prediction program, and a word processor.</b>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query mode: logistic_regression\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "<b>The author worked on writing and programming growing up. They wrote short stories and also started programming on an IBM 1401 computer in 9th grade using an early version of Fortran. Later, they got a microcomputer and wrote simple games, a rocket prediction program, and a word processor.</b>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query_modes = [\n",
    "    \"svm\",\n",
    "    \"linear_regression\",\n",
    "    \"logistic_regression\",\n",
    "]\n",
    "for query_mode in query_modes:\n",
    "    # set Logging to DEBUG for more detailed outputs\n",
    "    query_engine = index.as_query_engine(vector_store_query_mode=query_mode)\n",
    "    response = query_engine.query(\"What did the author do growing up?\")\n",
    "    print(f\"Query mode: {query_mode}\")\n",
    "    display(Markdown(f\"<b>{response}</b>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e30166cd",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0bab9fd7-b0b9-4be1-8f05-eeb19bbe287a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<b>The author worked on writing and programming growing up. They wrote short stories and also started programming on an IBM 1401 computer in 9th grade using an early version of Fortran. Later, they got a microcomputer and wrote simple games, a rocket prediction program, and a word processor.</b>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(f\"<b>{response}</b>\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c9f256c8-b5ed-42db-b4de-8bd78a9540b0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Now all I had to do was learn Italian.\n",
      "\n",
      "Only stranieri (foreigners) had to take this entrance exam. In retrospect it may well have been a way of excluding them, because there were so many stranieri attracted by the idea of studying art in Florence that the Italian students would otherwise have been outnumbered. I was in decent shape at painting and drawing from the RISD foundation that summer, but I still don't know how I managed to pass the written exam. I remember that I answered the essay question by writing about Cezanne, and that I cranked up the intellectual level as high as I could to make the most of my limited vocabulary. [2]\n",
      "\n",
      "I'm only up to age 25 and already there are such conspicuous patterns. Here I was, yet again about to attend some august institution in the hopes of learning about some prestigious subject, and yet again about to be disappointed. The students and faculty in the painting department at the Accademia were the nicest people you could imagine, but they had long since arrived at an arrangement whereby the students wouldn't require the faculty to teach anything, and in return the faculty wouldn't require the students to learn anything. And at the same time all involved would adhere outwardly to the conventions of a 19th century atelier. We actually had one of those little stoves, fed with kindling, that you see in 19th century studio paintings, and a nude model sitting as close to it as possible without getting burned. Except hardly anyone else painted her besides me. The rest of the students spent their time chatting or occasionally trying to imitate things they'd seen in American art magazines.\n",
      "\n",
      "Our model turned out to live just down the street from me. She made a living from a combination of modelling and making fakes for a local antique dealer. She'd copy an obscure old painting out of a book, and then he'd take the copy and maltreat it to make it look old. [3]\n",
      "\n",
      "While I was a student at the Accademia I started painting still lives in my bedroom at night. These paintings were tiny, because the room was, and because I painted them on leftover scraps of canvas, which was all I could afford at the time. Painting still lives is different from painting people, because the subject, as its name suggests, can't move. People can't sit for more than about 15 minutes at a time, and when they do they don't sit very still. So the traditional m.o. for painting people is to know how to paint a generic person, which you then modify to match the specific person you're painting. Whereas a still life you can, if you want, copy pixel by pixel from what you're seeing. You don't want to stop there, of course, or you get merely photographic accuracy, and what makes a still life interesting is that it's been through a head. You want to emphasize the visual cues that tell you, for example, that the reason the color changes suddenly at a certain point is that it's the edge of an object. By subtly emphasizing such things you can make paintings that are more realistic than photographs not just in some metaphorical sense, but in the strict information-theoretic sense. [4]\n",
      "\n",
      "I liked painting still lives because I was curious about what I was seeing. In everyday life, we aren't consciously aware of much we're seeing. Most visual perception is handled by low-level processes that merely tell your brain \"that's a water droplet\" without telling you details like where the lightest and darkest points are, or \"that's a bush\" without telling you the shape and position of every leaf. This is a feature of brains, not a bug. In everyday life it would be distracting to notice every leaf on every bush. But when you have to paint something, you have to look more closely, and when you do there's a lot to see. You can still be noticing new things after days of trying to paint something people usually take for granted, just as you can after days of trying to write an essay about something people usually take for granted.\n",
      "\n",
      "This is not the only way to paint. I'm not 100% sure it's even a good way to paint. But it seemed a good enough bet to be worth trying.\n",
      "\n",
      "Our teacher, professor Ulivi, was a nice guy. He could see I worked hard, and gave me a good grade, which he wrote down in a sort of passport each student had. But the Accademia wasn't teaching me anything except Italian, and my money was running out, so at the end of the first year I went back to the US.\n"
     ]
    }
   ],
   "source": [
    "print(response.source_nodes[0].text)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0da9092e",
   "metadata": {},
   "source": [
    "**Query Index with custom embedding string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d57f2c87",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.indices.query.schema import QueryBundle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bbecbdb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_bundle = QueryBundle(\n",
    "    query_str=\"What did the author do growing up?\",\n",
    "    custom_embedding_strs=[\"The author grew up painting.\"],\n",
    ")\n",
    "query_engine = index.as_query_engine()\n",
    "response = query_engine.query(query_bundle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d4d1e028",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<b>The context does not provide information about what the author did growing up.</b>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(f\"<b>{response}</b>\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d7ff3d56",
   "metadata": {},
   "source": [
    "**Use maximum marginal relevance**\n",
    "\n",
    "Instead of ranking vectors purely by similarity, adds diversity to the documents by penalizing documents similar to ones that have already been found based on <a href=\"https://www.cs.cmu.edu/~jgc/publication/The_Use_MMR_Diversity_Based_LTMIR_1998.pdf\">MMR</a> . A lower mmr_treshold increases diversity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "60a27232",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = index.as_query_engine(\n",
    "    vector_store_query_mode=\"mmr\", vector_store_kwargs={\"mmr_threshold\": 0.2}\n",
    ")\n",
    "response = query_engine.query(\"What did the author do growing up?\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5636a15c-8938-4809-958b-03b8c445ecbd",
   "metadata": {},
   "source": [
    "#### Get Sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "db22a939-497b-4b1f-9aed-f22d9ca58c92",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Source (Doc id: fa51aa2a-af68-450f-bb00-786df71f2cdc): What I Worked On\n",
      "\n",
      "February 2021\n",
      "\n",
      "Before college the two main things I worked on, outside of schoo...\n",
      "\n",
      "> Source (Doc id: 4636483a-a416-4971-804f-abfb80a44378): Now all I had to do was learn Italian.\n",
      "\n",
      "Only stranieri (foreigners) had to take this entrance exa...\n"
     ]
    }
   ],
   "source": [
    "print(response.get_formatted_sources())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c0c5d984-db20-4679-adb1-1ea956a64150",
   "metadata": {},
   "source": [
    "#### Query Index with Filters\n",
    "\n",
    "We can also filter our queries using metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "59b8379d-f08f-4334-8525-6ddf4d13e33f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import Document\n",
    "\n",
    "doc = Document(text=\"target\", metadata={\"tag\": \"target\"})\n",
    "\n",
    "index.insert(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "aa281be0-1c7d-4d9c-a208-0ee5b7ab9953",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.vector_stores.types import ExactMatchFilter, MetadataFilters\n",
    "\n",
    "filters = MetadataFilters(filters=[ExactMatchFilter(key=\"tag\", value=\"target\")])\n",
    "\n",
    "retriever = index.as_retriever(\n",
    "    similarity_top_k=20,\n",
    "    filters=filters,\n",
    ")\n",
    "\n",
    "source_nodes = retriever.retrieve(\"What did the author do growing up?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c1c5ab85-25e4-4460-8b6a-3c119d92ba48",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "# retrieves only our target node, even though we set the top k to 20\n",
    "print(len(source_nodes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2b32da86",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target\n",
      "{'tag': 'target'}\n"
     ]
    }
   ],
   "source": [
    "print(source_nodes[0].text)\n",
    "print(source_nodes[0].metadata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "226b0c59",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}