examples/basic-concepts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic panthera concepts\n",
    "\n",
    "This is an introductory guide to the concepts driving panthera and its usage.\n",
    "\n",
    "## Setup\n",
    "\n",
    "Let's add panthera to our classpath and require it to start playing around."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       ":ok"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(require '[clojupyter.misc.helper :as helper])\n",
    "(helper/add-dependencies '[panthera \"0.1-alpha.7\"])\n",
    ":ok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nil"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(require '[panthera.panthera :as pt])\n",
    "(require '[libpython-clj.python :as py])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `show` function is a helper to render data-frames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#'user/show"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(require '[clojupyter.display :as display])\n",
    "\n",
    "(defn show\n",
    "  [obj]\n",
    "  (display/html\n",
    "    (py/call-attr obj \"to_html\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Series\n",
    "\n",
    "*Serieses* are like vectors that act also as columns for *data-frames*. One *series* must have all the contained data with the same data type and if there is more than one type when you create a *series* than this one takes the most relaxed one.\n",
    "\n",
    "### Create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    2\n",
       "2    3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pt/series [1 2 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we print the *series* we see on the left its index and on the right its values. As you can see below the *series* itself we get the underlying data type (*dtype*) as well. Let's swap 3 with \"a\" and see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    2\n",
       "2    a\n",
       "dtype: object"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pt/series [1 2 \"a\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the *dtype* it's become `object`, which in *panthera* means either `string` or something that can be represented with a `string` and is not a primitive.\n",
    "\n",
    "If we get this data back to Clojure we'll see that we get the underlying original representation with mixed data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1 2 \"a\"]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(vec (pt/series [1 2 \"a\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that we can always move from a representation to another without many problems. A *series* can be treated as a Clojure vector if we want to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1 2 3)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(map inc (pt/series (range 3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But when we do this we lose metadata tied to it. The difference with regular vectors is mostly this metadata, a *series* specifically:\n",
    "\n",
    "- can have a name\n",
    "- has a *dtype*\n",
    "- has an index that can be freely named\n",
    "\n",
    "Let's see a few examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name    my-series\n",
       "dtype: object"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pt/series {:name \"my-series\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We just created an empty *series* with the name \"my-series\" to show that it can exist even with just metadata. The map passed as an argument lets you add other options to the function call without bothering about their position (in Python there is a clear distinction between *arguments* and *keyword arguments*, [more info](https://treyhunner.com/2018/04/keyword-arguments-in-python/)).\n",
    "\n",
    "We can combine arguments together to get the wanted outcome"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "idx    1\n",
       "Name: my-series, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pt/series 1 {:name \"my-series\" :index [\"idx\"]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing and subsetting\n",
    "\n",
    "Now \"my-series\" has a name, a value and a named index. This distinction is very important in *panthera*: indexing can be done by name and by position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    0\n",
       "d    3\n",
       "Name: my-series, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(-> (pt/series (range 5) {:name \"my-series\" :index [\"a\" \"b\" \"c\" \"d\" \"e\"]})\n",
    "    (pt/select-rows [0 3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    0\n",
       "d    3\n",
       "Name: my-series, dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(-> (pt/series (range 5) {:name \"my-series\" :index [\"a\" \"b\" \"c\" \"d\" \"e\"]})\n",
    "    (pt/select-rows [\"a\" \"d\"] :loc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above we were able to get the same values from the *series*, but the first time we used pure positional indexing, while the second one we used named indexing.\n",
    "\n",
    "This isn't something logical, it just works like this in pandas. So you'll have to memorize:\n",
    "\n",
    "- `:iloc`: positional indexing\n",
    "- `:loc`: named indexing or booleans\n",
    "\n",
    "Be aware that the result of this can be this behaviour:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100    0\n",
       "103    3\n",
       "Name: my-series, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(-> (pt/series (range 5) {:name \"my-series\" :index (map #(+ 100 %) (range 5))})\n",
    "    (pt/select-rows [0 3] :iloc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100    0\n",
       "103    3\n",
       "Name: my-series, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(-> (pt/series (range 5) {:name \"my-series\" :index (map #(+ 100 %) (range 5))})\n",
    "    (pt/select-rows [100 103] :loc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "3    3\n",
       "Name: my-series, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(-> (pt/series (range 5) {:name \"my-series\"})\n",
    "    (pt/select-rows [0 3] :loc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens above is that somewhat unexpectedly we get always the same values. Let's review every cell by itself:\n",
    "\n",
    "- the first time our series can be thought as a map like `{100 0 101 1 102 2 ...}`, but this in panthera doesn't change the fact that the first value is 0, the second is 1 and so on. So by getting `[0 3]` the result is a *series* with the first and fourth values\n",
    "- the second time we ask for named indices, and this for Clojurians is probably the clearest case: `(select-keys {100 0 101 1 102 2 ...} [100 103])` would give the same result\n",
    "- the last case is probably the least clear, we ask for named indices (`:loc`), but they are integers and they are positional. This happens because when we don't have named indices both *serieses* and *data-frames* assign a monotonically increasing index that has the value of the index itself as a label. If we had to represent a panthera index in pure Clojure it would be something like `{0 \"0\" 1 \"1\" 2 \"2\" ...}`\n",
    "\n",
    "There's another way to subset by index: slicing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    3\n",
       "4    4\n",
       "5    5\n",
       "dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(-> (pt/series (range 10))\n",
    "    (pt/select-rows (pt/slice 3 6)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    0\n",
       "b    1\n",
       "c    2\n",
       "d    3\n",
       "Name: my-series, dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(-> (pt/series (range 5) {:name \"my-series\" :index [\"a\" \"b\" \"c\" \"d\" \"e\"]})\n",
    "    (pt/select-rows (pt/slice \"a\" \"d\") :loc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Math and stats\n",
    "\n",
    "Math is easy with panthera! The only thing to keep in mind is that operations are **vectorized**, so something like `(+ [1 2 3] 1)` would result in `[2 3 4]`.\n",
    "\n",
    "To avoid confusion the panthera operations are named differently than the core functions (`+`, `-`, `*`, etc)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2\n",
       "1    3\n",
       "2    4\n",
       "dtype: int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pt/add (pt/series [1 2 3]) 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     0\n",
       "1     1\n",
       "2     8\n",
       "3    27\n",
       "4    64\n",
       "dtype: int64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pt/pow (pt/series (range 5)) 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    1\n",
       "2    1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pt/add (pt/series [1 2 3]) 1 (pt/series [-1 -2 -3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only note about these operations is that in order to work the first argument has to be a panthera data structure.\n",
    "\n",
    "There are more advanced stats functions besides the more regular ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.5"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pt/mean (pt/series (range 10)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10.712688874485469"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pt/kurtosis (pt/series (concat (range 10) [100])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.2568924988901746"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pt/skew (pt/series (concat (range 10) [100])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "837.3636363636364"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pt/var (pt/series (concat (range 10) [100])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.0"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pt/corr (pt/series (range 10)) (pt/series (range 9 0 -1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Clojure (clojupyter-v0.2.2)",
   "language": "clojure",
   "name": "clojupyter"
  },
  "language_info": {
   "file_extension": ".clj",
   "mimetype": "text/x-clojure",
   "name": "clojure",
   "version": "1.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}