lectures/notebooks/Lecture_07_Introduction_To_Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Fundamentals of Information Systems\n",
    "\n",
    "## Python Programming (for Data Science)\n",
    "\n",
    "### Master's Degree in Data Science\n",
    "\n",
    "#### Gabriele Tolomei\n",
    "<a href=\"mailto:gtolomei@math.unipd.it\">gtolomei@math.unipd.it</a><br/>\n",
    "University of Padua, Italy<br/>\n",
    "2018/2019<br/>\n",
    "November, 12 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 7: Introduction to <code>pandas</code>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What is <code>pandas</code>?\n",
    "\n",
    "<center>![](./img/pandas.png)</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Ok, seriously what *is* <code>pandas</code>?\n",
    "\n",
    "-  Very likely, one of the most important tool for a (Pythonista) Data Scientist!\n",
    "\n",
    "-  It contains data structures and data manipulation tools designed to make data **cleaning** and **analysis** fast and easy in Python.\n",
    "\n",
    "-  Acronym derived from \"**p**ython **an**d **d**ata **a**nalysi**s**\" or, maybe, \"**pan**el **da**ta\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What Does It Provide Us With?\n",
    "\n",
    "-  Often used in tandem with numerical computing tools like <code>**numpy**</code> and <code>**scipy**</code>, as well as machine learning modules like <code>**statsmodels**</code> and <code>**scikit-learn**</code>, and data visualization libraries like <code>**matplotlib**</code>. \n",
    "\n",
    "-  It uses significant parts of <code>**numpy**</code> style of array-based computing, especially vectorized functions for data processing without <code>**for**</code> loops.\n",
    "\n",
    "-  You can find more about <code>**pandas**</code> on the official [website](http://pandas.pydata.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "We import numpy as we will need it very often in combination with pandas\n",
    "\"\"\"\n",
    "import numpy as np\n",
    "\"\"\"\n",
    "As any other third-party module, the pandas module has to be imported before it can be used.\n",
    "If you installed Python with Anaconda, pandas would be just available to you.\n",
    "This is usually how pandas is imported and aliased. Although you could also\n",
    "use another syntax like 'from pandas import *', I strongly encourage you to define an alias,\n",
    "as this will help you to identify pandas' functions in your code.\n",
    "\"\"\"\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Introduction to <code>pandas</code> Data Structures\n",
    "\n",
    "-  Two workhorse data structures: <code>**Series**</code> and <code>**DataFrame**</code>. \n",
    "\n",
    "-  While they are not a universal solution for every problem, they provide a solid, easy-to-use basis for most applications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>Series</code>\n",
    "\n",
    "-  A <code>**Series**</code> is a one-dimensional array-like object containing a **sequence of values** (similar to <code>**numpy**</code>'s arrays) and an associated array of data labels, called its <code>**index**</code>.\n",
    "\n",
    "-  The string representation of a <code>**Series**</code> displayed interactively shows the index on the left and the values on the right. \n",
    "\n",
    "-  If no index is specified for the data, a default one consisting of the integers $0$ through $n-1$ (where $n$ is the length of the data) is created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    4\n",
      "1    7\n",
      "2   -5\n",
      "3    3\n",
      "dtype: int64\n",
      "\n",
      "The values of the Series object are: [ 4  7 -5  3]\n",
      "The index of the Series object is: RangeIndex(start=0, stop=4, step=1)\n"
     ]
    }
   ],
   "source": [
    "# Create a Series object (from a bulit-in Python list)\n",
    "s = pd.Series([4, 7, -5, 3])\n",
    "# Print the string representation of the Series object above\n",
    "print(s)\n",
    "print()\n",
    "# We can get the array representation and index object of the Series \n",
    "# via its values and index attributes, respectively\n",
    "# Print the array of values\n",
    "print(\"The values of the Series object are: {}\".format(s.values))\n",
    "# Print the array of indices\n",
    "print(\"The index of the Series object is: {}\".format(s.index)) # like range(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "d    4\n",
      "b    7\n",
      "a   -5\n",
      "c    3\n",
      "dtype: int64\n",
      "\n",
      "The index of the Series object is: Index(['d', 'b', 'a', 'c'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Often it will be desirable to create a Series with an index \n",
    "identifying each data point with a label.\n",
    "\"\"\"\n",
    "s2 = pd.Series([4, 7, -5, 3], index=['d', 'b', 'a', 'c'])\n",
    "print(s2)\n",
    "print()\n",
    "print(\"The index of the Series object is: {}\".format(s2.index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value indexed by 'a' is: -5\n",
      "After the assignment, the new Series object is:\n",
      "d     4\n",
      "b     7\n",
      "a    -5\n",
      "c    42\n",
      "dtype: int64\n",
      "\n",
      "The sub-series is:\n",
      "c    42\n",
      "b     7\n",
      "d     4\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Compared with numpy arrays, you can use values in the index \n",
    "when selecting single values or a set of values.\n",
    "\"\"\"\n",
    "# Accessing the element using the index value (i.e., not just an integer as with numpy!)\n",
    "print(\"The value indexed by 'a' is: {}\".format(s2['a']))\n",
    "# Assign a new value corresponding to an index\n",
    "s2['c'] = 42\n",
    "print(\"After the assignment, the new Series object is:\\n{}\".format(s2))\n",
    "print()\n",
    "# Indexing a subset of the elements using a subset of the index\n",
    "print(\"The sub-series is:\\n{}\".format(s2[['c', 'b', 'd']]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Select only the strictly positive elements of the series:\n",
      "d     4\n",
      "b     7\n",
      "c    42\n",
      "dtype: int64\n",
      "\n",
      "Double each element of the series:\n",
      "d     8\n",
      "b    14\n",
      "a   -10\n",
      "c    84\n",
      "dtype: int64\n",
      "\n",
      "Apply the exponential function to each element of the series:\n",
      "d    5.459815e+01\n",
      "b    1.096633e+03\n",
      "a    6.737947e-03\n",
      "c    1.739275e+18\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Using numpy functions or numpy-like operations, such as filtering with a boolean array, \n",
    "scalar multiplication, or applying math functions, will preserve the index-value link.\n",
    "\"\"\"\n",
    "print(\"Select only the strictly positive elements of the series:\\n{}\".format(s2[s2 > 0]))\n",
    "print()\n",
    "double_s2 = s2 * 2\n",
    "print(\"Double each element of the series:\\n{}\".format(double_s2))\n",
    "print()\n",
    "exp_s2 = np.exp(s2)\n",
    "print(\"Apply the exponential function to each element of the series:\\n{}\".format(exp_s2))\n",
    "# Note the implicit type conversion after the operation above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The series object created from the dictionary is:\n",
      "Ohio      35000\n",
      "Texas     71000\n",
      "Oregon    16000\n",
      "Utah       5000\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Another way to think about a pandas Series is as a fixed-length, ordered built-in dict, \n",
    "as it is a mapping of index values to data values. \n",
    "Therefore, Series can be used in many contexts where you might use a dict.\n",
    "\"\"\"\n",
    "# Suppose you have a dictionary containing the following key-value pairs\n",
    "sdata = {'Ohio': 35000, 'Texas': 71000, 'Oregon': 16000, 'Utah': 5000}\n",
    "# You can create a Series object from the dictionary above\n",
    "s3 = pd.Series(sdata)\n",
    "print(\"The series object created from the dictionary is:\\n{}\".format(s3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The series object created from the dictionary and index is:\n",
      "California        NaN\n",
      "Texas         71000.0\n",
      "Oregon        16000.0\n",
      "Ohio          35000.0\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "When only passing a dict, the index in the resulting Series will have the dict's keys\n",
    "in sorted order. You can override this by passing the dict keys \n",
    "in the order you want them to appear in the resulting Series.\n",
    "\"\"\"\n",
    "states = ['California', 'Texas', 'Oregon', 'Ohio']\n",
    "s4 = pd.Series(sdata, index=states)\n",
    "print(\"The series object created from the dictionary and index is:\\n{}\".format(s4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Check which elements of the series are null:\n",
      "California     True\n",
      "Texas         False\n",
      "Oregon        False\n",
      "Ohio          False\n",
      "dtype: bool\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Here, 3 values found in sdata were placed in the appropriate locations, \n",
    "but since no value for 'California' was found, it appears as NaN (Not a Number),\n",
    "which is considered in pandas to mark missing or NA values. \n",
    "We will use the terms \"missing\" or \"NA\" to refer to missing data. \n",
    "The 'isnull' and 'notnull' functions in pandas should be used to detect missing data.\n",
    "\"\"\"\n",
    "# Test for NA\n",
    "# We can either invoke the method on the Series object\n",
    "print(\"Check which elements of the series are null:\\n{}\".format(s4.isnull()))\n",
    "# Or use the standard pandas function and pass the Series as input argument\n",
    "#print(\"Check which elements of the series are null:\\n{}\".format(pd.isnull(s4)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Check which elements of the series are NOT null:\n",
      "California    False\n",
      "Texas          True\n",
      "Oregon         True\n",
      "Ohio           True\n",
      "dtype: bool\n"
     ]
    }
   ],
   "source": [
    "# Test for NOT NA\n",
    "# We can either invoke the method on the Series object\n",
    "print(\"Check which elements of the series are NOT null:\\n{}\".format(s4.notnull()))\n",
    "# Or use the standard pandas function and pass the Series as input argument\n",
    "#print(\"Check which elements of the series are NOT null:\\n{}\".format(pd.notnull(s4)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Series s3:\n",
      "Ohio      35000\n",
      "Texas     71000\n",
      "Oregon    16000\n",
      "Utah       5000\n",
      "dtype: int64\n",
      "\n",
      "Series s4:\n",
      "California        NaN\n",
      "Texas         71000.0\n",
      "Oregon        16000.0\n",
      "Ohio          35000.0\n",
      "dtype: float64\n",
      "\n",
      "Series s3 + s4:\n",
      "California         NaN\n",
      "Ohio           70000.0\n",
      "Oregon         32000.0\n",
      "Texas         142000.0\n",
      "Utah               NaN\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Series automatically aligns differently-indexed data in arithmetic operations.\n",
    "\"\"\"\n",
    "print(\"Series s3:\\n{}\".format(s3))\n",
    "print()\n",
    "print(\"Series s4:\\n{}\".format(s4))\n",
    "print()\n",
    "print(\"Series s3 + s4:\\n{}\".format(s3 + s4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "state\n",
      "California        NaN\n",
      "Texas         71000.0\n",
      "Oregon        16000.0\n",
      "Ohio          35000.0\n",
      "Name: population, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Both the Series object itself and its index have a name attribute, \n",
    "which integrates with other key areas of pandas functionality.\n",
    "\"\"\"\n",
    "s4.name = 'population'\n",
    "s4.index.name = 'state'\n",
    "print(s4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alice         NaN\n",
      "Bob       71000.0\n",
      "Carl      16000.0\n",
      "Denise    35000.0\n",
      "Name: population, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "A Series's index can be altered in place by assignment.\n",
    "\"\"\"\n",
    "s4.index = ['Alice', 'Bob', 'Carl', 'Denise']\n",
    "print(s4)\n",
    "# What happens if there is a length mismatch between the old and the new index?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Length mismatch: Expected axis has 4 elements, new values have 6 elements",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-13-878932cea0c8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mA\u001b[0m \u001b[0mSeries\u001b[0m\u001b[0;31m'\u001b[0m\u001b[0ms\u001b[0m \u001b[0mindex\u001b[0m \u001b[0mcan\u001b[0m \u001b[0mbe\u001b[0m \u001b[0maltered\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mplace\u001b[0m \u001b[0mby\u001b[0m \u001b[0massignment\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \"\"\"\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0ms4\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'Alice'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Bob'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Carl'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Denise'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Eddie'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Fran'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms4\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/pandas/core/generic.py\u001b[0m in \u001b[0;36m__setattr__\u001b[0;34m(self, name, value)\u001b[0m\n\u001b[1;32m   5078\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5079\u001b[0m             \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getattribute__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 5080\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__setattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   5081\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5082\u001b[0m             \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/properties.pyx\u001b[0m in \u001b[0;36mpandas._libs.properties.AxisProperty.__set__\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/pandas/core/series.py\u001b[0m in \u001b[0;36m_set_axis\u001b[0;34m(self, axis, labels, fastpath)\u001b[0m\n\u001b[1;32m    378\u001b[0m         \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__setattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'_index'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabels\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    379\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mfastpath\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 380\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_axis\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maxis\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabels\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    381\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    382\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_set_subtyp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_all_dates\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/pandas/core/internals/managers.py\u001b[0m in \u001b[0;36mset_axis\u001b[0;34m(self, axis, new_labels)\u001b[0m\n\u001b[1;32m    153\u001b[0m             raise ValueError(\n\u001b[1;32m    154\u001b[0m                 \u001b[0;34m'Length mismatch: Expected axis has {old} elements, new '\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 155\u001b[0;31m                 'values have {new} elements'.format(old=old_len, new=new_len))\n\u001b[0m\u001b[1;32m    156\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    157\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maxes\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0maxis\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnew_labels\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Length mismatch: Expected axis has 4 elements, new values have 6 elements"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "A Series's index can be altered in place by assignment.\n",
    "\"\"\"\n",
    "s4.index = ['Alice', 'Bob', 'Carl', 'Denise', 'Eddie', 'Fran']\n",
    "print(s4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>DataFrame</code>\n",
    "\n",
    "-  A <code>**DataFrame**</code> represents a rectangular table of data and contains an **ordered collection of columns**.\n",
    "\n",
    "-  Each column can be a different value type (numeric, string, boolean, etc.).\n",
    "\n",
    "-  The <code>**DataFrame**</code> has both a row and column index; it can be thought of as a dict of <code>**Series**</code> all sharing the same index. \n",
    "\n",
    "-  Under the hood, the data is stored as one or more two-dimensional blocks rather than a list, dict, or some other collection of one-dimensional arrays (the exact details are outside the scope of this class)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "<span style=\"color: red\">**NOTE:**</span>\n",
    "While a <code>**DataFrame**</code> is physically two-dimensional, you can use it to represent higher-dimensional data in a tabular format using **hierarchical indexing**, one of the more advanced data-handling features in <code>**pandas**</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The DataFrame created is as follows:\n",
      "    state  year  population\n",
      "0    Ohio  2000         1.5\n",
      "1    Ohio  2001         1.7\n",
      "2    Ohio  2002         3.6\n",
      "3  Nevada  2001         2.4\n",
      "4  Nevada  2002         2.9\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "There are many ways to construct a DataFrame, \n",
    "though one of the most common is from a dict of equal-length lists or numpy arrays.\n",
    "\"\"\"\n",
    "data = {'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada'],\n",
    "        'year': [2000, 2001, 2002, 2001, 2002],\n",
    "        'population': [1.5, 1.7, 3.6, 2.4, 2.9]}\n",
    "\n",
    "df = pd.DataFrame(data)\n",
    "# The resulting DataFrame will have its index assigned automatically as with Series, \n",
    "# and the columns are placed in sorted order\n",
    "print(\"The DataFrame created is as follows:\\n{}\".format(df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    state  year  population\n",
      "0    Ohio  2000         1.5\n",
      "1    Ohio  2001         1.7\n",
      "2    Ohio  2002         3.6\n",
      "3  Nevada  2001         2.4\n",
      "4  Nevada  2002         2.9\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>state</th>\n",
       "      <th>year</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ohio</td>\n",
       "      <td>2000</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ohio</td>\n",
       "      <td>2001</td>\n",
       "      <td>1.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ohio</td>\n",
       "      <td>2002</td>\n",
       "      <td>3.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Nevada</td>\n",
       "      <td>2001</td>\n",
       "      <td>2.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nevada</td>\n",
       "      <td>2002</td>\n",
       "      <td>2.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    state  year  population\n",
       "0    Ohio  2000         1.5\n",
       "1    Ohio  2001         1.7\n",
       "2    Ohio  2002         3.6\n",
       "3  Nevada  2001         2.4\n",
       "4  Nevada  2002         2.9"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "When you work with large DataFrames it is useful to show the first n rows\n",
    "The 'head()' method without any input argument returns the first 5 rows.\n",
    "Otherwise, you can call the method specifying the number of records to be displayed.\n",
    "\"\"\"\n",
    "print(df.head())\n",
    "# You can also just type head()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   year   state  population\n",
      "0  2000    Ohio         1.5\n",
      "1  2001    Ohio         1.7\n",
      "2  2002    Ohio         3.6\n",
      "3  2001  Nevada         2.4\n",
      "4  2002  Nevada         2.9\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "If you specify a sequence of columns at creation time, \n",
    "the DataFrame's columns will be arranged in that order.\n",
    "\"\"\"\n",
    "# Note that column names must reflect key names of the input dictionary 'data'\n",
    "df2 = pd.DataFrame(data, columns=['year', 'state', 'population'])\n",
    "print(df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       year   state  population debt\n",
      "one    2000    Ohio         1.5  NaN\n",
      "two    2001    Ohio         1.7  NaN\n",
      "three  2002    Ohio         3.6  NaN\n",
      "four   2001  Nevada         2.4  NaN\n",
      "five   2002  Nevada         2.9  NaN\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "If you pass a column that isn't contained in the dict, \n",
    "it will appear with missing values (NaN) in the result.\n",
    "\"\"\"\n",
    "# Here we are adding an extra column 'debt' and we assign a string-based index,\n",
    "# instead of the default integer range.\n",
    "df3 = pd.DataFrame(data, columns=['year', 'state', 'population', 'debt'], \n",
    "                      index=['one', 'two', 'three', 'four', 'five'])\n",
    "print(df3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "one        Ohio\n",
      "two        Ohio\n",
      "three      Ohio\n",
      "four     Nevada\n",
      "five     Nevada\n",
      "Name: state, dtype: object\n",
      "\n",
      "one      1.5\n",
      "two      1.7\n",
      "three    3.6\n",
      "four     2.4\n",
      "five     2.9\n",
      "Name: population, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "A column in a DataFrame can be retrieved as a Series,\n",
    "either by dict-like notation or by attribute.\n",
    "\"\"\"\n",
    "# Accessing column using dict-like notation\n",
    "print(df3['state'])\n",
    "print()\n",
    "# Accessing column using attribute ('.column_name') notation\n",
    "print(df3.population)\n",
    "\n",
    "# Note that the returned Series have the same index as the DataFrame, \n",
    "# and their name attribute has been appropriately set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "year          2002\n",
       "state         Ohio\n",
       "population     3.6\n",
       "debt           NaN\n",
       "Name: three, dtype: object"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Rows can also be retrieved by position or name by a couple of methods, \n",
    "such as the 'loc' indexing field (much more on this later).\n",
    "\"\"\"\n",
    "# Accessing the row indexed by 'three'\n",
    "df3.loc['three']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       year   state  population   debt\n",
      "one    2000    Ohio         1.5  73.42\n",
      "two    2001    Ohio         1.7  73.42\n",
      "three  2002    Ohio         3.6  73.42\n",
      "four   2001  Nevada         2.4  73.42\n",
      "five   2002  Nevada         2.9  73.42\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Columns can be modified by assignment. \n",
    "For example, the empty 'debt' column could be assigned a scalar value or an array of values.\n",
    "\"\"\"\n",
    "# Assigning the same scalar value to the column 'debt'\n",
    "df3['debt'] = 73.42\n",
    "print(df3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       year   state  population  debt\n",
      "one    2000    Ohio         1.5   1.3\n",
      "two    2001    Ohio         1.7   6.4\n",
      "three  2002    Ohio         3.6   0.5\n",
      "four   2001  Nevada         2.4   0.0\n",
      "five   2002  Nevada         2.9  -9.8\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Columns can be modified by assignment. \n",
    "For example, the empty 'debt' column could be assigned a scalar value or an array of values.\n",
    "\"\"\"\n",
    "# Assigning an array of values to the column 'debt'\n",
    "df3['debt'] = np.array([1.3, 6.4, 0.5, 0.0, -9.8])\n",
    "print(df3)\n",
    "# When assigning lists or arrays to a column, \n",
    "# the value's length must match the length of the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       year   state  population  debt\n",
      "one    2000    Ohio         1.5   NaN\n",
      "two    2001    Ohio         1.7  -1.2\n",
      "three  2002    Ohio         3.6   NaN\n",
      "four   2001  Nevada         2.4  -1.5\n",
      "five   2002  Nevada         2.9  -1.7\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "We can also assign a Series to a DataFrame column.\n",
    "In this case, its labels will be realigned exactly to the DataFrame's index, \n",
    "inserting missing values (NaN) in any holes.\n",
    "\"\"\"\n",
    "val = pd.Series([-1.2, -1.5, -1.7], index=['two', 'four', 'five'])\n",
    "df3['debt'] = val\n",
    "print(df3)\n",
    "# If one or more elements in the specified index are not present\n",
    "# in the DataFrame's index, the assignment for those elements won't occur"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       year   state  population  debt  eastern\n",
      "one    2000    Ohio         1.5   NaN     True\n",
      "two    2001    Ohio         1.7  -1.2     True\n",
      "three  2002    Ohio         3.6   NaN     True\n",
      "four   2001  Nevada         2.4  -1.5    False\n",
      "five   2002  Nevada         2.9  -1.7    False\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Assigning a column that doesn't exist will create a new column.\n",
    "\"\"\"\n",
    "# The effect of the following statement is to:\n",
    "# 1. create a new column 'eastern'\n",
    "# 2. assign to that column a value which results from the boolean expression\n",
    "# Remember that boolean expressions computed over Series (or numpy arrays for that matters)\n",
    "# return a Series (array)\n",
    "df3['eastern'] = df3['state'] == 'Ohio'\n",
    "print(df3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       year   state  population  debt\n",
      "one    2000    Ohio         1.5   NaN\n",
      "two    2001    Ohio         1.7  -1.2\n",
      "three  2002    Ohio         3.6   NaN\n",
      "four   2001  Nevada         2.4  -1.5\n",
      "five   2002  Nevada         2.9  -1.7\n",
      "\n",
      "The columns of the DataFrame are: Index(['year', 'state', 'population', 'debt'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "To delete a column, we can use the 'del' keyword as with a dict.\n",
    "\"\"\"\n",
    "del df3['eastern']\n",
    "print(df3)\n",
    "print()\n",
    "print(\"The columns of the DataFrame are: {}\".format(df3.columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<span style=\"color: red\">**Note:**</span>\n",
    "The column returned when indexing a <code>**DataFrame**</code> is a **view** on the underlying data, <u>not</u> a copy. Thus, any in-place modifications to the <code>**Series**</code> will be reflected in the <code>**DataFrame**</code>. The column can be explicitly copied using the <code>**Series**</code>'s <code>**copy()**</code> method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      Nevada  California  Texas\n",
      "2000     NaN         3.6    5.5\n",
      "2001     2.4         3.7    5.8\n",
      "2002     2.9         4.1    7.0\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Another possible way of creating a pandas DataFrame is by using\n",
    "a dictionary of dictionaries (i.e., a nested dictionary).\n",
    "With this approach, pandas will interpret the outer dictionary keys as the columns\n",
    "and the inner keys as the row indices.\n",
    "\"\"\"\n",
    "population = {\n",
    "    'Nevada': {2001: 2.4, 2002: 2.9},\n",
    "    'California': {2000: 3.6, 2001: 3.7, 2002: 4.1},\n",
    "    'Texas': {2000: 5.5, 2001: 5.8, 2002: 7.0},\n",
    "}\n",
    "\n",
    "df4 = pd.DataFrame(population)\n",
    "print(df4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Possible Arguments to <code>DataFrame()</code> Constructor\n",
    "\n",
    "<img src=\"./img/pd_dataframe_1.png\" width=\"50%\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Possible Arguments to <code>DataFrame()</code> Constructor\n",
    "\n",
    "<img src=\"./img/pd_dataframe_2.png\" width=\"50%\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>Index</code> Objects\n",
    "\n",
    "-  <code>**Index**</code> objects are responsible for holding the axis **labels** and other metadata (like the axis name or names). \n",
    "\n",
    "-  Any array or other sequence of labels used when constructing a <code>**Series**</code> or <code>**DataFrame**</code> is internally converted to an <code>**Index**</code>.\n",
    "\n",
    "-  They are **_immutable_**, as this makes safer to share those objects among data structures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Suppose we create a Series with an explicit index (i.e., a built-in list)\n",
    "s = pd.Series(range(5), index=['a', 'b', 'c', 'd', 'e'])\n",
    "# Store the index property of the Series object\n",
    "index = s.index\n",
    "print(\"The index associated with the series is: {}\".format(index))\n",
    "# As any sequence-like objects, Index can be sliced over\n",
    "print(\"The sliced index associated with the series is: {}\".format(index[1:4]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Index objects are IMMUTABLE\n",
    "index[1] = 'beta'  # TypeError"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>Index</code> Object's Hierarchy\n",
    "\n",
    "<img src=\"./img/pd_index.png\" width=\"50%\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>Index</code> Object's Methods and Properties\n",
    "\n",
    "<img src=\"./img/pd_index_methods.png\" width=\"50%\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "We can eliminate one or more entries from an axis using the 'drop' method.\n",
    "This will return a new object with the entries that are left.\n",
    "\"\"\"\n",
    "s = pd.Series(np.arange(5.), index=['a', 'b', 'c', 'd', 'e'])\n",
    "print(\"Original Series:\\n{}\".format(s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Eliminate a single entry\n",
    "new_s = s.drop('b')\n",
    "print(\"Series after dropping entry 'b':\\n{}\".format(new_s))\n",
    "print()\n",
    "# Eliminate multiple entries\n",
    "new_s = s.drop(['a', 'e'])\n",
    "print(\"Series after dropping entries 'a' and 'e':\\n{}\".format(new_s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "With DataFrame, index values can be deleted from either axis: index (row) and column.\n",
    "\"\"\"\n",
    "df = pd.DataFrame(np.arange(16).reshape((4, 4)),\n",
    "                    index=['Ohio', 'Colorado', 'Utah', 'New York'],\n",
    "                    columns=['one', 'two', 'three', 'four'])\n",
    "print(\"Original DataFrame:\\n{}\".format(df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Eliminate entries on the default index axis (i.e., rows)\n",
    "new_df = df.drop(['Utah', 'Ohio'])\n",
    "print(\"Index-Dropped DataFrame (axis 0):\\n{}\".format(new_df))\n",
    "print()\n",
    "# Eliminate entries on the column axis\n",
    "new_df2 = df.drop(['two', 'four'], axis=1)\n",
    "print(\"Column-Dropped DataFrame (axis 1):\\n{}\".format(new_df2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Many functions, like drop, which modify the size or shape of a Series or DataFrame, \n",
    "can manipulate an object in place without returning a new object.\n",
    "\"\"\"\n",
    "# Eliminating a single entry on the row axis in-place\n",
    "df.drop('New York', inplace=True)\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Indexing, Selecting, Slicing, and Filtering\n",
    "\n",
    "-  <code>**Series**</code> indexing works analogously to <code>**numpy**</code> array indexing.\n",
    "\n",
    "-  In addition to just integers, you can also use the <code>**Series**</code>'s index values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Create a Series object\n",
    "s = pd.Series(np.arange(4.), index=['a', 'b', 'c', 'd'])\n",
    "print(s)\n",
    "print()\n",
    "# Accessing the 2nd element using integer indexing\n",
    "print(\"The 2nd element of the Series is: {}\".format(s[1]))\n",
    "# Accessing the 3rd element using index value 'c'\n",
    "print(\"The 3rd element of the Series is: {}\".format(s['c']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Selecting multiple entries using a list of integers\n",
    "print(\"The 1st and 3rd elements of the Series are:\\n{}\"\n",
    "      .format(s[[0, 2]]))\n",
    "print()\n",
    "# Selecting multiple entries using a list of index values\n",
    "print(\"The elements corresponding to entry 'a' and 'c' of the Series are:\\n{}\"\n",
    "      .format(s[['a', 'c']]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Slicing the series using integer slicing\n",
    "print(\"The first two elements of the Series are:\\n{}\".format(s[:2]))\n",
    "print()\n",
    "# Slicing using labels behaves differently than normal Python integer slicing \n",
    "# in that the endpoint is inclusive\n",
    "print(\"The first two elements of the Series are:\\n{}\".format(s['a':'b']))\n",
    "print()\n",
    "# Filtering on a series according to a boolean condition\n",
    "print(\"Select only those elements greater than 0 and less than or equal to 2:\\n{}\"\n",
    "      .format(s[(s > 0) & (s <= 2)]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Indexing into a DataFrame is for retrieving one or more columns, \n",
    "either with a single value or sequence.\n",
    "\"\"\"\n",
    "df = pd.DataFrame(np.arange(16).reshape((4, 4)),\n",
    "                  index=['Ohio', 'Colorado', 'Utah', 'New York'],\n",
    "                  columns=['one', 'two', 'three', 'four'])\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Accessing a single column in a DataFrame using column values (axis 1)\n",
    "print(\"Column 'two':\\n{}\".format(df['two']))\n",
    "print()\n",
    "# Accessing multiple columns in a DataFrame using column values (axis 1)\n",
    "print(\"Column 'three' and 'one':\\n{}\".format(df[['three', 'one']]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "We can use integer slicing as well as boolean array to select rows.\n",
    "\"\"\"\n",
    "# Integer slicing inside of [] slices ONLY the rows\n",
    "print(\"First two rows:\\n{}\".format(df[:2]))\n",
    "print()\n",
    "# Select all the rows corresponding to the entries where values in column 'three'\n",
    "# are greater than 5 (this will filter out the first row)\n",
    "print(df[df['three'] > 5])\n",
    "print()\n",
    "# Select all the rows corresponding to the entries where values in column 'two'\n",
    "# are smaller than 7 (this will filter out the last two rows)\n",
    "print(df[df['two'] < 7])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Boolean indexing on DataFrame works similar to Series.\n",
    "\"\"\"\n",
    "print(df < 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Assignment over boolean indexing on a DataFrame\n",
    "\"\"\"\n",
    "df[df < 3] = 0\n",
    "print(df)\n",
    "# This makes DataFrame syntactically more like \n",
    "# a two-dimensional numpy array in this particular case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "To indexing a DataFrame, there are two special operators: loc and iloc. \n",
    "They enable you to select a subset of the rows and columns from a DataFrame \n",
    "with numpy-like notation using either axis labels (loc) or integers (iloc).\n",
    "\"\"\"\n",
    "# Using 'loc' operator we can index both rows (axis 0) and columns (axis 1)\n",
    "# by means of labels rather than integers.\n",
    "# Extract the row corresponding to 'Utah'\n",
    "print(df.loc['Utah'])\n",
    "print()\n",
    "# Using 'iloc' operator we can index both rows (axis 0) and columns (axis 1)\n",
    "# by means of integers.\n",
    "# Extract the 3rd row (which corresponds to 'Utah')\n",
    "print(\"3rd row:\\n{}\".format(df.iloc[2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Using 'loc' operator we can index both rows (axis 0) and columns (axis 1)\n",
    "# by means of labels rather than integers.\n",
    "# Extract the row corresponding to 'Utah' and the column corresponding to 'two'\n",
    "print(df.loc['Utah', 'two'])\n",
    "print()\n",
    "# Using 'iloc' operator we can index both rows (axis 0) and columns (axis 1)\n",
    "# by means of integers.\n",
    "# Extract the 3rd row (which corresponds to 'Utah') and 2nd column ('two')\n",
    "print(\"3rd row and 2nd column:\\n{}\".format(df.iloc[2,1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Using 'loc' operator we can index both rows (axis 0) and columns (axis 1)\n",
    "# by means of labels rather than integers.\n",
    "# Extract the row corresponding to 'Utah' and the columns 'two' and 'three'\n",
    "print(df.loc['Utah', ['two', 'three']])\n",
    "print()\n",
    "# Using 'iloc' operator we can index both rows (axis 0) and columns (axis 1)\n",
    "# by means of integers.\n",
    "# Extract the 3rd row (Utah), 2nd and 3rd columns ('two', 'three')\n",
    "print(\"3rd row; 2nd and 3rd columns:\\n{}\".format(df.iloc[2, [1,2]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "print(\"1st and 4th row; 4th, 1st and 2nd column:\\n{}\"\n",
    "      .format(df.loc[['Ohio','New York'], ['four', 'one', 'two']]))\n",
    "print()\n",
    "print(\"1st and 4th row; 4th, 1st and 2nd column:\\n{}\".format(df.iloc[[0, 3], [3, 0, 1]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Mixed label/integer selection (DEPRECATED!)\n",
    "print(\"1st and 4th row; 4th, 1st and 2nd column:\\n{}\"\n",
    "      .format(df.ix[['Ohio', 'New York'], [3, 0, 1]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Slicing over two axis of a DataFrame.\n",
    "\"\"\"\n",
    "# Using loc in combination with label slicing (rightmost extreme included!)\n",
    "print(df.loc[:'Utah', 'two':'three'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Slicing over two axis of a DataFrame.\n",
    "\"\"\"\n",
    "# Using iloc in combination with integer slicing (rightmost extreme NOT included!)\n",
    "print(df.iloc[1:, :3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Indexing and Slicing can be mixed (e.g., indexing on one axis and slicing on the other).\n",
    "This can be done both using loc and iloc operator\n",
    "\"\"\"\n",
    "# Indexing over axis 0 and slicing over axis 1 using loc.\n",
    "# row indexed by 'New York', column indexed by 'one', 'two' and 'three'\n",
    "print(df.loc['New York', 'one':'three'])\n",
    "print()\n",
    "# Slicing over axis 0 and indexing over axis 1 using iloc.\n",
    "# first two rows, 3rd column\n",
    "print(df.iloc[:2, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Indexing Options for <code>DataFrame</code>\n",
    "\n",
    "<img src=\"./img/pd_indexing.png\" width=\"50%\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Arithmetic and Data Alignment\n",
    "\n",
    "-  An important <code>**pandas**</code> feature is the behavior of arithmetic between objects with different indexes.\n",
    "\n",
    "-  When adding together objects, if any index pairs are not the same, the respective index in the result will be the **union** of the index pairs. \n",
    "\n",
    "-  For users with database experience, this is similar to an automatic **outer join** on the index labels.\n",
    "\n",
    "-  The internal data alignment introduces <code>**NaN**</code> values in the label locations that don't overlap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Consider the following two Series objects.\n",
    "\"\"\"\n",
    "s1 = pd.Series([7.3, -2.5, 3.4, 1.5], index=['a', 'c', 'd', 'e'])\n",
    "s2 = pd.Series([-2.1, 3.6, -1.5, 4, 3.1], index=['a', 'c', 'e', 'f', 'g'])\n",
    "# Let's try to add s1 to s2 as follows.\n",
    "print(s1 + s2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "In the case of DataFrame, alignment is performed on both the rows and the columns.\n",
    "\"\"\"\n",
    "df1 = pd.DataFrame(np.arange(9.).reshape((3, 3)), columns=list('bcd'), \n",
    "                   index=['Ohio', 'Texas', 'Colorado'])\n",
    "print(df1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df2 = pd.DataFrame(np.arange(12.).reshape((4, 3)), columns=list('bde'),\n",
    "                   index=['Utah', 'Ohio', 'Texas', 'Oregon'])\n",
    "print(df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Adding these together returns a DataFrame whose index and columns\n",
    "are the union of the ones in each DataFrame.\n",
    "\"\"\"\n",
    "print(df1 + df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "In arithmetic operations between differently-indexed objects, \n",
    "you might want to fill with a special value, like 0, when an axis label \n",
    "is found in one object but not the other.\n",
    "\"\"\"\n",
    "df1 = pd.DataFrame(np.arange(12.).reshape((3, 4)), columns=list('abcd'))\n",
    "print(df1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "In arithmetic operations between differently-indexed objects, \n",
    "you might want to fill with a special value, like 0, when an axis label \n",
    "is found in one object but not the other.\n",
    "\"\"\"\n",
    "df2 = pd.DataFrame(np.arange(20.).reshape((4, 5)), columns=list('abcde'))\n",
    "print(df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "In arithmetic operations between differently-indexed objects, \n",
    "you might want to fill with a special value, like 0, when an axis label \n",
    "is found in one object but not the other.\n",
    "\"\"\"\n",
    "# df1 + df2 will result in NaN on those unmatched indices\n",
    "print(df1 + df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "In arithmetic operations between differently-indexed objects, \n",
    "you might want to fill with a special value, like 0, when an axis label \n",
    "is found in one object but not the other.\n",
    "\"\"\"\n",
    "# Using the add method on df1, we pass df2 and an argument to fill_value\n",
    "print(df1.add(df2, fill_value=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Arithmetic Operations\n",
    "\n",
    "<center>![](./img/pd_arithmetics.png)</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Operations between <code>DataFrame</code> and <code>Series</code>\n",
    "\n",
    "-  As with <code>**numpy**</code> arrays of different dimensions, arithmetic between <code>**DataFrame**</code> and <code>**Series**</code> is also defined.\n",
    "\n",
    "-  Let's first recall again how this works with <code>**numpy**</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Let's define a two-dimensional numpy array (i.e., a 3-by-4 matrix)\n",
    "X = np.arange(12).reshape(3,4)\n",
    "print(\"X =\\n{}\".format(X))\n",
    "# Let's define a one-dimensional numpy array (i.e., a 3-by-1 vector)\n",
    "y = np.arange(3).reshape(3,1)\n",
    "print(\"y =\\n{}\".format(y))\n",
    "print()\n",
    "print(\"X - y =\\n{}\".format(X-y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Let's see the same yet applied to a pandas DataFrame and a Series.\n",
    "\"\"\"\n",
    "df = pd.DataFrame(np.arange(12.).reshape((4, 3)), columns=list('bde'),\n",
    "                     index=['Utah', 'Ohio', 'Texas', 'Oregon'])\n",
    "\n",
    "print(\"DataFrame:\\n{}\".format(df))\n",
    "print()\n",
    "s = df.iloc[0]\n",
    "print(\"Series:\\n{}\".format(s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "By default, arithmetic between DataFrame and Series matches the index of the Series \n",
    "on the DataFrame's columns, broadcasting down the rows.\n",
    "\"\"\"\n",
    "print(df - s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "If an index value is not found in either the DataFrame's columns or the Series's index, \n",
    "the objects will be reindexed to form the union.\n",
    "\"\"\"\n",
    "s2 = pd.Series(range(3), index=['b', 'e', 'f'])\n",
    "print(df + s2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "If we want to instead broadcast over the columns, matching on the rows, \n",
    "you have to use one of the arithmetic methods. For example:\n",
    "\"\"\"\n",
    "# Get a Series corresponding to the 'd' column\n",
    "s3 = df['d']\n",
    "print(s3)\n",
    "# Subtract from the DataFrame the Series just extracted using axis\n",
    "# The axis number is the axis to match on. \n",
    "# In this case we mean to match on the DataFrame's row index (axis=0) and broadcast across.\n",
    "diff_df = df.sub(s3, axis=0) # sub creates a new object\n",
    "print(diff_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Function Application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "numpy ufuncs (element-wise array methods) also work with pandas objects.\n",
    "\"\"\"\n",
    "df = pd.DataFrame(np.random.randn(4, 3), columns=list('bde'),\n",
    "                     index=['Utah', 'Ohio', 'Texas', 'Oregon'])\n",
    "print(df)\n",
    "print()\n",
    "print(np.abs(df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Another frequent operation is applying a function on 1-D arrays to each column or row. \n",
    "DataFrame's 'apply' method does exactly this.\n",
    "\"\"\"\n",
    "# Define an anonymous (i.e., lambda) function\n",
    "# Note that the argument x here is meant to be an array (not a scalar!)\n",
    "f = lambda x: x.max() - x.min()\n",
    "# Apply the lambda function column-wise (default)\n",
    "print(\"Column-wise application of lambda:\\n{}\".format(df.apply(f))) \n",
    "# equals to df.apply(f, axis=0)\n",
    "print()\n",
    "# Apply the lambda function row-wise\n",
    "print(\"Row-wise application of lambda:\\n{}\".format(df.apply(f, axis=1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>apply</code>\n",
    "\n",
    "-  The function passed to <code>**apply**</code> is expected to work on a one-dimensional array of data (i.e., a <code>**Series**</code>) as input.\n",
    "\n",
    "-  It **does not** need to necessarily return a scalar value as output, in fact it can also return a <code>**Series**</code> with multiple values.\n",
    "\n",
    "-  Many of the most common array statistics (like <code>**sum**</code> and <code>**mean**</code>) are <code>**DataFrame**</code> built-in methods, therefore using <code>**apply**</code> is **not** necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Consider the following function which returns a Series instead of a scalar value.\n",
    "\"\"\"\n",
    "def foo(x):\n",
    "    return pd.Series([x.min(), x.max()], index=['min', 'max'])\n",
    "\n",
    "print(df.apply(foo))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>applymap</code>\n",
    "\n",
    "-  Sometimes, we may want to apply a function to each element of the <code>**DataFrame**</code>.\n",
    "\n",
    "-  In other words, the input of the function to be applied is a scalar rather than a one-dimensional array.\n",
    "\n",
    "-  <code>**applymap**</code> is the function to be used!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "For example, suppose we want to apply a string format to each element of a DataFrame.\n",
    "\"\"\"\n",
    "# define the function to be applied\n",
    "str_format = lambda x: '%.2f' % x\n",
    "\n",
    "print(df.applymap(str_format))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>map</code>\n",
    "\n",
    "-  The <code>**applymap**</code> function above is implemented through the <code>**map**</code> function available for <code>**Series**</code> objects.\n",
    "\n",
    "-  In other words, <code>**map**</code> is the <code>**Series**</code>' corresponding of <code>**DataFrame**</code>'s <code>**applymap**</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Now the function `str_format` is passed as input to Series' `map` function\n",
    "df['e'].map(str_format)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>**apply**</code>, <code>**applymap**</code>, and <code>**map**</code>\n",
    "\n",
    "\n",
    "Summing up, <code>**apply**</code> works on a row/column basis of a <code>**DataFrame**</code>, <code>**applymap**</code> works element-wise on a <code>**DataFrame**</code>, and <code>**map**</code> works element-wise on a <code>**Series**</code>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sorting by Index Using <code>sort_index</code>\n",
    "\n",
    "-  To sort lexicographically by row or column index, use the <code>**sort_index**</code> method, which returns a **new**, sorted object (i.e., not *in-place* sorting).\n",
    "\n",
    "-  <code>**sort_index**</code> works both for <code>**Series**</code> and <code>**DataFrame**</code> objects, with the latter we can sort by index on either axis.\n",
    "\n",
    "-  By default, data is sorted in **ascending order**, but can be sorted in descending order as well by calling the <code>**sort_index**</code> method with <code>**ascending=False**</code> argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Usage of sort_index in Series objects.\n",
    "\"\"\"\n",
    "# Define a Series object\n",
    "s = pd.Series(range(4), index=['d', 'a', 'c', 'b'])\n",
    "print(\"Original Series:\\n{}\".format(s))\n",
    "print()\n",
    "print(\"Sort Series by index:\\n{}\".format(s.sort_index()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Using sort_index with DataFrame objects, we can specify which axis the index refers to.\n",
    "\"\"\"\n",
    "# Create a DataFrame object\n",
    "df = pd.DataFrame(np.arange(8).reshape((2, 4)), index=['three', 'one'],\n",
    "                     columns=['d', 'a', 'b', 'c'])\n",
    "print(\"Original DataFrame:\\n{}\".format(df))\n",
    "print()\n",
    "# By default, sort_index operates on axis 0 (i.e., rows)\n",
    "print(\"Sort DataFrame by row index:\\n{}\".format(df.sort_index()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "print(\"Original DataFrame:\\n{}\".format(df))\n",
    "print()\n",
    "# sort_index can also operate on axis 1 (i.e., columns)\n",
    "print(\"Sort DataFrame by column index:\\n{}\".format(df.sort_index(axis=1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sorting by Values Using <code>sort_values</code> \n",
    "\n",
    "-  Both <code>**Series**</code> and <code>**DataFrame**</code> objects can be sorted by **values** using the <code>**sort_values**</code> method.\n",
    "\n",
    "-  Sorting the values of a <code>**Series**</code> behaves as expected; by convention, any missing values (<code>**NaN**</code>) are placed at the end of the sorted object.\n",
    "\n",
    "-  When sorting a <code>**DataFrame**</code>, we can use one or more columns as the sort keys. To do so, we need to pass one or more column names as input arguments of <code>**sort_values**</code>.\n",
    "\n",
    "-  Again, sorting happens to be in **ascending order** by default but this can be changed using the <code>**ascending=False**</code> argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Apply sort_values to a Seres object.\n",
    "\"\"\"\n",
    "# Create a Series object\n",
    "s = pd.Series([4, 7, -3, 2])\n",
    "print(\"Original Series:\\n{}\".format(s))\n",
    "print()\n",
    "print(\"Sort Series by values:\\n{}\".format(s.sort_values()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Apply sort_values to a Seres object which contains missing values.\n",
    "\"\"\"\n",
    "# Create a Series object with missing values\n",
    "s = pd.Series([4, np.nan, 5, -1, np.nan, 2])\n",
    "print(\"Original Series:\\n{}\".format(s))\n",
    "print()\n",
    "print(\"Sort Series by values:\\n{}\".format(s.sort_values()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Apply sort_values to a DataFrame object.\n",
    "\"\"\"\n",
    "# Create a DataFrame object\n",
    "df = pd.DataFrame({'b': [4, 7, -3, 2], 'a': [0, 1, 0, 1]})\n",
    "print(\"Original DataFrame:\\n{}\".format(df))\n",
    "print()\n",
    "# Sort values by a single column\n",
    "print(\"Sort DataFrame by values contained in column 'b':\\n{}\"\n",
    "      .format(df.sort_values(by='b')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Sort values by multiple columns\n",
    "print(\"Sort DataFrame by values contained in 2 columns 'a' and 'b':\\n{}\"\n",
    "      .format(df.sort_values(by=['a','b'])))\n",
    "print()\n",
    "print(\"Sort DataFrame by values contained in 2 columns 'a' and 'b' (not ascending):\\n{}\"\n",
    "      .format(df.sort_values(by=['a','b'], ascending=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Summarizing and Computing Descriptive Statistics\n",
    "\n",
    "-  <code>**pandas**</code> objects are equipped with a set of common mathematical and statistical methods. \n",
    "\n",
    "-  Most of these fall into the category of **reductions** or **summary statistics**\n",
    "\n",
    "-  Methods that extract a single value (like the **sum** or **mean**) from a <code>**Series**</code> or a <code>**Series**</code> of values from the rows or columns of a <code>**DataFrame**</code>. \n",
    "\n",
    "-  Compared with the similar methods of <code>**numpy**</code> arrays, they have built-in handling for missing data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Consider a DataFrame object as follows\n",
    "df = pd.DataFrame([[1.4, np.nan], [7.1, -4.5], [np.nan, np.nan], [0.75, -1.3]], \n",
    "                  index=['a', 'b', 'c', 'd'], columns=['one', 'two'])\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# By default, aggregation function operates on axis 0 (i.e., rows) as usual\n",
    "# For example, calling the sum method returns a Series containing column-wise sums.\n",
    "print(\"Column-wise sum:\\n{}\".format(df.sum()))\n",
    "print()\n",
    "# Instead, if we want row-wise sum, we can of course specify axis=1.\n",
    "print(\"Row-wise sum:\\n{}\".format(df.sum(axis=1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Note that NaN values are excluded unless the entire slice (row or column) is NaN. \n",
    "# This can be disabled using the skipna option.\n",
    "print(\"Row-wise Average including NaN:\\n{}\".format(df.mean(axis=1, skipna=False)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "A useful method is 'describe' which provides multiple summary statistics in one shot.\n",
    "\"\"\"\n",
    "print(df.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "If the Series or DataFrame object does not contain numeric data \n",
    "other statistics are produced.\n",
    "\"\"\"\n",
    "# Let's create a Series object containing non-numeric data\n",
    "s = pd.Series(['a', 'a', 'b', 'c'] * 4)\n",
    "print(\"Original Series:\\n{}\".format(s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Now, let's call the describe method\n",
    "print(s.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Table of Descriptive and Summary Statistics (1 of 2)\n",
    "\n",
    "<img src=\"./img/pd_aggregate_stats_1.png\" width=\"50%\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Table of Descriptive and Summary Statistics (2 of 2)\n",
    "\n",
    "<img src=\"./img/pd_aggregate_stats_2.png\" width=\"50%\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Unique Values, Value Counts, and Membership\n",
    "\n",
    "-  <code>**unique**</code> is a function that gives you an array of the unique values in a <code>**Series**</code>.\n",
    "\n",
    "-  The unique values are not necessarily returned in sorted order, but could be sorted afterwards (e.g., by calling <code>**np.sort()**</code>).\n",
    "\n",
    "-  <code>**value_counts**</code> computes a <code>**Series**</code> containing value frequencies.\n",
    "\n",
    "-  <code>**isin**</code> performs a vectorized set membership check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Usage of unique method.\n",
    "\"\"\"\n",
    "# Create a Series object\n",
    "s = pd.Series(['c', 'a', 'd', 'a', 'a', 'b', 'b', 'c', 'c'])\n",
    "# Get unique values and store them in a variable\n",
    "uniques = s.unique()\n",
    "print(\"Unique values are: {}\".format(uniques))\n",
    "print(\"Unique sorted values are: {}\".format(np.sort(uniques)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Usage of value_counts method.\n",
    "\"\"\"\n",
    "# Either as a method\n",
    "print(\"Value counts:\\n{}\".format(s.value_counts(sort=False)))\n",
    "print()\n",
    "# Or as a built-in pandas function\n",
    "print(\"Value counts:\\n{}\".format(pd.value_counts(s, sort=False)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Usage of isin method.\n",
    "\"\"\"\n",
    "# Create a boolean mask\n",
    "mask = s.isin(['b', 'c'])\n",
    "print(\"Boolean mask:\\n{}\".format(mask))\n",
    "print()\n",
    "print(\"Accessing elements of the Series object using the mask:\\n{}\".format(s[mask]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Many times, it will be useful to compute a histogram on multiple columns in a DataFrame.\n",
    "\"\"\"\n",
    "data = pd.DataFrame({'Q1': [1, 3, 4, 3, 4],\n",
    "                     'Q2': [2, 3, 1, 2, 3],\n",
    "                     'Q3': [1, 5, 2, 4, 4]})\n",
    "\n",
    "print(\"Original dataset:\\n{}\".format(data))\n",
    "print()\n",
    "# Passing pandas.value_counts to this DataFrame's apply function gives the following\n",
    "freq = data.apply(pd.value_counts).fillna(0)\n",
    "print(\"Column-wise frequency table:\\n{}\".format(freq))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# If we specify axis=1 the frequency is computed row-wise\n",
    "freq = data.apply(pd.value_counts, axis=1).fillna(0)\n",
    "print(\"Row-wise frequency table:\\n{}\".format(freq))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Table of Unique, Value Counts, and Set Membership Methods\n",
    "\n",
    "<img src=\"./img/pd_freq.png\" width=\"50%\" align=\"center\"/>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}