Best_Practices_To_Use_Pandas_Efficiently_As_A_Data_Scientist.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Best Practices To Use Pandas Efficiently As A Data Scientist"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Practicing [best practices](https://github.com/youssefHosni/Efficient-Python-for-Data-Scientists)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import time\n",
    "import numpy as np"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1st dataset:\n",
    "In each poker round, each player has five cards in hand, each one characterized by its symbol, which can be either hearts, diamonds, clubs, or spades, and its rank, which ranges from 1 to 13. The dataset consists of every possible combination of five cards one person can possess.\n",
    "\n",
    "* Sn: symbol of the n-th card where: 1 (Hearts), 2 (Diamonds), 3 (Clubs), 4 (Spades)\n",
    "* Rn: rank of the n-th card where: 1 (Ace), 2–10, 11 (Jack), 12 (Queen), 13 (King)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>S1</th>\n",
       "      <th>R1</th>\n",
       "      <th>S2</th>\n",
       "      <th>R2</th>\n",
       "      <th>S3</th>\n",
       "      <th>R3</th>\n",
       "      <th>S4</th>\n",
       "      <th>R4</th>\n",
       "      <th>S5</th>\n",
       "      <th>R5</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>13</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>13</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   S1  R1  S2  R2  S3  R3  S4  R4  S5  R5  Class\n",
       "0   1  10   1  11   1  13   1  12   1   1      9\n",
       "1   2  11   2  13   2  10   2  12   2   1      9\n",
       "2   3  12   3  11   3  13   3  10   3   1      9\n",
       "3   4  10   4  11   4   1   4  13   4  12      9\n",
       "4   4   1   4  13   4  12   4  11   4  10      9"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poker_data = pd.read_csv('https://raw.githubusercontent.com/youssefHosni/Efficient-Python-for-Data-Scientists/main/Datasets/poker_hand.csv')\n",
    "poker_data.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2nd dataset:\n",
    "The dataset includes, among other information, the most popular names in the US by year, gender, and ethnicity. For example, the name Chloe was ranked second in popularity among all female newborns of Asian and Pacific Islander ethnicity in 2011."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year of Birth</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Child's First Name</th>\n",
       "      <th>Count</th>\n",
       "      <th>Rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
       "      <td>SOPHIA</td>\n",
       "      <td>119</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
       "      <td>CHLOE</td>\n",
       "      <td>106</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
       "      <td>EMILY</td>\n",
       "      <td>93</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
       "      <td>OLIVIA</td>\n",
       "      <td>89</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
       "      <td>EMMA</td>\n",
       "      <td>75</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Year of Birth  Gender                   Ethnicity Child's First Name  \\\n",
       "0           2011  FEMALE  ASIAN AND PACIFIC ISLANDER             SOPHIA   \n",
       "1           2011  FEMALE  ASIAN AND PACIFIC ISLANDER              CHLOE   \n",
       "2           2011  FEMALE  ASIAN AND PACIFIC ISLANDER              EMILY   \n",
       "3           2011  FEMALE  ASIAN AND PACIFIC ISLANDER             OLIVIA   \n",
       "4           2011  FEMALE  ASIAN AND PACIFIC ISLANDER               EMMA   \n",
       "\n",
       "   Count  Rank  \n",
       "0    119     1  \n",
       "1    106     2  \n",
       "2     93     3  \n",
       "3     89     4  \n",
       "4     75     5  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names = pd.read_csv('https://raw.githubusercontent.com/youssefHosni/Efficient-Python-for-Data-Scientists/main/Datasets/Popular_Baby_Names.csv')\n",
    "names.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3th datase: For each customer, we have various characteristics, including the total amount paid, the tips left to the waiter, the day of the week, and the time of the day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>sex</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>16.99</td>\n",
       "      <td>1.01</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.34</td>\n",
       "      <td>1.66</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21.01</td>\n",
       "      <td>3.50</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23.68</td>\n",
       "      <td>3.31</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>24.59</td>\n",
       "      <td>3.61</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   total_bill   tip     sex smoker  day    time  size\n",
       "0       16.99  1.01  Female     No  Sun  Dinner     2\n",
       "1       10.34  1.66    Male     No  Sun  Dinner     3\n",
       "2       21.01  3.50    Male     No  Sun  Dinner     3\n",
       "3       23.68  3.31    Male     No  Sun  Dinner     2\n",
       "4       24.59  3.61  Female     No  Sun  Dinner     4"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "restaurant = pd.read_csv('https://raw.githubusercontent.com/youssefHosni/Efficient-Python-for-Data-Scientists/main/Datasets/restaurant_data.csv')\n",
    "restaurant.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  1. The computational time"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First using list comprehension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using the list_comprehension: 0.10726189613342285 sec\n"
     ]
    }
   ],
   "source": [
    "#using List comprehension \n",
    "list_comp_start_time = time.time()\n",
    "result = [i*i for i in range(0,1000000)]\n",
    "list_comp_end_time = time.time()\n",
    "print(\"Time using the list_comprehension: {} sec\".format(list_comp_end_time -\n",
    "list_comp_start_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using the for loop: 0.24948501586914062 sec\n"
     ]
    }
   ],
   "source": [
    "# Using For loop\n",
    "for_loop_start_time= time.time()\n",
    "result=[]\n",
    "for i in range(0,1000000):\n",
    "  result.append(i*i)\n",
    "for_loop_end_time= time.time()\n",
    "print(\"Time using the for loop: {} sec\".format(for_loop_end_time - for_loop_start_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another example: calculate the sum by useing brute force in which we will add one by one to a million."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using the formula: 0.0 sec\n"
     ]
    }
   ],
   "source": [
    "def sum_formula(N):\n",
    "  return N*(N+1)/2\n",
    "  \n",
    "# Using the formula\n",
    "formula_start_time = time.time()\n",
    "formula_result = sum_formula(1000000)\n",
    "formula_end_time = time.time()\n",
    "\n",
    "print(\"Time using the formula: {} sec\".format(formula_end_time - formula_start_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another more efficient method is to use a formula to calculate it. When we want to calculate the sum of all the integer numbers from 1 up to a number, let’s say N, we can multiply N by N+1, and then divide by 2, and this will give us the result we want. This problem was actually given to some students back in Germany in the 19th century, and a bright student called Carl-Friedrich Gauss devised this formula to solve the problem in seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using brute force: 0.07296013832092285 sec\n"
     ]
    }
   ],
   "source": [
    "def sum_brute_force(N):\n",
    "  res = 0\n",
    "  for i in range(1,N+1):\n",
    "    res+=i\n",
    "  return res\n",
    "\n",
    "# Using brute force\n",
    "bf_start_time = time.time()\n",
    "bf_result = sum_brute_force(1000000)\n",
    "bf_end_time = time.time()\n",
    "\n",
    "print(\"Time using brute force: {} sec\".format(bf_end_time - bf_start_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Selecting & Replacing Values Effectively\n",
    "These two tasks are selecting specific and random rows and columns efficiently and the usage of the replace() function for replacing one or multiple values using lists and dictionaries"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.1. Selecting Rows & Columns Efficiently using .iloc[] & .loc[]\n",
    "\n",
    "We will use iloc[] for the index number locator and .loc[] for the index name locator.\n",
    "In the example below we will select the first 500 rows of the poker dataset. Firstly by using the .loc[] function, and then by using the .iloc[] function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using .loc[] : 0.006474971771240234 sec\n"
     ]
    }
   ],
   "source": [
    "# Specify the range of rows to select\n",
    "rows = range(0, 500)\n",
    "# Time selecting rows using .loc[]\n",
    "loc_start_time = time.time()\n",
    "poker_data.loc[rows]\n",
    "loc_end_time = time.time()\n",
    "print(\"Time using .loc[] : {} sec\".format(loc_end_time - loc_start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using .iloc[]: 0.0009987354278564453 sec\n"
     ]
    }
   ],
   "source": [
    "# Specify the range of rows to select\n",
    "rows = range(0, 500)\n",
    "# Time selecting rows using .iloc[]\n",
    "iloc_start_time = time.time()\n",
    "poker_data.iloc[rows]\n",
    "iloc_end_time = time.time()\n",
    "print(\"Time using .iloc[]: {} sec\".format(iloc_end_time - iloc_start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use them to select columns not only rows. In the next example, we will select the first three columns using both methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using .iloc[]: 0.0009984970092773438 sec\n"
     ]
    }
   ],
   "source": [
    "iloc_start_time = time.time()\n",
    "poker_data.iloc[:,:3]\n",
    "iloc_end_time = time.time()\n",
    "print(\"Time using .iloc[]: {} sec\".format(iloc_end_time - iloc_start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using selection by name: 0.008021354675292969 sec\n"
     ]
    }
   ],
   "source": [
    "names_start_time = time.time()\n",
    "poker_data[['S1', 'R1', 'S2']]\n",
    "names_end_time = time.time()\n",
    "print(\"Time using selection by name: {} sec\".format(names_end_time - names_start_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.2. Replacing Values in a DataFrame Effectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year of Birth</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Child's First Name</th>\n",
       "      <th>Count</th>\n",
       "      <th>Rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
       "      <td>SOPHIA</td>\n",
       "      <td>119</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
       "      <td>CHLOE</td>\n",
       "      <td>106</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
       "      <td>EMILY</td>\n",
       "      <td>93</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
       "      <td>OLIVIA</td>\n",
       "      <td>89</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>ASIAN AND PACIFIC ISLANDER</td>\n",
       "      <td>EMMA</td>\n",
       "      <td>75</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Year of Birth  Gender                   Ethnicity Child's First Name  \\\n",
       "0           2011  FEMALE  ASIAN AND PACIFIC ISLANDER             SOPHIA   \n",
       "1           2011  FEMALE  ASIAN AND PACIFIC ISLANDER              CHLOE   \n",
       "2           2011  FEMALE  ASIAN AND PACIFIC ISLANDER              EMILY   \n",
       "3           2011  FEMALE  ASIAN AND PACIFIC ISLANDER             OLIVIA   \n",
       "4           2011  FEMALE  ASIAN AND PACIFIC ISLANDER               EMMA   \n",
       "\n",
       "   Count  Rank  \n",
       "0    119     1  \n",
       "1    106     2  \n",
       "2     93     3  \n",
       "3     89     4  \n",
       "4     75     5  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['FEMALE', 'MALE'], dtype=object)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names['Gender'].unique()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the female gender is represented with two values both uppercase and lowercase. This is very common in real data and an easy way to do so is to replace one of the values with the other to keep it consistent throughout the whole dataset. There are two ways to do it the first one is simply defining which values we want to replace, and then what we want to replace them with. This is shown in the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Replace values using .loc[]: 0.008749961853027344 sec\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dovile.meskauskaite\\AppData\\Local\\Temp\\ipykernel_20224\\1448346069.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  names['Gender'].loc[names.Gender=='female'] = 'FEMALE'\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "names['Gender'].loc[names.Gender=='female'] = 'FEMALE'\n",
    "end_time = time.time()\n",
    "\n",
    "pandas_time = end_time - start_time\n",
    "print(\"Replace values using .loc[]: {} sec\".format(pandas_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second method is to use the panda's built-in function .replace() as shown in the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using replace(): 0.003954648971557617 sec\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "names['Gender'].replace('female', 'FEMALE', inplace=True)\n",
    "end_time = time.time()\n",
    "replace_time = end_time - start_time\n",
    "\n",
    "print(\"Time using replace(): {} sec\".format(replace_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there is a difference in time complexity with the built-in function 157% faster than using the .loc() method to find the rows and columns index of the values and replace it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The differnce: 121.25761138240792 %\n"
     ]
    }
   ],
   "source": [
    "print('The differnce: {} %'.format((pandas_time- replace_time )/replace_time*100))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also replace multiple values using lists. Our objective is to change all ethnicities classified as WHITE NON-HISPANIC or WHITE NON-HISP to WNH. Using the .loc[] function, we will locate babies of the ethnicities we are looking for, using the ‘or’ statement (which in Python is symbolized by the pipe). We will then assign the new value. As always, we also measure the CPU time needed for this operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results from the above operation calculated in 0.008122682571411133 seconds\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dovile.meskauskaite\\AppData\\Local\\Temp\\ipykernel_20224\\1006042807.py:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  names['Ethnicity'].loc[(names[\"Ethnicity\"] == 'WHITE NON HISPANIC') |\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "names['Ethnicity'].loc[(names[\"Ethnicity\"] == 'WHITE NON HISPANIC') |\n",
    "(names[\"Ethnicity\"] == 'WHITE NON HISP')] = 'WNH'\n",
    "\n",
    "end_time = time.time()\n",
    "pandas_time= end_time - start_time\n",
    "print(\"Results from the above operation calculated in %s seconds\" %(pandas_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also do the same operation using the .replace() pandas built-in function as the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using .replace(): 0.015821456909179688 sec\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "names['Ethnicity'].replace(['WHITE NON HISPANIC','WHITE NON HISP'],\n",
    "'WNH', inplace=True)\n",
    "\n",
    "end_time = time.time()\n",
    "replace_time = end_time - start_time\n",
    "\n",
    "print(\"Time using .replace(): {} sec\".format(replace_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that again using the .replace() method is much faster than using the .loc[] method. To have better intuition of how much faster it is let's run the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The differnce: -48.66033755274262 %\n"
     ]
    }
   ],
   "source": [
    "print('The differnce: {} %'.format((pandas_time- replace_time )/replace_time*100))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The .replace() method is 87% faster than using the .loc[] method. If your data is huge and need a lot of cleaning this tip will decrease the computational time of your data cleaning and makes your pandas code much faster and hence more efficient.\n",
    "\n",
    "Finally, we can also use dictionaries to replace both single and multiple values in your DataFrame. This will be very helpful if you would like to multiple replacing functions in one command.\n",
    "\n",
    "We’re going to use dictionaries to replace every male’s gender with BOY and every female’s gender with GIRL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using .replace() with dictionary: 0.006957530975341797 sec\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "names['Gender'].replace({'MALE':'BOY', 'FEMALE':'GIRL', 'female': 'girl'}, inplace=True)\n",
    "end_time = time.time()\n",
    "dict_time = end_time - start_time\n",
    "print(\"Time using .replace() with dictionary: {} sec\".format(dict_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using multiple .replace(): 0.0029926300048828125 sec\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "names['Gender'].replace('MALE', 'BOY', inplace=True)\n",
    "names['Gender'].replace('FEMALE', 'GIRL', inplace=True)\n",
    "names['Gender'].replace('female', 'girl', inplace=True)\n",
    "\n",
    "end_time = time.time()\n",
    "\n",
    "list_time = end_time - start_time\n",
    "print(\"Time using multiple .replace(): {} sec\".format(list_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The differnce: -56.98718388047427 %\n"
     ]
    }
   ],
   "source": [
    "print('The differnce: {} %'.format((list_time- dict_time )/dict_time*100))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could do the same thing with lists, but it’s more verbose. If we compare both methods, we can see that dictionaries run approximately 22% faster. In general, working with dictionaries in Python is very efficient compared to lists: looking through a list requires a pass in every element of the list while looking at a dictionary directs instantly to the key that matches the entry. The comparison is a little unfair though since both structures serve different purposes.\n",
    "\n",
    "Using dictionaries allows you to replace the same values on several different columns. In all the previous examples, we specified the column from which the values to replace came. We’re now going to replace several values from the same column with one common value. We want to classify all ethnicities into three big categories: Black, Asian and White. The syntax again is very simple. We use nested dictionaries here: the outer key is the column in which we want to replace values. The value of this outer key is another dictionary, where the keys are the ethnicities to replace, and the values for the new ethnicity (Black, Asian or White)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using .replace() with dictionary: 0.006242990493774414 sec\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "names.replace({'Ethnicity': {'ASIAN AND PACI': 'ASIAN', 'ASIAN AND PACIFIC ISLANDER': 'ASIAN',\n",
    "'BLACK NON HISPANIC': 'BLACK', 'BLACK NON HISP': 'BLACK',\n",
    "'WHITE NON HISPANIC': 'WHITE', 'WHITE NON HISP': 'WHITE'}})\n",
    "print(\"Time using .replace() with dictionary: {} sec\".format (time.time() - start_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.3. Summary of best practices for selecting and replacing values\n",
    "* Selecting rows and columns is faster using the .iloc[] function. So it is better to use unless it is easier or more convenient to use .loc[] and the speed is not a priority or you are just doing it once.\n",
    "* Using the built-in replace() function is much faster than just using conventional methods.\n",
    "* Replacing multiple values using python dictionaries is faster than using lists."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Iterate Effectively Through Pandas DataFrame"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a data scientist, you will need to iterate through your dataframe extensively, especially in the data preparation and exploration phase, so it is important to be able to do this efficiently, as it will save you much time and give space for more important work. We will walk through three methods to make your loops much faster and more efficient:\n",
    "\n",
    "* Looping using the .iterrows() function\n",
    "* Looping using the .apply() function\n",
    "* Vectorization"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.1. Looping effectively using .iterrows()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we talk about how to use the .iterrows() function to improve the looping process, let’s refresh the notion of a generator function.\n",
    "\n",
    "Generators are a simple tool to create iterators. Inside the body of a generator, instead of return statements, you will find only yield() statements. There can be just one, or several yield() statements. Here, we can see a generator, city_name_generator(), that produces four city names. We assign the generator to the variable city_names for simplicity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def city_name_generator():\n",
    "  yield('New York')\n",
    "  yield('London')\n",
    "  yield('Tokyo')\n",
    "  yield('Sao Paolo')\n",
    "\n",
    "city_names = city_name_generator()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access the elements that the generator yields we can use Python’s next() function. Each time the next() command is used, the generator will produce the next value to yield, until there are no more values to yield. We here have 4 cities. Let’s run the next command four times and see what it returns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'New York'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(city_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'London'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(city_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Tokyo'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(city_names)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go back to the .iterrows() function. The .iterrows() function is a property of every pandas DataFrame. When called, it produces a list with two elements. We will use this generator to iterate through each line of our poker DataFrame. The first element is the index of the row, while the second element contains a pandas Series of each feature of the row: the Symbol and the Rank for each of the five cards. It is very similar to the notion of the enumerate() function, which when applied to a list, returns each element along with its index.\n",
    "\n",
    "The most intuitive way to iterate through a Pandas DataFrame is to use the range() function, which is often called crude looping. This is shown with the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using range(): 0.0019943714141845703 sec\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "for index in range(poker_data.shape[0]):\n",
    "  next\n",
    "print(\"Time using range(): {} sec\".format(time.time() - start_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One smarter way to iterate through a pandas DataFrame is to use the .iterrows() function, which is optimized for this task. We simply define the ‘for’ loop with two iterators, one for the number of each row and the other for all the values.\n",
    "\n",
    "Inside the loop, the next() command indicates that the loop moves to the next value of the iterator, without actually doing something."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using .iterrows(): 1.0295677185058594 sec\n"
     ]
    }
   ],
   "source": [
    "data_generator = poker_data.iterrows()\n",
    "start_time = time.time()\n",
    "for index, values in data_generator:\n",
    "  next\n",
    "print(\"Time using .iterrows(): {} sec\".format(time.time() - start_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comparing the two computational times we can also notice that the use of .iterrows() does not improve the speed of iterating through pandas DataFrame. It is very useful though when we need a cleaner way to use the values of each row while iterating through the dataset."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.2. Looping effectively using .apply()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will use the .apply() function to be able to perform a specific task while iterating through a pandas DataFrame. The .apply() function does exactly what it says; it applies another function to the whole DataFrame.\n",
    "\n",
    "The syntax of the .apply() function is simple: we create a mapping, using a lambda function in this case, and then declare the function we want to apply to every cell. Here, we’re applying the square root function to every cell of the DataFrame. In terms of speed, it matches the speed of just using the NumPy sqrt() function over the whole DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>S1</th>\n",
       "      <th>R1</th>\n",
       "      <th>S2</th>\n",
       "      <th>R2</th>\n",
       "      <th>S3</th>\n",
       "      <th>R3</th>\n",
       "      <th>S4</th>\n",
       "      <th>R4</th>\n",
       "      <th>S5</th>\n",
       "      <th>R5</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.162278</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.316625</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.605551</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.464102</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.414214</td>\n",
       "      <td>3.316625</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>3.605551</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>3.162278</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>3.464102</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.732051</td>\n",
       "      <td>3.464102</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>3.316625</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>3.605551</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>3.162278</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.162278</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.316625</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.605551</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.464102</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.605551</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.464102</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.316625</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.162278</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         S1        R1        S2        R2        S3        R3        S4  \\\n",
       "0  1.000000  3.162278  1.000000  3.316625  1.000000  3.605551  1.000000   \n",
       "1  1.414214  3.316625  1.414214  3.605551  1.414214  3.162278  1.414214   \n",
       "2  1.732051  3.464102  1.732051  3.316625  1.732051  3.605551  1.732051   \n",
       "3  2.000000  3.162278  2.000000  3.316625  2.000000  1.000000  2.000000   \n",
       "4  2.000000  1.000000  2.000000  3.605551  2.000000  3.464102  2.000000   \n",
       "\n",
       "         R4        S5        R5  Class  \n",
       "0  3.464102  1.000000  1.000000    3.0  \n",
       "1  3.464102  1.414214  1.000000    3.0  \n",
       "2  3.162278  1.732051  1.000000    3.0  \n",
       "3  3.605551  2.000000  3.464102    3.0  \n",
       "4  3.316625  2.000000  3.162278    3.0  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_sqrt = poker_data.apply(lambda x: np.sqrt(x), axis =0 )\n",
    "data_sqrt.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But what happens when the function of interest is taking more than one cell as an input? For example, what if we want to calculate the sum of the rank of all the cards in each hand? In this case, we will use the .apply() function the same way as we did before, but we need to add ‘axis=1’ at the end of the line to specify we’re applying the function to each row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using .apply(): 0.1598052978515625 sec\n"
     ]
    }
   ],
   "source": [
    "apply_start_time = time.time()\n",
    "poker_data[['R1', 'R2', 'R3', 'R4', 'R5']].apply(lambda x: sum(x), axis=1)\n",
    "apply_end_time = time.time()\n",
    "apply_time = apply_end_time - apply_start_time\n",
    "print(\"Time using .apply(): {} sec\".format(apply_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we will use the .iterrows() function we saw previously, and compare their efficiency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using .iterrows(): 1.0417206287384033 sec\n"
     ]
    }
   ],
   "source": [
    "for_loop_start_time = time.time()\n",
    "for ind, value in poker_data.iterrows():\n",
    "  sum([value[1], value[3], value[5], value[7], value[9]])\n",
    "for_loop_end_time = time.time()\n",
    "\n",
    "for_loop_time = for_loop_end_time - for_loop_start_time\n",
    "print(\"Time using .iterrows(): {} sec\".format(for_loop_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the .apply() function is significantly faster than the .iterrows() function, with a magnitude of around 400 percent, which is a massive improvement!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The differnce: 551.8686443712403 %\n"
     ]
    }
   ],
   "source": [
    "print('The differnce: {} %'.format((for_loop_time - apply_time) / apply_time * 100))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did with rows, we can do exactly the same thing for the columns; apply one function to each column. By replacing the axis=1 with axis=0, we can apply the sum function on every column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using .apply(): 0.01398777961730957 sec\n"
     ]
    }
   ],
   "source": [
    "apply_start_time = time.time()\n",
    "poker_data[['R1', 'R2', 'R3', 'R4', 'R5']].apply(lambda x: sum(x), axis=0)\n",
    "apply_end_time = time.time()\n",
    "apply_time = apply_end_time - apply_start_time\n",
    "print(\"Time using .apply(): {} sec\".format(apply_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By comparing the .apply() function with the native panda's function for summing over rows, we can see that pandas’ native .sum() functions perform the same operation faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using pandas: 0.02072882652282715 sec\n"
     ]
    }
   ],
   "source": [
    "pandas_start_time = time.time()\n",
    "poker_data[['R1', 'R1', 'R3', 'R4', 'R5']].sum(axis=0)\n",
    "pandas_end_time = time.time()\n",
    "pandas_time = pandas_end_time - pandas_start_time\n",
    "print(\"Time using pandas: {} sec\".format(pandas_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The differnce: -32.520156884395526 %\n"
     ]
    }
   ],
   "source": [
    "print('The differnce: {} %'.format((apply_time - pandas_time) / pandas_time * 100))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.3.Looping effectively using vectorization"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand how we can reduce the amount of iteration performed by the function, recall that the fundamental units of Pandas, DataFrames, and Series, are both based on arrays. Pandas perform more efficiently when an operation is performed to a whole array than to each value separately or sequentially. This can be achieved through vectorization. Vectorization is the process of executing operations on entire arrays.\n",
    "\n",
    "In the code below we want to calculate the sum of the ranks of all the cards in each hand. In order to do that, we slice the poker dataset keeping only the columns that contain the ranks of each card. Then, we call the built-in .sum() property of the DataFrame, using the parameter axis = 1 to denote that we want the sum for each row. In the end, we print the sum of the first five rows of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using pandas vectorization: 0.020956039428710938 sec\n"
     ]
    }
   ],
   "source": [
    "start_time_vectorization = time.time()\n",
    "\n",
    "poker_data[['R1', 'R2', 'R3', 'R4', 'R5']].sum(axis=1)\n",
    "end_time_vectorization = time.time()\n",
    "\n",
    "vectorization_time = end_time_vectorization  - start_time_vectorization\n",
    "print(\"Time using pandas vectorization: {} sec\".format(vectorization_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We saw previously various methods that perform functions applied to a DataFrame faster than simply iterating through all the rows of the DataFrame. Our goal is to find the most efficient method to perform this task.\n",
    "\n",
    "Using .iterrows() to loop through the DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using .iterrows() 1.000380039215088 seconds \n"
     ]
    }
   ],
   "source": [
    "data_generator = poker_data.iterrows()\n",
    "\n",
    "start_time_iterrows = time.time()\n",
    "\n",
    "for index, value in data_generator:\n",
    "  sum([value[1], value[3], value[5], value[7]])\n",
    "\n",
    "end_time_iterrows = time.time()\n",
    "iterrows_time = end_time_iterrows - start_time_iterrows\n",
    "print(\"Time using .iterrows() {} seconds \" .format(iterrows_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the .apply() mehtod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using apply() 0.1296522617340088 seconds\n"
     ]
    }
   ],
   "source": [
    "start_time_apply = time.time()\n",
    "poker_data[['R1', 'R2', 'R3', 'R4', 'R5']].apply(lambda x: sum(x),axis=1)\n",
    "end_time_apply = time.time()\n",
    "\n",
    "apply_time = end_time_apply - start_time_apply\n",
    "\n",
    "print(\"Time using apply() {} seconds\"  .format(apply_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comparing the time it takes to sum the ranks of all the cards in each hand using vectorization, the .iterrows() function, and the .apply() function, we can see that the vectorization method performs much better.\n",
    "\n",
    "We can also use another vectorization method to effectively iterate through the DataFrame which is using Numpy arrays to vectorize the DataFrame.\n",
    "\n",
    "The NumPy library, which defines itself as a “fundamental package for scientific computing in Python”, performs operations under the hood in optimized, pre-compiled C code. Similar to pandas working with arrays, NumPy operates on arrays called ndarrays. A major difference between Series and ndarrays is that ndarrays leave out many operations such as indexing, data type checking, etc. As a result, operations on NumPy arrays can be significantly faster than operations on pandas Series. NumPy arrays can be used in place of the pandas Series when the additional functionality offered by the pandas Series isn’t critical.\n",
    "\n",
    "For the problems we explore in this article, we could use NumPy ndarrays instead of the pandas series. The question at stake is whether this would be more efficient or not.\n",
    "\n",
    "Again, we will calculate the sum of the ranks of all the cards in each hand. We convert our rank arrays from pandas Series to NumPy arrays simply by using the .values method of pandas Series, which returns a pandas Series as a NumPy ndarray. As with vectorization on the series, passing the NumPy array directly into the function will lead pandas to apply the function to the entire vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using NumPy vectorization: 0.006017446517944336 sec\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "poker_data[['R1', 'R2', 'R3', 'R4', 'R5']].values.sum(axis=1)\n",
    "\n",
    "print(\"Time using NumPy vectorization: {} sec\" .format(time.time() - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using the pandas vectorization 0.004062652587890625 seconds\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "poker_data[['R1', 'R2', 'R3', 'R4', 'R5']].sum(axis=1)\n",
    "print(\"Time using the pandas vectorization %s seconds\" % (time.time() - start_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.4. Summary of best practices for looping through DataFrame\n",
    "* Using .iterrows() does not improve the speed of iterating through the DataFrame but it is more efficient.\n",
    "* The .apply() function performs faster when we want to iterate through all the rows of a pandas DataFrame, but is slower when we perform the same operation through a column.\n",
    "* Vectorizing over the pandas series achieves the overwhelming majority of optimization needs for everyday calculations. However, if speed is of the highest priority, we can call in reinforcements in the form of the NumPy Python library."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Transforming Data Efficevly using .groupby()\n",
    "In this last section of the article, we will use how to use the .groupby() function effectively to group the entries of a DataFrame according to the values of a specific feature. The .groupby() method is applied to a DataFrame and groups it according to a feature. Then, we can apply some simple or more complicated functions on that grouped object. This is a very important tool for every data scientist working on tabular or structured data as it will help you to manipulate data easily and in a more effective way."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.1. Common functions used with .groupby()\n",
    "\n",
    "One of the simplest methods to apply to an aggregated group is the .count(). In the example below we will apply this to the restaurant dataset. At first, we group the restaurant data according to whether the customer was a smoker or not. Then, we apply the .count() method. We obtain the count of smokers and non-smokers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        total_bill  tip  sex  day  time  size\n",
      "smoker                                       \n",
      "No             151  151  151  151   151   151\n",
      "Yes             93   93   93   93    93    93\n"
     ]
    }
   ],
   "source": [
    "restaurant_grouped = restaurant.groupby('smoker')\n",
    "print(restaurant_grouped.count())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is no surprise that we get the same results for all the features, as the .count() method counts the number of occurrences of each group in each feature. As there are no missing values in our data, the results should be the same in all columns.\n",
    "\n",
    "After grouping the entries of the DataFrame according to the values of a specific feature, we can apply any kind of transformation we are interested in. Here, we are going to apply the z score, a normalization transformation, which is the distance between each value and the mean, divided by the standard deviation. This is a very useful transformation in statistics, often used with the z-test in standardized testing. To apply this transformation to the grouped object, we just need to call the .transform() method containing the lambda transformation we defined.\n",
    "\n",
    "This time, we will group according to the type of meal: was it a dinner or a lunch? As the z-score transformation is a group-related transformation, the resulting table is just the original table. For each element, we subtract the mean and divide it by the standard deviation of the group it belongs to. We can also see that numerical transformation are applied only to numerical features of the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dovile.meskauskaite\\AppData\\Local\\Temp\\ipykernel_20224\\2071484734.py:5: FutureWarning: Dropping invalid columns in DataFrameGroupBy.transform is deprecated. In a future version, a TypeError will be raised. Before calling .transform, select only columns which should be valid for the function.\n",
      "  restaurant_transformed = restaurant_grouped.transform(zscore)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.416446</td>\n",
       "      <td>-1.457045</td>\n",
       "      <td>-0.692873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.143855</td>\n",
       "      <td>-1.004475</td>\n",
       "      <td>0.405737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.023282</td>\n",
       "      <td>0.276645</td>\n",
       "      <td>0.405737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.315339</td>\n",
       "      <td>0.144355</td>\n",
       "      <td>-0.692873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.414880</td>\n",
       "      <td>0.353234</td>\n",
       "      <td>1.504347</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   total_bill       tip      size\n",
       "0   -0.416446 -1.457045 -0.692873\n",
       "1   -1.143855 -1.004475  0.405737\n",
       "2    0.023282  0.276645  0.405737\n",
       "3    0.315339  0.144355 -0.692873\n",
       "4    0.414880  0.353234  1.504347"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zscore = lambda x: (x - x.mean() ) / x.std()\n",
    "\n",
    "restaurant_grouped = restaurant.groupby('time')\n",
    "\n",
    "restaurant_transformed = restaurant_grouped.transform(zscore)\n",
    "restaurant_transformed.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the transform() method simplifies things a lot, is it actually more efficient than using native Python code? As we did before, we first group our data, this time according to sex. Then we apply the z-score transformation we applied before, measuring its efficiency. We omit the code for measuring the time of each operation here, as you are already familiar with this. We can see that with the use of the transform() function, we achieve a massive speed improvement. On top of that, we’re only using one line to perform the operation of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dovile.meskauskaite\\AppData\\Local\\Temp\\ipykernel_20224\\1730956826.py:1: FutureWarning: Dropping invalid columns in DataFrameGroupBy.transform is deprecated. In a future version, a TypeError will be raised. Before calling .transform, select only columns which should be valid for the function.\n",
      "  restaurant.groupby('sex').transform(zscore)\n",
      "C:\\Users\\dovile.meskauskaite\\AppData\\Local\\Temp\\ipykernel_20224\\1730956826.py:3: FutureWarning: The default value of numeric_only in DataFrameGroupBy.mean is deprecated. In a future version, numeric_only will default to False. Either specify numeric_only or select only columns which should be valid for the function.\n",
      "  mean_female = restaurant.groupby('sex').mean()['total_bill']['Female']\n",
      "C:\\Users\\dovile.meskauskaite\\AppData\\Local\\Temp\\ipykernel_20224\\1730956826.py:4: FutureWarning: The default value of numeric_only in DataFrameGroupBy.mean is deprecated. In a future version, numeric_only will default to False. Either specify numeric_only or select only columns which should be valid for the function.\n",
      "  mean_male = restaurant.groupby('sex').mean()['total_bill']['Male']\n",
      "C:\\Users\\dovile.meskauskaite\\AppData\\Local\\Temp\\ipykernel_20224\\1730956826.py:5: FutureWarning: The default value of numeric_only in DataFrameGroupBy.std is deprecated. In a future version, numeric_only will default to False. Either specify numeric_only or select only columns which should be valid for the function.\n",
      "  std_female = restaurant.groupby('sex').std()['total_bill']['Female']\n",
      "C:\\Users\\dovile.meskauskaite\\AppData\\Local\\Temp\\ipykernel_20224\\1730956826.py:6: FutureWarning: The default value of numeric_only in DataFrameGroupBy.std is deprecated. In a future version, numeric_only will default to False. Either specify numeric_only or select only columns which should be valid for the function.\n",
      "  std_male = restaurant.groupby('sex').std()['total_bill']['Male']\n",
      "C:\\Users\\dovile.meskauskaite\\AppData\\Local\\Temp\\ipykernel_20224\\1730956826.py:10: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  restaurant.iloc[i][0] = (restaurant.iloc[i][0] - mean_female)/std_female\n",
      "C:\\Users\\dovile.meskauskaite\\AppData\\Local\\Temp\\ipykernel_20224\\1730956826.py:12: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  restaurant.iloc[i][0] = (restaurant.iloc[i][0] - mean_male)/std_male\n"
     ]
    }
   ],
   "source": [
    "restaurant.groupby('sex').transform(zscore)\n",
    "\n",
    "mean_female = restaurant.groupby('sex').mean()['total_bill']['Female']\n",
    "mean_male = restaurant.groupby('sex').mean()['total_bill']['Male']\n",
    "std_female = restaurant.groupby('sex').std()['total_bill']['Female']\n",
    "std_male = restaurant.groupby('sex').std()['total_bill']['Male']\n",
    "\n",
    "for i in range(len(restaurant)):\n",
    "  if restaurant.iloc[i][2] == 'Female':\n",
    "    restaurant.iloc[i][0] = (restaurant.iloc[i][0] - mean_female)/std_female\n",
    "  else:\n",
    "    restaurant.iloc[i][0] = (restaurant.iloc[i][0] - mean_male)/std_male"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.2. Missing value imputation using .groupby() & .transform()\n",
    "\n",
    "Now that we have seen why and how to use the transform() function on a grouped pandas object, we will address a very specific task that is imputing missing value.\n",
    "\n",
    "Before we actually see how we can use the transform() function for missing value imputation, we will see how many missing values there are in our variable of interest in each of the groups. We can see below the number of data points in each of the “time” feature and they are 176+68 = 244."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time\n",
       "Dinner    176\n",
       "Lunch      68\n",
       "Name: total_bill, dtype: int64"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prior_counts = restaurant.groupby('time')\n",
    "prior_counts['total_bill'].count()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will create a restaurant_nan dataset, in which the total bill of 10% random observations was set to NaN using the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "p = 0.1 #percentage missing data required\n",
    "\n",
    "mask = np.random.choice([np.nan,1], size=len(restaurant), p=[p,1-p])\n",
    "restaurant_nan = restaurant.copy()\n",
    "restaurant_nan['total_bill'] =  restaurant_nan['total_bill'] * mask"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's print the number of data points in each of the “time” feature and we can see that they are now 155+62 = 217. Since the total data points we have are 244 then the missing data points are 24 which is equal to 10%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time\n",
       "Dinner    176\n",
       "Lunch      68\n",
       "Name: total_bill, dtype: int64"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prior_counts = restaurant.groupby('time')\n",
    "prior_counts['total_bill'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time\n",
       "Dinner    156\n",
       "Lunch      63\n",
       "Name: total_bill, dtype: int64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_counts = restaurant_nan.groupby('time')\n",
    "missing_counts['total_bill'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time using the pandas vectorization 0.010112762451171875 seconds\n"
     ]
    }
   ],
   "source": [
    "# Missing value imputation\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "missing_trans = lambda x: x.fillna(x.mean())\n",
    "restaurant_nan_grouped = restaurant_nan.groupby('time')['total_bill']\n",
    "restaurant_nan_grouped.transform(missing_trans)\n",
    "print(\"Time using the pandas vectorization %s seconds\" % (time.time() - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results from the above operation calculated in 0.0813286304473877 seconds\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "mean_din = restaurant_nan.loc[restaurant_nan.time =='Dinner']['total_bill'].mean()\n",
    "mean_lun = restaurant_nan.loc[restaurant_nan.time == 'Lunch']['total_bill'].mean()\n",
    "\n",
    "for row in range(len(restaurant_nan)):\n",
    "  if restaurant_nan.iloc[row]['time'] == 'Dinner':\n",
    "    restaurant_nan.loc[row, 'total_time'] = mean_din\n",
    "  else:\n",
    "    restaurant_nan.loc[row, 'total_time'] = mean_lun\n",
    "print(\"Results from the above operation calculated in %s seconds\" % (time.time() - start_time))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.3. Data filtration using the .groupby() & .filter()\n",
    "\n",
    "Now we will discuss how we can use the filter() function on a grouped pandas object. This allows us to include only a subset of those groups, based on some specific conditions.\n",
    "\n",
    "Often, after grouping the entries of a DataFrame according to a specific feature, we are interested in including only a subset of those groups, based on some conditions. Some examples of filtration conditions are the number of missing values, the mean of a specific feature, or the number of occurrences of the group in the dataset.\n",
    "\n",
    "We are interested in finding the mean amount of tips given, on the days when the mean amount paid to the waiter is more than 20 USD. The .filter() function accepts a lambda function that operates on a DataFrame of each of the groups. In this example, the lambda function selects “total_bill” and checks that the mean() is greater than 20. If that lambda function returns True, then the mean() of the tip is calculated. If we compare the total mean of the tips, we can see that there is a difference between the two values, meaning that the filtering was performed correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "restaurant_grouped = restaurant.groupby('day')\n",
    "filter_trans = lambda x : x['total_bill'].mean() > 20\n",
    "restaurant_filtered = restaurant_grouped.filter(filter_trans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.115276073619632\n"
     ]
    }
   ],
   "source": [
    "print(restaurant_filtered['tip'].mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.99827868852459\n"
     ]
    }
   ],
   "source": [
    "print(restaurant['tip'].mean())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we attempt to perform this operation without using groupby(), we end up with this inefficient code. At first, we use a list comprehension to extract the entries of the DataFrame that refer to days that have a mean meal greater than $20 and then use a for loop to append them into a list and calculate the mean. It might seem very intuitive, but as we see, it’s also very inefficient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dovile.meskauskaite\\AppData\\Local\\Temp\\ipykernel_20224\\3026924126.py:6: FutureWarning: The series.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  restaurant_filtered=restaurant_filtered.append(j,ignore_index=True)\n"
     ]
    }
   ],
   "source": [
    "t=[restaurant.loc[restaurant['day'] == i]['tip'] for i in restaurant['day'].unique()\n",
    "  if restaurant.loc[restaurant['day'] == i]['total_bill'].mean()>20]\n",
    "restaurant_filtered = t[0]\n",
    "\n",
    "for j in t[1:]:\n",
    "  restaurant_filtered=restaurant_filtered.append(j,ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.1152760736196323\n"
     ]
    }
   ],
   "source": [
    "print(restaurant_filtered.mean())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Summary of Best Practices"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Selecting rows and columns is faster using the .iloc[] function. So it is better to use unless it is easier or more convenient to use .loc[] and the speed is not a priority or you are just doing it once.\n",
    "* Using the built-in replace() function is much faster than just using conventional methods.\n",
    "* Replacing multiple values using python dictionaries is faster than using lists.\n",
    "* Using .iterrows() does not improve the speed of iterating through the DataFrame but it is more efficient.\n",
    "* The .apply() function performs faster when we want to iterate through all the rows of a pandas DataFrame, but is slower when we perform the same operation through a column.\n",
    "* Vectorizing over the pandas series achieves the overwhelming majority of optimization needs for everyday calculations. However, if speed is of the highest priority, we can call in reinforcements in the form of the NumPy Python library.\n",
    "* Using .groupby() to group it according to a certain feature and then using other functions to apply it to the data is much faster than using the conventional coding method."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}