code/03-data-analysis-with-pandas/12-dataframe-modification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back](./11-dataframe-properties.ipynb)\n",
    "\n",
    "---\n",
    "## `Modifying DataFrames`\n",
    "\n",
    "- [Modifying individual elements from DataFrame](#modifying-dataframe-elements)\n",
    "  - [Adding elements to DataFrame](#adding-elements-to-dataframe)\n",
    "  - [Combining DataFrames](#combining-dataframe)\n",
    "  - [Remove operations on a DataFrame](#remove-operations-on-dataframe)\n",
    "- [Modifying entire rows and columns in DataFrame](#modifying-dataframe-rows-and-columns)\n",
    "- [Shuffling and relabelling DataFrames](#shuffling-and-relabelling-dataframes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Initial Setup`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas\n",
    "\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 2:\n",
      "      col1  col2  col3\n",
      "row1  10.0   2.0   NaN\n",
      "row4   6.0   NaN   0.0\n",
      "row3   9.0   1.0   7.0\n",
      "row2   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Data set-up\n",
    "\n",
    "df1 = pd.DataFrame({\n",
    "  'col1': {'row1':1, 'row2':1, 'row3':3},\n",
    "  'col2': {'row1':4, 'row3':9, 'row4':6},\n",
    "  'col3': {'row1':10, 'row2':8, 'row4':6}\n",
    "  })\n",
    "\n",
    "df2 = pd.DataFrame({\n",
    "  'col1': {'row1':10, 'row4':6, 'row3':9},\n",
    "  'col2': {'row1':2, 'row3':1, 'row2':6},\n",
    "  'col3': {'row3':7, 'row2':6, 'row4':0}\n",
    "  })\n",
    "\n",
    "def reset_df1():\n",
    "  global df1\n",
    "  df1 = pd.DataFrame({\n",
    "      'col1': {'row1': 1, 'row2': 1, 'row3': 3},\n",
    "      'col2': {'row1': 4, 'row3': 9, 'row4': 6},\n",
    "      'col3': {'row1': 10, 'row2': 8, 'row4': 6}\n",
    "  })\n",
    "  print_df1()\n",
    "\n",
    "def reset_df2():\n",
    "  global df2\n",
    "  df2 = pd.DataFrame({\n",
    "      'col1': {'row1': 10, 'row4': 6, 'row3': 9},\n",
    "      'col2': {'row1': 2, 'row3': 1, 'row2': 6},\n",
    "      'col3': {'row3': 7, 'row2': 6, 'row4': 0}\n",
    "  })\n",
    "  print_df2()\n",
    "\n",
    "def print_df1():\n",
    "  print('Original DataFrame 1:')\n",
    "  print(df1)\n",
    "  divider()\n",
    "\n",
    "def print_df2():\n",
    "  print('Original DataFrame 2:')\n",
    "  print(df2)\n",
    "  divider()\n",
    "\n",
    "def divider():\n",
    "  print('-'*80)\n",
    "\n",
    "print_df1()\n",
    "print_df2()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Modifying DataFrame elements`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Adding elements to DataFrame`\n",
    "\n",
    "- Assign a row or column that doesn't exist already\n",
    "- Use the append function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Updating / Modifying DataFrame 1:\n",
      "Original DataFrame 1:\n",
      "      col1  col2  col3  col4\n",
      "row1   1.0   4.0  10.0     1\n",
      "row2   1.0   NaN   8.0     2\n",
      "row3   3.0   9.0   NaN     3\n",
      "row4   NaN   6.0   6.0     4\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Assigning new row / column\n",
    "works similar to a Dictionary\n",
    "\"\"\"\n",
    "\n",
    "print_df1()\n",
    "\n",
    "\"\"\"\n",
    "It is better to keep it the same length (rows / columns) as of existing DataFrame\n",
    "Else it may either cut them off / fill with NaN\n",
    "\"\"\"\n",
    "# This would not work either, because this (row labels) is not the structure on the existing DataFrame\n",
    "# df1['col4'] = pd.Series({'a':1, 'b':2})\n",
    "\n",
    "# Adding a column to DataFrame 1\n",
    "print('Updating / Modifying DataFrame 1:')\n",
    "df1['col4'] = [1, 2, 3, 4]\n",
    "\n",
    "print_df1()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3  col4\n",
      "row1   1.0   4.0  10.0     1\n",
      "row2   1.0   NaN   8.0     2\n",
      "row3   3.0   9.0   NaN     3\n",
      "row4   NaN   6.0   6.0     4\n",
      "--------------------------------------------------------------------------------\n",
      "Updating / Modifying DataFrame 1:\n",
      "Original DataFrame 1:\n",
      "      col1  col2  col3  col4\n",
      "row1   1.0   4.0  10.0     1\n",
      "row2   1.0   NaN   8.0     2\n",
      "row3   3.0   9.0   NaN     3\n",
      "row4   NaN   6.0   6.0     4\n",
      "row5  20.0  30.0  40.0    50\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Assigning new row / column\n",
    "works similar to a Dictionary\n",
    "\"\"\"\n",
    "\n",
    "print_df1()\n",
    "\n",
    "\"\"\"\n",
    "It is better to keep it the same length (rows / columns) as of existing DataFrame\n",
    "Else it may either cut them off / fill with NaN\n",
    "\"\"\"\n",
    "# This would not work either, because this (column labels) is not the structure on the existing DataFrame\n",
    "# df1.loc['row5'] = pd.Series({'a':1, 'b':2})\n",
    "\n",
    "# Adding a column to DataFrame 1\n",
    "print('Updating / Modifying DataFrame 1:')\n",
    "df1.loc['row5'] = [20, 30, 40, 50]\n",
    "\n",
    "print_df1()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3  col4\n",
      "row1   1.0   4.0  10.0     1\n",
      "row2   1.0   NaN   8.0     2\n",
      "row3   3.0   9.0   NaN     3\n",
      "row4   NaN   6.0   6.0     4\n",
      "row5  20.0  30.0  40.0    50\n",
      "--------------------------------------------------------------------------------\n",
      "New DataFrame 1:\n",
      "   col1  col2  col3  col4\n",
      "0   1.0   4.0  10.0     1\n",
      "1   1.0   NaN   8.0     2\n",
      "2   3.0   9.0   NaN     3\n",
      "3   NaN   6.0   6.0     4\n",
      "4  20.0  30.0  40.0    50\n",
      "5  11.0  12.0  13.0    14\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Assigning new row / column, using append function\n",
    "\n",
    "print_df1()\n",
    "\n",
    "\"\"\"\n",
    "  Can append a DataFrame or Series\n",
    "  Adding a new row, so need to add as many elements as the no. of columns\n",
    "  Need to set ignore_index=True\n",
    "\n",
    "  ignore_index : bool, default False\n",
    "    If True, the resulting axis will be labeled 0, 1, …, n - 1.\n",
    "\"\"\"\n",
    "new_df1 = df1.append(pd.Series([11, 12, 13, 14], index=['col1', 'col2', 'col3', 'col4']), ignore_index=True)\n",
    "print('New DataFrame 1:')\n",
    "print(new_df1)\n",
    "divider()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Combining DataFrame`\n",
    "\n",
    "Combining takes one DataFrame and stick on the end if we use **concat** operation<br>\n",
    "Or if we just use the **combine_first** function, this actually combines it and overrides any **NaN** values that it needs to\n",
    "\n",
    "Both of these operations return a new DataFrame and leave the original one intact\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 2:\n",
      "      col1  col2  col3\n",
      "row1  10.0   2.0   NaN\n",
      "row4   6.0   NaN   0.0\n",
      "row3   9.0   1.0   7.0\n",
      "row2   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Result of combining DataFrame 2 with DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   6.0   8.0\n",
      "row3   3.0   9.0   7.0\n",
      "row4   6.0   6.0   6.0\n"
     ]
    }
   ],
   "source": [
    "reset_df1()\n",
    "print_df2()\n",
    "\n",
    "# Combine DataFrame 2 with DataFrame1\n",
    "combined = df1.combine_first(df2)\n",
    "print('Result of combining DataFrame 2 with DataFrame 1:')\n",
    "print(combined)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, so the above result is because it overrides everything from DataFrame 2 in DataFrame 1 if a value is **NaN**\n",
    "\n",
    "**`NOTE:`** The ordering of rows in DataFrame 1 and DataFrame2 are different!\n",
    "\n",
    "On the other hand, the below example, using concat, it will take one DataFrame and stick it to the end of another DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 2:\n",
      "      col1  col2  col3\n",
      "row1  10.0   2.0   NaN\n",
      "row4   6.0   NaN   0.0\n",
      "row3   9.0   1.0   7.0\n",
      "row2   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Concat DataFrame 1 and DataFrame 2:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "row1  10.0   2.0   NaN\n",
      "row4   6.0   NaN   0.0\n",
      "row3   9.0   1.0   7.0\n",
      "row2   NaN   6.0   6.0\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "print_df2()\n",
    "\n",
    "concat_df = pd.concat([df1, df2])\n",
    "print('Concat DataFrame 1 and DataFrame 2:')\n",
    "print(concat_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Things to NOTE**\n",
    "\n",
    "We actually have duplicate row labels, this is a desired behavior.\n",
    "\n",
    "But when we want to fetch a row, it might fetch more than one row that has the same label, for example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      col1  col2  col3\n",
      "row2   1.0   NaN   8.0\n",
      "row2   NaN   6.0   6.0\n"
     ]
    }
   ],
   "source": [
    "print(concat_df.loc['row2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also concatenate **DataFrame slices**\n",
    "\n",
    "for example, first two rows from DataFrame 1 and last two rows from DataFrame 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 2:\n",
      "      col1  col2  col3\n",
      "row1  10.0   2.0   NaN\n",
      "row4   6.0   NaN   0.0\n",
      "row3   9.0   1.0   7.0\n",
      "row2   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   9.0   1.0   7.0\n",
      "row2   NaN   6.0   6.0\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "print_df2()\n",
    "\n",
    "combined_df = pd.concat([df1[:2], df2[-2:]])\n",
    "print(combined_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Remove operations on DataFrame`\n",
    "\n",
    "This is very similar to remove operations on **Pandas Series**, just that we have more axes to work with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "After dropna(), rows:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "After dropna(), columns:\n",
      "Empty DataFrame\n",
      "Columns: []\n",
      "Index: [row1, row2, row3, row4]\n"
     ]
    }
   ],
   "source": [
    "# dropna function\n",
    "\n",
    "\"\"\"\n",
    "This helps us to drop either rows with NAs or columns with NAs\n",
    "This also return a new DataFrame\n",
    "\"\"\"\n",
    "print_df1()\n",
    "\n",
    "dropped = df1.dropna()\n",
    "print('After dropna(), rows:')\n",
    "print(dropped)\n",
    "\n",
    "# with axis set to drop columns\n",
    "dropped = df1.dropna(axis=1)\n",
    "print('After dropna(), columns:')\n",
    "print(dropped)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to drop specific columns or indexes (rows) from DataFrames, we need to use .drop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "After dropping row4 and col3 from DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "\n",
    "# We want to drop 'row4' and 'col3'\n",
    "new_df1 = df1.drop(index=['row4'], columns=['col3'])\n",
    "print('After dropping row4 and col3 from DataFrame 1:')\n",
    "print(new_df1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "After dropping col2 from DataFrame 1:\n",
      "Original DataFrame 1:\n",
      "      col1  col3\n",
      "row1   1.0  10.0\n",
      "row2   1.0   8.0\n",
      "row3   3.0   NaN\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "\n",
    "# Alternatively, we can use the del function\n",
    "# This will modify the original DataFrame and will work only on columns\n",
    "del df1['col2']\n",
    "print('After dropping col2 from DataFrame 1:')\n",
    "print_df1()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "After popping col2 from DataFrame 1:\n",
      "Original DataFrame 1:\n",
      "      col1  col3\n",
      "row1   1.0  10.0\n",
      "row2   1.0   8.0\n",
      "row3   3.0   NaN\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Popped column:\n",
      "row1    4.0\n",
      "row2    NaN\n",
      "row3    9.0\n",
      "row4    6.0\n",
      "Name: col2, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "reset_df1()\n",
    "\n",
    "# Also, we can use the pop function\n",
    "# This will modify the original DataFrame, will work only on columns and also return the popped column\n",
    "popped = df1.pop('col2')\n",
    "print('After popping col2 from DataFrame 1:')\n",
    "print_df1()\n",
    "\n",
    "print('Popped column:')\n",
    "print(popped)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Modifying DataFrame Rows and Columns`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "reset_df1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Reassigning elements`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1     2   4.0  10.0\n",
      "row2     3   NaN   8.0\n",
      "row3     4   9.0   NaN\n",
      "row4     5   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   2.0   4.0  10.0\n",
      "row2   3.0   NaN   8.0\n",
      "row3   4.0   9.0   NaN\n",
      "row4   5.0   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Modify the first column\n",
    "\n",
    "df1['col1'] = [2, 3, 4, 5]\n",
    "print_df1()\n",
    "\n",
    "# Data type would change to float if at least one value is float\n",
    "df1['col1'] = [2, 3, 4, 5.0]\n",
    "print_df1()\n",
    "\n",
    "# And we cannot have more value than what the DataFrame is already having\n",
    "# Below would throw an error, because there are only 4 rows and a 5th value will not be accepted (also, fewer values will also throw error)\n",
    "# df1['col1'] = [2, 3, 4, 5, 8]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   NaN   4.0  10.0\n",
      "row2   NaN   NaN   8.0\n",
      "row3   NaN   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Providing incorrect labels, would insert the values, but has a strange behavior\n",
    "df1['col1'] = pd.Series({'1': 1, '2': 2, '3': 3, '4': 4})\n",
    "print_df1()\n",
    "\n",
    "reset_df1()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1  10.0  11.0  12.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "\n",
    "# Modifying first row\n",
    "# It can be list of values / Pandas Series\n",
    "df1.loc['row1'] = [10, 11, 12]\n",
    "print_df1()\n",
    "\n",
    "# Types gets dynamically updated based on the existing data-type of the column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   NaN   NaN   NaN\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "reset_df1()\n",
    "\n",
    "# Modifying rows with Pandas Series\n",
    "df1.loc['row1'] = pd.Series({'1': 1, '2': 2, '3': 3})\n",
    "# This in fact does the same thing as we did for columns\n",
    "print_df1()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   0.0   8.0\n",
      "row3   3.0   9.0   0.0\n",
      "row4   0.0   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "reset_df1()\n",
    "\n",
    "# Setting all the NaN values in a DataFrame\n",
    "# This take in a few different labels\n",
    "# If we just pass an value and not the access, it'll update across the DataFrame\n",
    "# If we specify an axis, it'll only do it specifically on that axis\n",
    "# And this will return a new DataFrame and keep the original intact\n",
    "df1 = df1.fillna(0)\n",
    "print_df1()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Shuffling and Relabelling DataFrames`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `Restructuring DataFrames`\n",
    "\n",
    "We can restructure based on columns or rows or both\n",
    "\n",
    "We can also use the **reindex function** to assign new rows / new columns to the DataFrame\n",
    "\n",
    "> **reindex()**\n",
    "\n",
    "- If we are just going to perform the labels, it is going to be a row specific **reindex**\n",
    "- If we want to **reindex** our columns, then we need to add the `axis=1`\n",
    "- This is one of the few functions that is row specific\n",
    "\n",
    "Generally speaking about DataFrames, when we use the word **index** we are actually speaking about the **row labels**\n",
    "\n",
    "And then we are speaking about **columns**, then we are speaking about **column labels**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Restructured DataFrame:\n",
      "      col3  col1  col2\n",
      "row1  10.0   1.0   4.0\n",
      "row2   8.0   1.0   NaN\n",
      "row3   NaN   3.0   9.0\n",
      "row4   6.0   NaN   6.0\n"
     ]
    }
   ],
   "source": [
    "reset_df1()\n",
    "\n",
    "# Adding new column\n",
    "# This return a new DataFrame\n",
    "restructured_df = df1.reindex(columns=['col3', 'col1', 'col2'])\n",
    "print('Restructured DataFrame:')\n",
    "print(restructured_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add new **columns** this way\n",
    "\n",
    "We can specify a new column, but it will fill with **NaN**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Restructured DataFrame:\n",
      "      col3  col1  col2  col4\n",
      "row1  10.0   1.0   4.0   NaN\n",
      "row2   8.0   1.0   NaN   NaN\n",
      "row3   NaN   3.0   9.0   NaN\n",
      "row4   6.0   NaN   6.0   NaN\n"
     ]
    }
   ],
   "source": [
    "reset_df1()\n",
    "\n",
    "# Adding new column\n",
    "# This return a new DataFrame\n",
    "restructured_df = df1.reindex(columns=['col3', 'col1', 'col2', 'col4'])\n",
    "print('Restructured DataFrame:')\n",
    "print(restructured_df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also change the column labels, by simply reassigning them to the columns\n",
    "\n",
    "**NOTE:** this will modify the original DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 1:\n",
      "        0    1     2\n",
      "row1  1.0  4.0  10.0\n",
      "row2  1.0  NaN   8.0\n",
      "row3  3.0  9.0   NaN\n",
      "row4  NaN  6.0   6.0\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "reset_df1()\n",
    "\n",
    "# Reassign the column labels\n",
    "# Will be getting error if there are more or less values in the list\n",
    "df1.columns = ['0', '1', '2']\n",
    "print_df1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of these restructuring applied on the columns can also be applied to rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Restructured DataFrame rows:\n",
      "      col1  col2  col3\n",
      "row4   NaN   6.0   6.0\n",
      "row3   3.0   9.0   NaN\n",
      "row2   1.0   NaN   8.0\n",
      "row1   1.0   4.0  10.0\n"
     ]
    }
   ],
   "source": [
    "reset_df1()\n",
    "\n",
    "# Restructuring rows\n",
    "restructured_df = df1.reindex(index=['row4', 'row3', 'row2', 'row1'])\n",
    "print('Restructured DataFrame rows:')\n",
    "print(restructured_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also add rows, similar to the columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Restructured DataFrame by adding a new row:\n",
      "      col1  col2  col3\n",
      "row4   NaN   6.0   6.0\n",
      "row3   3.0   9.0   NaN\n",
      "row2   1.0   NaN   8.0\n",
      "row1   1.0   4.0  10.0\n",
      "row0   NaN   NaN   NaN\n"
     ]
    }
   ],
   "source": [
    "reset_df1()\n",
    "\n",
    "# Adding new row\n",
    "restructured_df = df1.reindex(index=['row4', 'row3', 'row2', 'row1', 'row0'])\n",
    "print('Restructured DataFrame by adding a new row:')\n",
    "print(restructured_df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also reassign indexes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2  col3\n",
      "row1   1.0   4.0  10.0\n",
      "row2   1.0   NaN   8.0\n",
      "row3   3.0   9.0   NaN\n",
      "row4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 1:\n",
      "    col1  col2  col3\n",
      "r1   1.0   4.0  10.0\n",
      "r2   1.0   NaN   8.0\n",
      "r3   3.0   9.0   NaN\n",
      "r4   NaN   6.0   6.0\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "reset_df1()\n",
    "\n",
    "# This will modify the original DataFrame\n",
    "# Must provide value to each of the rows in the DataFrame, else we'll get an error\n",
    "df1.index = ['r1', 'r2', 'r3', 'r4']\n",
    "print_df1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "[next](./13-dataframe-operations.ipynb)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "0b4484ae063bf9ef0d5115063857dd0e382a2c73ed49a11a000ca56af03302b6"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}