getting_started_notebooks/basics/Getting_Started_with_cuDF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started with cuDF\n",
    "#### By Yi Dong, Paul Hendricks\n",
    "-------\n",
    "\n",
    "While the world’s data doubles each year, CPU computing has hit a brick wall with the end of Moore’s law. For the same reasons, scientific computing and deep learning has turned to NVIDIA GPU acceleration, data analytics and machine learning where GPU acceleration is ideal. \n",
    "\n",
    "NVIDIA created RAPIDS – an open-source data analytics and machine learning acceleration platform that leverages GPUs to accelerate computations. RAPIDS is based on Python, has pandas-like and Scikit-Learn-like interfaces, is built on Apache Arrow in-memory data format, and can scale from 1 to multi-GPU to multi-nodes. RAPIDS integrates easily into the world’s most popular data science Python-based workflows. RAPIDS accelerates data science end-to-end – from data prep, to machine learning, to deep learning. And through Arrow, Spark users can easily move data into the RAPIDS platform for acceleration.\n",
    "\n",
    "In this notebook, we will also show how to get started with GPU DataFrames using cuDF in RAPIDS.\n",
    "\n",
    "**Table of Contents**\n",
    "\n",
    "* Setup\n",
    "* Loading data into a GPU DataFrame (GDF)\n",
    "  * Loading data into a Pandas DataFrame\n",
    "  * Converting a Pandas DataFrame to a GDF\n",
    "* Working with the GDF\n",
    "  * Take a look at the columns and their data types\n",
    "  * Slice the GDF\n",
    "  * Modify data types\n",
    "  * Manipulate data with a user-defined function (UDF)\n",
    "  * Sort the data\n",
    "  * Filter the data\n",
    "  * One-hot encode categorical columns\n",
    "  * Split the data into training and validation sets\n",
    "  * Turn the GDFs into matrices\n",
    "* Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "This notebook was tested using the `nvcr.io/nvidia/rapidsai/rapidsai:0.5-cuda10.0-runtime-ubuntu18.04-gcc7-py3.7` Docker container from [NVIDIA GPU Cloud](https://ngc.nvidia.com) and run on the NVIDIA Tesla V100 GPU. Please be aware that your system may be different and you may need to modify the code or install packages to run the below examples. \n",
    "\n",
    "If you think you have found a bug or an error, please file an issue here: https://github.com/rapidsai/notebooks/issues\n",
    "\n",
    "Before we begin, let's check out our hardware setup by running the `nvidia-smi` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's see what CUDA version we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvcc --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last, let's ensure that we have graphviz installed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!apt update\n",
    "!apt install -y graphviz\n",
    "!conda install graphviz\n",
    "!conda install python-graphviz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading data into a GPU DataFrame (GDF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading data into a Pandas DataFrame\n",
    "\n",
    "It's easy to load almost any sort of data (json, csv, etc) into a Pandas DataFrame. <br>\n",
    "For example, let's import some census data from a compressed CSV file on disk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd; print('pandas Version:', pd.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read data from csv file into pandas dataframe\n",
    "df = pd.read_csv('../../data/ipums/ipums_easy.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Read more on using a Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting a Pandas DataFrame to a GDF\n",
    "\n",
    "Next, we use our `pandas.DataFrame` and to create a `cudf.dataframe.DataFrame` object using the `from_pandas` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cudf\n",
    "\n",
    "# convert the Panda dataframe into a GPU dataframe\n",
    "gdf = cudf.DataFrame.from_pandas(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And that's it! For the most part, working with GPU DataFrames will be the same as working with Pandas DataFrames. See the [cuDF documentation](https://cudf.readthedocs.io/en/latest/index.html) for more information.\n",
    "\n",
    "## Working with the GDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Take a look at the columns and their data types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the columns and their datatypes in this gdf\n",
    "gdf.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Slice the GDF\n",
    "\n",
    "Woah! This GDF has a lot of columns, let's make it more manageable..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# only select certain columns (and overwrite the gdf)\n",
    "column_names = [\n",
    "    'INCEARN', 'PERWT', 'ADJUST', 'STATEICP', 'ROOMS', 'BEDROOMS',\n",
    "     'PHONE', 'VEHICLES', 'RACE', 'SEX', 'AGE', 'VETSTAT'\n",
    "]\n",
    "gdf = gdf.loc[:, column_names]\n",
    "\n",
    "# show the first 5 records of each column\n",
    "print(gdf.head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modify data types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gdf.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like `INCEARN` and `PERWT` are integers when they should be floats. Let's fix that..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# convert the following two int64 columns to float64 data type\n",
    "gdf['INCEARN'] = gdf['INCEARN'].astype(np.float64)\n",
    "gdf['PERWT'] = gdf['PERWT'].astype(np.float64)\n",
    "\n",
    "# take another look\n",
    "gdf.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manipulate data with a user-defined function (UDF)\n",
    "\n",
    "`INCEARN` is a column in our dataset that supposedly represents income earned; however, it does not truly represent the amount of income earned when adjusted for inflation. The `ADJUST` column represents the dollar inflation factor, which we can use to adjust `INCEARN` to the amount that the individual would have earned during the calender year. In our dataset, `ADJUST` is constant over all rows.\n",
    "\n",
    "Below, we will define a simple function `adjust_incearn` that takes `INCEARN` and and multiplies it by a constant - in this case, the dollar inflation factor. We'll use the `applymap` method in our `cudf.dataframe.DataFrame` object to apply an element-wise function to transform the values in the Column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a function to adjust the incearn column\n",
    "# so it more accurately represents income earned\n",
    "adjust = gdf['ADJUST'][0]   # take constant from first row\n",
    "print('adjustment factor: {}'.format(adjust))\n",
    "def adjust_incearn(incearn):\n",
    "    return adjust * incearn;\n",
    "\n",
    "# apply it to the 'population' column\n",
    "gdf['INCEARN'] = gdf['INCEARN'].applymap(adjust_incearn)\n",
    "\n",
    "# drop the ADJUST column\n",
    "gdf.drop_column('ADJUST')\n",
    "\n",
    "# compute the mean\n",
    "print('mean adjusted income: {}'.format(gdf['INCEARN'].mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sort the data\n",
    "\n",
    "Next, let's sort out data to do some light exploration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sort the gdf by the INCEARN column\n",
    "gdf = gdf.sort_values(by='INCEARN', ascending=True)\n",
    "# reset the index so we can use loc slicing later\n",
    "gdf = gdf.reset_index()\n",
    "print(gdf.head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like we have some negative income values. Let's filter those out...\n",
    "\n",
    "### Filter the data\n",
    "\n",
    "We'll use the `query` method to filter our dataset. The `query` method takes as argument a boolean expression very similar to the `query` method for the `pandas.DataFrame` class. However, the `cudf.dataframe.DataFrame` implementation uses Numba to compile a GPU kernel. \n",
    "\n",
    "For more information on the syntax for arguments into `query`, see the Pandas documentation: \n",
    "\n",
    "https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.query.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# how many records do we have?\n",
    "print(\"{} = Original # of records\".format(len(gdf)))\n",
    "\n",
    "# filter out\n",
    "gdf = gdf.query('INCEARN >= 0')\n",
    "\n",
    "# how many records do we have left?\n",
    "print(\"{} = New # of records\".format(len(gdf)))\n",
    "\n",
    "# sanity check...\n",
    "print(gdf.head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-hot encode categorical columns\n",
    "\n",
    "Next, let's prepare our categorical columns. Machine learning models won't take strings as inputs, so we need to go to each column and convert its string representations to a numerical representation. The most common way to convert a Column with `n` elements and `k` unique categories to a numerical representation is to create a matrix of shape `n` by `k` and impute a 1 in cell `(i, j)` if the `ith` element is of category `j` and 0 otherwise, where $j \\in k$. This is known as one-hot encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the categorical columns\n",
    "cat_cols = set(['STATEICP', 'RACE', 'SEX', 'VETSTAT'])\n",
    "# store the unique values for each category column\n",
    "uniques = {}\n",
    "\n",
    "# iterate through each categorical column and one-hot\n",
    "# encode it using the unique values it has\n",
    "for k in cat_cols:\n",
    "    uniques[k] = gdf[k].unique_k(k=1000)\n",
    "    cats = uniques[k][1:]  # drop first\n",
    "    gdf = gdf.one_hot_encoding(k, prefix=k, cats=cats)\n",
    "    del gdf[k]\n",
    "    \n",
    "# we should see many more columns since the categorical\n",
    "# columns will get expanded due to one-hot encoding\n",
    "gdf.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the data into training and validation sets\n",
    "\n",
    "Next, let's split out data into an 80% train dataset and a 20% validation dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# enforce float64 data type on ALL columns\n",
    "for k in gdf.columns:\n",
    "    gdf[k] = gdf[k].astype(np.float64)\n",
    "\n",
    "# set the fractions for training and validation\n",
    "fractions = {\n",
    "    \"train\": 0.8,\n",
    "    \"valid\": 0.2\n",
    "}\n",
    "\n",
    "# validation splitpoint\n",
    "splitpoint = int(len(gdf) * fractions[\"train\"])\n",
    "print('splitpoint: {} of {} is {}'.format(fractions[\"train\"], len(gdf), splitpoint))\n",
    "\n",
    "# break the gdf up into training and validation sets\n",
    "gdfs = {\n",
    "    \"train\": gdf.loc[:splitpoint],\n",
    "    \"valid\": gdf.loc[splitpoint:]\n",
    "}\n",
    "print('gdfs[\"train\"] has {} rows'.format(len(gdfs[\"train\"])))\n",
    "print('gdfs[\"valid\"] has {} rows'.format(len(gdfs[\"valid\"])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Turn the GDFs into matrices\n",
    "\n",
    "Lastly, we want to convert our GPU DataFrame to a GPU Matrix for usage as input to other machine learning libraries such as cuML and XGBoost. We can use the `as_gpu_matrix` method to facillitate this conversion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# produce gpu matrices (to input to ML libraries, etc.)\n",
    "# this step should not be necessary in the near future\n",
    "# (should be able to use gdf as input)\n",
    "matrices = {\n",
    "    \"train\": {\n",
    "        \"x\": gdfs[\"train\"].as_gpu_matrix(columns=gdf.columns[1:]),\n",
    "        \"y\": gdfs[\"train\"].as_gpu_matrix(columns=[gdf.columns[0]])\n",
    "    },\n",
    "    \"valid\": {\n",
    "        \"x\": gdfs[\"valid\"].as_gpu_matrix(columns=gdf.columns[1:]),\n",
    "        \"y\": gdfs[\"valid\"].as_gpu_matrix(columns=[gdf.columns[0]])\n",
    "    }\n",
    "}\n",
    "\n",
    "# check the matrix shapes (sanity check)\n",
    "print('matrices[\"train\"][\"x\"] shape:', matrices[\"train\"][\"x\"].shape)\n",
    "print('matrices[\"train\"][\"y\"] shape:', matrices[\"train\"][\"y\"].shape)\n",
    "print('matrices[\"valid\"][\"x\"] shape:', matrices[\"valid\"][\"x\"].shape)\n",
    "print('matrices[\"valid\"][\"y\"] shape:', matrices[\"valid\"][\"y\"].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "To learn more about RAPIDS, be sure to check out: \n",
    "\n",
    "* [Open Source Website](http://rapids.ai)\n",
    "* [GitHub](https://github.com/rapidsai/)\n",
    "* [Press Release](https://nvidianews.nvidia.com/news/nvidia-introduces-rapids-open-source-gpu-acceleration-platform-for-large-scale-data-analytics-and-machine-learning)\n",
    "* [NVIDIA Blog](https://blogs.nvidia.com/blog/2018/10/10/rapids-data-science-open-source-community/)\n",
    "* [Developer Blog](https://devblogs.nvidia.com/gpu-accelerated-analytics-rapids/)\n",
    "* [NVIDIA Data Science Webpage](https://www.nvidia.com/en-us/deep-learning-ai/solutions/data-science/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}