01 - Data Sources/01 - BigQuery - Table Data Source.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c28f7fa2",
   "metadata": {},
   "source": [
    "![ga4](https://www.google-analytics.com/collect?v=2&tid=G-6VDTYWLKX6&cid=1&en=page_view&sid=1&dl=statmike%2Fvertex-ai-mlops%2F01+-+Data+Sources&dt=01+-+BigQuery+-+Table+Data+Source.ipynb)\n",
    "\n",
    "# 01 - BigQuery - Table Data Source\n",
    "Use BigQuery to load and prepare data for machine learning:\n",
    "\n",
    "**Video Walkthrough of this notebook:**\n",
    "\n",
    "Includes conversational walkthrough and more explanatory information than the notebook:\n",
    "<p align=\"center\" width=\"100%\"><center><a href=\"https://youtu.be/Z5whg20WvS8\" target=\"_blank\" rel=\"noopener noreferrer\"><img src=\"../architectures/thumbnails/playbutton/01.png\" width=\"40%\"></a></center></p>\n",
    "\n",
    "**Prerequisites:**\n",
    "-  [00 - Environment Setup](../00%20-%20Setup/00%20-%20Environment%20Setup.ipynb)\n",
    "\n",
    "**Resources:**\n",
    "-  [Python Client For Google BigQuery](https://googleapis.dev/python/bigquery/latest/index.html)\n",
    "-  [Download BigQuery Data to Pandas](https://cloud.google.com/bigquery/docs/bigquery-storage-python-pandas)\n",
    "-  [BigQuery Template Notebooks](https://github.com/GoogleCloudPlatform/bigquery-notebooks/tree/main/notebooks/official/template_notebooks)\n",
    "- Using BigQuery From Python, Notebooks in This Repository:\n",
    "    - [Tips/BigQuery - Python Client](../Tips/BigQuery%20-%20Python%20Client.ipynb)\n",
    "    - [03 - BigQuery ML (BQML)/Introduction to BigQuery ML (BQML)](../03%20-%20BigQuery%20ML%20(BQML)/Introduction%20to%20BigQuery%20ML%20(BQML).ipynb)\n",
    "    - [Applied Forecasting/1 - BigQuery Time Series Forecasting Data Review and Preparation](../Applied%20Forecasting/1%20-%20BigQuery%20Time%20Series%20Forecasting%20Data%20Review%20and%20Preparation.ipynb)\n",
    "\n",
    "**Conceptual Flow & Workflow**\n",
    "\n",
    "<p align=\"center\">\n",
    "  <img alt=\"Conceptual Flow\" src=\"../architectures/slides/01_arch.png\" width=\"45%\">\n",
    "&nbsp; &nbsp; &nbsp; &nbsp;\n",
    "  <img alt=\"Workflow\" src=\"../architectures/slides/01_console.png\" width=\"45%\">\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d0c695d-002d-48ca-9d52-839f95fdfee0",
   "metadata": {},
   "source": [
    "---\n",
    "## Source Data\n",
    "\n",
    "**Overview**\n",
    "\n",
    "This notebook imports source data for this project into Google BigQuery.  All the remaining notebooks utilize BigQuery as the source and leverage API's native to the machine learning approaches they feature.\n",
    "\n",
    "In the enviornment setup notebook (00), a BigQuery source table was exported to CSV format in a Cloud Storage Bucket. This notebook, `01 - BigQuery - Table Data Source`, starts the machine learning lifecycle by importing source data and preparing it for machine learning.\n",
    "\n",
    "All of these workflows utilize tabular data to fit a supervised learning model: predict a target variable by learning patterns in feature columns.  The type of supervised learning used in these projects is classification: models with a target variable that has multiple discrete classes.  \n",
    "\n",
    "**The Data**\n",
    "\n",
    "The source data is first exported to Google Cloud Storage in CSV format below.  The BigQuery source table is `bigquery-public-data.ml_datasets.ulb_fraud_detection`.  This is a table of credit card transactions that are classified as fradulant, `Class = 1`, or normal `Class = 0`.    \n",
    "- The data can be researched further at this [Kaggle link](https://www.kaggle.com/mlg-ulb/creditcardfraud).\n",
    "- Read mode about BigQuery public datasets [here](https://cloud.google.com/bigquery/public-data)\n",
    "\n",
    "**Description of the Data**\n",
    "\n",
    "This is a table of 284,807 credit card transactions classified as fradulant or normal in the column `Class`.  In order protect confidentiality, the original features have been transformed using [principle component analysis (PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis) into 28 features named `V1, V2, ... V28` (float).  Two descriptive features are provided without transformation by PCA:\n",
    "- `Time` (integer) is the seconds elapsed between the transaction and the earliest transaction in the table\n",
    "- `Amount` (float) is the value of the transaction\n",
    ">**Quick Note on PCA**<p>PCA is an unsupervised learning technique: there is not a target variable.  PCA is commonlly used as a variable/feature reduction technique.  If you have 100 features then you could reduce it to a number p (say 10) projected features.  The choice of this number is a balance of how well it can explain the variance of the full feature space and reducing the number of features.  Each projected feature is orthogonal to each other feature, meaning there is no correlation between these new projected features.</p>\n",
    "\n",
    "**Preparation of the Data**\n",
    "\n",
    "This notebook adds two columns to the source data and stores it in a new table with suffix `_prepped`.  \n",
    "- `transaction_id` (string) a unique id for the row/transaction\n",
    "- `splits` (string) this divided the tranactions into sets for `TRAIN` (80%), `VALIDATE` (10%), and `TEST` (10%)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5046940",
   "metadata": {},
   "source": [
    "---\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0ae60b2",
   "metadata": {},
   "source": [
    "inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d6f0503a-e864-4170-ade9-0ebabd14efcd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'statmike-mlops-349915'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project = !gcloud config get-value project\n",
    "PROJECT_ID = project[0]\n",
    "PROJECT_ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "56a5bc43",
   "metadata": {},
   "outputs": [],
   "source": [
    "REGION = 'us-central1'\n",
    "EXPERIMENT = '01'\n",
    "SERIES = '01'\n",
    "\n",
    "# source data\n",
    "BQ_PROJECT = PROJECT_ID\n",
    "BQ_DATASET = 'fraud'\n",
    "BQ_TABLE = 'fraud'\n",
    "\n",
    "# Data source for this series of notebooks: Described above\n",
    "BQ_SOURCE = 'bigquery-public-data.ml_datasets.ulb_fraud_detection'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4a963be",
   "metadata": {},
   "source": [
    "packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8a37b83a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.cloud import bigquery\n",
    "from google.cloud import storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e24cebb6",
   "metadata": {},
   "source": [
    "clients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9e515410",
   "metadata": {},
   "outputs": [],
   "source": [
    "bq = bigquery.Client(project = PROJECT_ID)\n",
    "gcs = storage.Client(project = PROJECT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6601398",
   "metadata": {},
   "source": [
    "parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1f1e2af3",
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET = PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "504c3a89-d98f-471c-ae54-3b9157651f0b",
   "metadata": {},
   "source": [
    "---\n",
    "## Store the Source Data in GCS Storage Bucket\n",
    "Check to see if the export exist and create if not:\n",
    "- export from bigquery table to GCS bucket as CSV\n",
    "    - the table is referenced in the `BQ_SOURCE` variable at the top of this notebook\n",
    "- [Exporting Table Data](https://cloud.google.com/bigquery/docs/exporting-data#python)\n",
    "- [BigQuery Python Client](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client#google_cloud_bigquery_client_Client_extract_table)\n",
    "\n",
    "For more tips on interacting with GCS using the Python Client review the tips notebook [Python Client for GCS](../Tips/Python%20Client%20for%20GCS.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6144857f-1ecc-4712-925b-4de69b6df0a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "file = f\"{SERIES}/{EXPERIMENT}/data/{BQ_TABLE}.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "65714181-b946-4c5c-b990-6f7ac4f2ca2a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The file has already been created at: gs://statmike-mlops-349915/01/01/data/fraud.csv\n"
     ]
    }
   ],
   "source": [
    "bucketDef = gcs.bucket(BUCKET)\n",
    "if storage.Blob(bucket = bucketDef, name = file).exists(gcs):\n",
    "    print(f'The file has already been created at: gs://{bucketDef.name}/{file}')\n",
    "else:\n",
    "    source = bigquery.TableReference.from_string(BQ_SOURCE)\n",
    "    extract = bq.extract_table(source = source, destination_uris = [f'gs://{bucketDef.name}/{file}'])\n",
    "    print('Creating the export ...')\n",
    "    extract.result()\n",
    "    print(f'Exported the table to: gs://{bucketDef.name}/{file}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "648c19f8-677e-4219-a0ff-bcfe1fd932a0",
   "metadata": {},
   "source": [
    "list files in the bucket:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ed784872-405b-4d64-a70b-4482529b5fc4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Blob: statmike-mlops-349915, 01/01/data/fraud.csv, 1664756345743350>]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(bucketDef.list_blobs(prefix = f'{SERIES}/{EXPERIMENT}'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f40c2ea7-0649-4073-88a4-3e47af62f0c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the files in the console here:\n",
      "https://console.cloud.google.com/storage/browser/statmike-mlops-349915/01;tab=objects&project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f'Review the files in the console here:\\nhttps://console.cloud.google.com/storage/browser/{PROJECT_ID}/{SERIES};tab=objects&project={PROJECT_ID}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7685eddc",
   "metadata": {},
   "source": [
    "---\n",
    "## Create BigQuery Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c77cab37",
   "metadata": {},
   "source": [
    "List BigQuery datasets in the project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2b49b948-5fa8-4d4e-a7d3-b2adb19362c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "forecasting_8_tournament\n",
      "fraud\n",
      "model_deployment_monitoring_1961322035766362112\n"
     ]
    }
   ],
   "source": [
    "datasets = list(bq.list_datasets())\n",
    "for d in datasets:\n",
    "    print(d.dataset_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b0b3aca",
   "metadata": {},
   "source": [
    "Create the dataset if missing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "61001b78-418f-42a1-a4c4-d74a1f3aff9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = bigquery.Dataset(f\"{BQ_PROJECT}.{BQ_DATASET}\")\n",
    "ds.location = REGION\n",
    "ds.labels = {'experiment': f'{EXPERIMENT}'}\n",
    "ds = bq.create_dataset(dataset = ds, exists_ok = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11a50638-515e-49d0-b479-25e92138ae96",
   "metadata": {},
   "source": [
    "List BigQuery datasets in the project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b39f4471-38f4-4b97-8d9c-7ee3122dbee9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "forecasting_8_tournament\n",
      "fraud\n",
      "model_deployment_monitoring_1961322035766362112\n"
     ]
    }
   ],
   "source": [
    "datasets = list(bq.list_datasets())\n",
    "for d in datasets:\n",
    "    print(d.dataset_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34eeac68",
   "metadata": {},
   "source": [
    "---\n",
    "## Create BigQuery Table\n",
    "- import data from Cloud Storage Bucket\n",
    "- [Loading CSV data from Cloud Storage](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv)\n",
    "- [BigQuery Python Client](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client#google_cloud_bigquery_client_Client_extract_table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "86063a1e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Table ...\n",
      "Finished creating table: statmike-mlops-349915.fraud.fraud\n"
     ]
    }
   ],
   "source": [
    "from google.cloud.exceptions import NotFound\n",
    "try:\n",
    "    table = bq.get_table(f'{BQ_PROJECT}.{BQ_DATASET}.{BQ_TABLE}')\n",
    "    if table:\n",
    "        print(f'The table already exists: {BQ_PROJECT}.{BQ_DATASET}.{BQ_TABLE}')\n",
    "except NotFound as error:\n",
    "    print(f'Creating Table ...')\n",
    "    destination = bigquery.TableReference.from_string(f\"{BQ_PROJECT}.{BQ_DATASET}.{BQ_TABLE}\")\n",
    "    job_config = bigquery.LoadJobConfig(\n",
    "        write_disposition = 'WRITE_TRUNCATE',\n",
    "        source_format = bigquery.SourceFormat.CSV,\n",
    "        autodetect = True,\n",
    "        labels = {'experiment': f'{EXPERIMENT}'}\n",
    "    )\n",
    "    job = bq.load_table_from_uri(f\"gs://{bucketDef.name}/{file}\", destination, job_config = job_config)\n",
    "    job.result()\n",
    "    print(f'Finished creating table: {BQ_PROJECT}.{BQ_DATASET}.{BQ_TABLE}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44285e37-6ac7-478d-9709-dd11a068a4dc",
   "metadata": {},
   "source": [
    "### Retrieve and Review a Sample From The Table:\n",
    "> **Note:** The `LIMIT 5` statement does limit the number of rows returned by BigQuery to 5 but BigQuery still does a full table scan.  If you have a table larger than 1GB and want to limit the rows scanned for a quick review like then then replacing `LIMIT 5` with `TABLESAMPLE SYSTEM (1 PERCENT)` would be more efficient.  For tables under 1GB it will still return the full table.  More on [Table Sampling](https://cloud.google.com/bigquery/docs/table-sampling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "11924cf8-a040-4ee5-85ae-788cba106b56",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Time</th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "      <th>V9</th>\n",
       "      <th>...</th>\n",
       "      <th>V21</th>\n",
       "      <th>V22</th>\n",
       "      <th>V23</th>\n",
       "      <th>V24</th>\n",
       "      <th>V25</th>\n",
       "      <th>V26</th>\n",
       "      <th>V27</th>\n",
       "      <th>V28</th>\n",
       "      <th>Amount</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>282</td>\n",
       "      <td>-0.356466</td>\n",
       "      <td>0.725418</td>\n",
       "      <td>1.971749</td>\n",
       "      <td>0.831343</td>\n",
       "      <td>0.369681</td>\n",
       "      <td>-0.107776</td>\n",
       "      <td>0.751610</td>\n",
       "      <td>-0.120166</td>\n",
       "      <td>-0.420675</td>\n",
       "      <td>...</td>\n",
       "      <td>0.020804</td>\n",
       "      <td>0.424312</td>\n",
       "      <td>-0.015989</td>\n",
       "      <td>0.466754</td>\n",
       "      <td>-0.809962</td>\n",
       "      <td>0.657334</td>\n",
       "      <td>-0.043150</td>\n",
       "      <td>-0.046401</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>380</td>\n",
       "      <td>-1.299837</td>\n",
       "      <td>0.881817</td>\n",
       "      <td>1.452842</td>\n",
       "      <td>-1.293698</td>\n",
       "      <td>-0.025105</td>\n",
       "      <td>-1.170103</td>\n",
       "      <td>0.861610</td>\n",
       "      <td>-0.193934</td>\n",
       "      <td>0.592001</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.272563</td>\n",
       "      <td>-0.360853</td>\n",
       "      <td>0.223911</td>\n",
       "      <td>0.598930</td>\n",
       "      <td>-0.397705</td>\n",
       "      <td>0.637141</td>\n",
       "      <td>0.234872</td>\n",
       "      <td>0.021379</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>820</td>\n",
       "      <td>-0.937481</td>\n",
       "      <td>0.401649</td>\n",
       "      <td>1.882689</td>\n",
       "      <td>-0.362001</td>\n",
       "      <td>0.751088</td>\n",
       "      <td>-0.899262</td>\n",
       "      <td>0.880557</td>\n",
       "      <td>-0.181650</td>\n",
       "      <td>-0.211657</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.001757</td>\n",
       "      <td>0.097379</td>\n",
       "      <td>-0.324050</td>\n",
       "      <td>0.436521</td>\n",
       "      <td>0.509674</td>\n",
       "      <td>0.454116</td>\n",
       "      <td>-0.201804</td>\n",
       "      <td>-0.175439</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1193</td>\n",
       "      <td>1.130646</td>\n",
       "      <td>0.625391</td>\n",
       "      <td>0.837987</td>\n",
       "      <td>2.506543</td>\n",
       "      <td>-0.107116</td>\n",
       "      <td>-0.245548</td>\n",
       "      <td>0.099603</td>\n",
       "      <td>-0.041457</td>\n",
       "      <td>-0.867319</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.017154</td>\n",
       "      <td>-0.014311</td>\n",
       "      <td>0.086559</td>\n",
       "      <td>0.393496</td>\n",
       "      <td>0.332062</td>\n",
       "      <td>-0.066378</td>\n",
       "      <td>0.013858</td>\n",
       "      <td>0.025382</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2371</td>\n",
       "      <td>-0.878833</td>\n",
       "      <td>0.133657</td>\n",
       "      <td>2.534047</td>\n",
       "      <td>2.609811</td>\n",
       "      <td>1.510839</td>\n",
       "      <td>2.075778</td>\n",
       "      <td>-0.384729</td>\n",
       "      <td>0.230300</td>\n",
       "      <td>-0.367956</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.296422</td>\n",
       "      <td>-0.255485</td>\n",
       "      <td>-0.583298</td>\n",
       "      <td>-1.677514</td>\n",
       "      <td>0.050524</td>\n",
       "      <td>0.250409</td>\n",
       "      <td>-0.223149</td>\n",
       "      <td>-0.420764</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284802</th>\n",
       "      <td>154599</td>\n",
       "      <td>0.667714</td>\n",
       "      <td>3.041502</td>\n",
       "      <td>-5.845112</td>\n",
       "      <td>5.967587</td>\n",
       "      <td>0.213863</td>\n",
       "      <td>-1.462923</td>\n",
       "      <td>-2.688761</td>\n",
       "      <td>0.677764</td>\n",
       "      <td>-3.447596</td>\n",
       "      <td>...</td>\n",
       "      <td>0.329760</td>\n",
       "      <td>-0.941383</td>\n",
       "      <td>-0.006075</td>\n",
       "      <td>-0.958925</td>\n",
       "      <td>0.239298</td>\n",
       "      <td>-0.067356</td>\n",
       "      <td>0.821048</td>\n",
       "      <td>0.426175</td>\n",
       "      <td>6.74</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284803</th>\n",
       "      <td>90676</td>\n",
       "      <td>-2.405580</td>\n",
       "      <td>3.738235</td>\n",
       "      <td>-2.317843</td>\n",
       "      <td>1.367442</td>\n",
       "      <td>0.394001</td>\n",
       "      <td>1.919938</td>\n",
       "      <td>-3.106942</td>\n",
       "      <td>-10.764403</td>\n",
       "      <td>3.353525</td>\n",
       "      <td>...</td>\n",
       "      <td>10.005998</td>\n",
       "      <td>-2.454964</td>\n",
       "      <td>1.684957</td>\n",
       "      <td>0.118263</td>\n",
       "      <td>-1.531380</td>\n",
       "      <td>-0.695308</td>\n",
       "      <td>-0.152502</td>\n",
       "      <td>-0.138866</td>\n",
       "      <td>6.99</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284804</th>\n",
       "      <td>34634</td>\n",
       "      <td>0.333499</td>\n",
       "      <td>1.699873</td>\n",
       "      <td>-2.596561</td>\n",
       "      <td>3.643945</td>\n",
       "      <td>-0.585068</td>\n",
       "      <td>-0.654659</td>\n",
       "      <td>-2.275789</td>\n",
       "      <td>0.675229</td>\n",
       "      <td>-2.042416</td>\n",
       "      <td>...</td>\n",
       "      <td>0.469212</td>\n",
       "      <td>-0.144363</td>\n",
       "      <td>-0.317981</td>\n",
       "      <td>-0.769644</td>\n",
       "      <td>0.807855</td>\n",
       "      <td>0.228164</td>\n",
       "      <td>0.551002</td>\n",
       "      <td>0.305473</td>\n",
       "      <td>18.96</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284805</th>\n",
       "      <td>96135</td>\n",
       "      <td>-1.952933</td>\n",
       "      <td>3.541385</td>\n",
       "      <td>-1.310561</td>\n",
       "      <td>5.955664</td>\n",
       "      <td>-1.003993</td>\n",
       "      <td>0.983049</td>\n",
       "      <td>-4.587235</td>\n",
       "      <td>-4.892184</td>\n",
       "      <td>-2.516752</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.998091</td>\n",
       "      <td>1.133706</td>\n",
       "      <td>-0.041461</td>\n",
       "      <td>-0.215379</td>\n",
       "      <td>-0.865599</td>\n",
       "      <td>0.212545</td>\n",
       "      <td>0.532897</td>\n",
       "      <td>0.357892</td>\n",
       "      <td>18.96</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284806</th>\n",
       "      <td>4462</td>\n",
       "      <td>-2.303350</td>\n",
       "      <td>1.759247</td>\n",
       "      <td>-0.359745</td>\n",
       "      <td>2.330243</td>\n",
       "      <td>-0.821628</td>\n",
       "      <td>-0.075788</td>\n",
       "      <td>0.562320</td>\n",
       "      <td>-0.399147</td>\n",
       "      <td>-0.238253</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.294166</td>\n",
       "      <td>-0.932391</td>\n",
       "      <td>0.172726</td>\n",
       "      <td>-0.087330</td>\n",
       "      <td>-0.156114</td>\n",
       "      <td>-0.542628</td>\n",
       "      <td>0.039566</td>\n",
       "      <td>-0.153029</td>\n",
       "      <td>239.93</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>284807 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          Time        V1        V2        V3        V4        V5        V6  \\\n",
       "0          282 -0.356466  0.725418  1.971749  0.831343  0.369681 -0.107776   \n",
       "1          380 -1.299837  0.881817  1.452842 -1.293698 -0.025105 -1.170103   \n",
       "2          820 -0.937481  0.401649  1.882689 -0.362001  0.751088 -0.899262   \n",
       "3         1193  1.130646  0.625391  0.837987  2.506543 -0.107116 -0.245548   \n",
       "4         2371 -0.878833  0.133657  2.534047  2.609811  1.510839  2.075778   \n",
       "...        ...       ...       ...       ...       ...       ...       ...   \n",
       "284802  154599  0.667714  3.041502 -5.845112  5.967587  0.213863 -1.462923   \n",
       "284803   90676 -2.405580  3.738235 -2.317843  1.367442  0.394001  1.919938   \n",
       "284804   34634  0.333499  1.699873 -2.596561  3.643945 -0.585068 -0.654659   \n",
       "284805   96135 -1.952933  3.541385 -1.310561  5.955664 -1.003993  0.983049   \n",
       "284806    4462 -2.303350  1.759247 -0.359745  2.330243 -0.821628 -0.075788   \n",
       "\n",
       "              V7         V8        V9  ...        V21       V22       V23  \\\n",
       "0       0.751610  -0.120166 -0.420675  ...   0.020804  0.424312 -0.015989   \n",
       "1       0.861610  -0.193934  0.592001  ...  -0.272563 -0.360853  0.223911   \n",
       "2       0.880557  -0.181650 -0.211657  ...  -0.001757  0.097379 -0.324050   \n",
       "3       0.099603  -0.041457 -0.867319  ...  -0.017154 -0.014311  0.086559   \n",
       "4      -0.384729   0.230300 -0.367956  ...  -0.296422 -0.255485 -0.583298   \n",
       "...          ...        ...       ...  ...        ...       ...       ...   \n",
       "284802 -2.688761   0.677764 -3.447596  ...   0.329760 -0.941383 -0.006075   \n",
       "284803 -3.106942 -10.764403  3.353525  ...  10.005998 -2.454964  1.684957   \n",
       "284804 -2.275789   0.675229 -2.042416  ...   0.469212 -0.144363 -0.317981   \n",
       "284805 -4.587235  -4.892184 -2.516752  ...  -1.998091  1.133706 -0.041461   \n",
       "284806  0.562320  -0.399147 -0.238253  ...  -0.294166 -0.932391  0.172726   \n",
       "\n",
       "             V24       V25       V26       V27       V28  Amount  Class  \n",
       "0       0.466754 -0.809962  0.657334 -0.043150 -0.046401    0.00      0  \n",
       "1       0.598930 -0.397705  0.637141  0.234872  0.021379    0.00      0  \n",
       "2       0.436521  0.509674  0.454116 -0.201804 -0.175439    0.00      0  \n",
       "3       0.393496  0.332062 -0.066378  0.013858  0.025382    0.00      0  \n",
       "4      -1.677514  0.050524  0.250409 -0.223149 -0.420764    0.00      0  \n",
       "...          ...       ...       ...       ...       ...     ...    ...  \n",
       "284802 -0.958925  0.239298 -0.067356  0.821048  0.426175    6.74      1  \n",
       "284803  0.118263 -1.531380 -0.695308 -0.152502 -0.138866    6.99      1  \n",
       "284804 -0.769644  0.807855  0.228164  0.551002  0.305473   18.96      1  \n",
       "284805 -0.215379 -0.865599  0.212545  0.532897  0.357892   18.96      1  \n",
       "284806 -0.087330 -0.156114 -0.542628  0.039566 -0.153029  239.93      1  \n",
       "\n",
       "[284807 rows x 31 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = f\"\"\"\n",
    "SELECT *\n",
    "FROM `{BQ_PROJECT}.{BQ_DATASET}.{BQ_TABLE}` TABLESAMPLE SYSTEM (1 PERCENT)\n",
    "#LIMIT 5\n",
    "\"\"\"\n",
    "bq.query(query = query).to_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e85ccdce-e6a9-4735-9997-e5ae77a6f86c",
   "metadata": {},
   "source": [
    "### Check out this table in BigQuery Console:\n",
    "- Click: https://console.cloud.google.com/bigquery\n",
    "- Make sure project selected is the one from this notebook\n",
    "- Under Explore, expand this project and review the dataset and table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d439a1dc-dfdd-4c82-8302-8dc1ffe44d28",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Direct Link To This Project In BigQuery:\n",
      "https://console.cloud.google.com/bigquery?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"Direct Link To This Project In BigQuery:\\nhttps://console.cloud.google.com/bigquery?project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f133efb-8232-4b21-814f-23b815dc60f2",
   "metadata": {},
   "source": [
    "---\n",
    "## Review Data in BigQuery\n",
    "Additional SQL queries could be used to review the data.  This section shows moving the table to a Pandas dataframe for local review in Python:\n",
    "\n",
    "> **Note:** <p>This query only selects one column.  This means BigQuery scans less data as it does not process the other columns.  </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f4ebf6f5-a178-4011-9ab6-fa8329c578c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = f\"\"\"\n",
    "SELECT Class\n",
    "FROM `{BQ_PROJECT}.{BQ_DATASET}.{BQ_TABLE}`\n",
    "\"\"\"\n",
    "df = bq.query(query = query).to_dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "674f716e-8b9d-4c92-9b46-c166b869f6ae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    284315\n",
       "1       492\n",
       "Name: Class, dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Class'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ba5d17e5-07c3-48fd-b682-aa471f3548f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.998273\n",
       "1    0.001727\n",
       "Name: Class, dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Class'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35b34894",
   "metadata": {},
   "source": [
    "---\n",
    "## Prepare Data for Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c982157",
   "metadata": {},
   "source": [
    "Create a prepped version of the data with test/train splits using SQL DDL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "a50becbb-60ab-45f9-acd5-9beb5b6755f0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<google.cloud.bigquery.table._EmptyRowIterator at 0x7f8321ae70d0>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = f\"\"\"\n",
    "CREATE TABLE IF NOT EXISTS `{BQ_PROJECT}.{BQ_DATASET}.{BQ_TABLE}_prepped` AS\n",
    "WITH add_id AS(SELECT *, GENERATE_UUID() transaction_id FROM `{BQ_PROJECT}.{BQ_DATASET}.{BQ_TABLE}`)\n",
    "SELECT *,\n",
    "    CASE \n",
    "        WHEN MOD(ABS(FARM_FINGERPRINT(transaction_id)),10) < 8 THEN \"TRAIN\" \n",
    "        WHEN MOD(ABS(FARM_FINGERPRINT(transaction_id)),10) < 9 THEN \"VALIDATE\"\n",
    "        ELSE \"TEST\"\n",
    "    END AS splits\n",
    "FROM add_id\n",
    "\"\"\"\n",
    "job = bq.query(query = query)\n",
    "job.result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "77d6e2a9-ec35-477c-9d50-429326acc41d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.274"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(job.ended-job.started).total_seconds()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a991ff73-a545-4eed-a709-b8292efd89ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "if job.estimated_bytes_processed:\n",
    "    print(f'{job.estimated_bytes_processed/1000000} MB')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "372b2c41",
   "metadata": {},
   "source": [
    "Review the test/train split:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "4a94adb9-81d7-4710-ac50-e0d93dd5523a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>splits</th>\n",
       "      <th>Count</th>\n",
       "      <th>Percentage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>VALIDATE</td>\n",
       "      <td>28244</td>\n",
       "      <td>9.916891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TRAIN</td>\n",
       "      <td>228061</td>\n",
       "      <td>80.075630</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TEST</td>\n",
       "      <td>28502</td>\n",
       "      <td>10.007479</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     splits   Count  Percentage\n",
       "0  VALIDATE   28244    9.916891\n",
       "1     TRAIN  228061   80.075630\n",
       "2      TEST   28502   10.007479"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = f\"\"\"\n",
    "SELECT splits, count(*) as Count, 100*count(*) / (sum(count(*)) OVER()) as Percentage\n",
    "FROM `{BQ_PROJECT}.{BQ_DATASET}.{BQ_TABLE}_prepped`\n",
    "GROUP BY splits\n",
    "\"\"\"\n",
    "bq.query(query = query).to_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3006ed11",
   "metadata": {},
   "source": [
    "Retrieve a subset of the data to a Pandas dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2ea70141-07db-46b0-a31e-0968befcd37b",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = f\"\"\"\n",
    "SELECT * \n",
    "FROM `{BQ_PROJECT}.{BQ_DATASET}.{BQ_TABLE}_prepped`\n",
    "LIMIT 5\n",
    "\"\"\"\n",
    "data = bq.query(query = query).to_dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "326d2b3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Time</th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "      <th>V9</th>\n",
       "      <th>...</th>\n",
       "      <th>V23</th>\n",
       "      <th>V24</th>\n",
       "      <th>V25</th>\n",
       "      <th>V26</th>\n",
       "      <th>V27</th>\n",
       "      <th>V28</th>\n",
       "      <th>Amount</th>\n",
       "      <th>Class</th>\n",
       "      <th>transaction_id</th>\n",
       "      <th>splits</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>35337</td>\n",
       "      <td>1.092844</td>\n",
       "      <td>-0.013230</td>\n",
       "      <td>1.359829</td>\n",
       "      <td>2.731537</td>\n",
       "      <td>-0.707357</td>\n",
       "      <td>0.873837</td>\n",
       "      <td>-0.796130</td>\n",
       "      <td>0.437707</td>\n",
       "      <td>0.396770</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.167647</td>\n",
       "      <td>0.027557</td>\n",
       "      <td>0.592115</td>\n",
       "      <td>0.219695</td>\n",
       "      <td>0.036970</td>\n",
       "      <td>0.010984</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>a1b10547-d270-48c0-b902-7a0f735dadc7</td>\n",
       "      <td>TEST</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60481</td>\n",
       "      <td>1.238973</td>\n",
       "      <td>0.035226</td>\n",
       "      <td>0.063003</td>\n",
       "      <td>0.641406</td>\n",
       "      <td>-0.260893</td>\n",
       "      <td>-0.580097</td>\n",
       "      <td>0.049938</td>\n",
       "      <td>-0.034733</td>\n",
       "      <td>0.405932</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.057718</td>\n",
       "      <td>0.104983</td>\n",
       "      <td>0.537987</td>\n",
       "      <td>0.589563</td>\n",
       "      <td>-0.046207</td>\n",
       "      <td>-0.006212</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>814c62c8-ade4-47d5-bf83-313b0aafdee5</td>\n",
       "      <td>TEST</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>139587</td>\n",
       "      <td>1.870539</td>\n",
       "      <td>0.211079</td>\n",
       "      <td>0.224457</td>\n",
       "      <td>3.889486</td>\n",
       "      <td>-0.380177</td>\n",
       "      <td>0.249799</td>\n",
       "      <td>-0.577133</td>\n",
       "      <td>0.179189</td>\n",
       "      <td>-0.120462</td>\n",
       "      <td>...</td>\n",
       "      <td>0.180776</td>\n",
       "      <td>-0.060226</td>\n",
       "      <td>-0.228979</td>\n",
       "      <td>0.080827</td>\n",
       "      <td>0.009868</td>\n",
       "      <td>-0.036997</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>d08a1bfa-85c5-4f1b-9537-1c5a93e6afd0</td>\n",
       "      <td>TEST</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>162908</td>\n",
       "      <td>-3.368339</td>\n",
       "      <td>-1.980442</td>\n",
       "      <td>0.153645</td>\n",
       "      <td>-0.159795</td>\n",
       "      <td>3.847169</td>\n",
       "      <td>-3.516873</td>\n",
       "      <td>-1.209398</td>\n",
       "      <td>-0.292122</td>\n",
       "      <td>0.760543</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.171627</td>\n",
       "      <td>0.214333</td>\n",
       "      <td>-0.159652</td>\n",
       "      <td>-0.060883</td>\n",
       "      <td>1.294977</td>\n",
       "      <td>0.120503</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>802f3307-8e5a-4475-b795-5d5d8d7d0120</td>\n",
       "      <td>TEST</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>165236</td>\n",
       "      <td>2.180149</td>\n",
       "      <td>0.218732</td>\n",
       "      <td>-2.637726</td>\n",
       "      <td>0.348776</td>\n",
       "      <td>1.063546</td>\n",
       "      <td>-1.249197</td>\n",
       "      <td>0.942021</td>\n",
       "      <td>-0.547652</td>\n",
       "      <td>-0.087823</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.176957</td>\n",
       "      <td>0.563779</td>\n",
       "      <td>0.730183</td>\n",
       "      <td>0.707494</td>\n",
       "      <td>-0.131066</td>\n",
       "      <td>-0.090428</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>c8a5b93a-1598-4689-80be-4f9f5df0b8ce</td>\n",
       "      <td>TEST</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Time        V1        V2        V3        V4        V5        V6  \\\n",
       "0   35337  1.092844 -0.013230  1.359829  2.731537 -0.707357  0.873837   \n",
       "1   60481  1.238973  0.035226  0.063003  0.641406 -0.260893 -0.580097   \n",
       "2  139587  1.870539  0.211079  0.224457  3.889486 -0.380177  0.249799   \n",
       "3  162908 -3.368339 -1.980442  0.153645 -0.159795  3.847169 -3.516873   \n",
       "4  165236  2.180149  0.218732 -2.637726  0.348776  1.063546 -1.249197   \n",
       "\n",
       "         V7        V8        V9  ...       V23       V24       V25       V26  \\\n",
       "0 -0.796130  0.437707  0.396770  ... -0.167647  0.027557  0.592115  0.219695   \n",
       "1  0.049938 -0.034733  0.405932  ... -0.057718  0.104983  0.537987  0.589563   \n",
       "2 -0.577133  0.179189 -0.120462  ...  0.180776 -0.060226 -0.228979  0.080827   \n",
       "3 -1.209398 -0.292122  0.760543  ... -1.171627  0.214333 -0.159652 -0.060883   \n",
       "4  0.942021 -0.547652 -0.087823  ... -0.176957  0.563779  0.730183  0.707494   \n",
       "\n",
       "        V27       V28  Amount  Class                        transaction_id  \\\n",
       "0  0.036970  0.010984     0.0      0  a1b10547-d270-48c0-b902-7a0f735dadc7   \n",
       "1 -0.046207 -0.006212     0.0      0  814c62c8-ade4-47d5-bf83-313b0aafdee5   \n",
       "2  0.009868 -0.036997     0.0      0  d08a1bfa-85c5-4f1b-9537-1c5a93e6afd0   \n",
       "3  1.294977  0.120503     0.0      0  802f3307-8e5a-4475-b795-5d5d8d7d0120   \n",
       "4 -0.131066 -0.090428     0.0      0  c8a5b93a-1598-4689-80be-4f9f5df0b8ce   \n",
       "\n",
       "   splits  \n",
       "0    TEST  \n",
       "1    TEST  \n",
       "2    TEST  \n",
       "3    TEST  \n",
       "4    TEST  \n",
       "\n",
       "[5 rows x 33 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82d575d1",
   "metadata": {},
   "source": [
    "---\n",
    "## Remove Resources\n",
    "see notebook \"99 - Cleanup\""
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-11.m104",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-11:m104"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}