penguins-cohort.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "00b7ba7b-433c-463c-8e5e-8b975a5be463",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Penguins in Production\n",
    "\n",
    "This notebook aims to create a [SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) to build an end-to-end Machine Learning system to solve the problem of classifying penguin species.\n",
    "\n",
    "This example uses the [Penguins dataset](https://www.kaggle.com/parulpandey/palmer-archipelago-antarctica-penguin-data).\n",
    "\n",
    "<img src='https://imgur.com/orZWHly.png' alt='Penguins dataset' width=\"900\">\n",
    "\n",
    "Amazon SageMaker is free to try. Your free tier starts from the first month you create your first SageMaker resource and lasts two months. Check out the  [Amazon SageMaker Pricing](https://aws.amazon.com/sagemaker/pricing/) for more information. Also, we'll be working extensively with [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html) and the [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/). Keep their documentation handy.\n",
    "\n",
    "This notebook was created by [Santiago L. Valdarrama](https://twitter.com/svpino) as part of the [Machine Learning School](https://www.ml.school) program."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db6203b0-5a40-4a14-b9bb-6e092f1bb2e7",
   "metadata": {},
   "source": [
    "Let's ensure we are running the latest version of the SakeMaker SDK. **Restart the Kernel** after you run the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "72eb4984-3c21-4033-be45-2de2cfc1257c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "awscli 1.27.162 requires PyYAML<5.5,>=3.10, but you have pyyaml 6.0 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mName: sagemaker\n",
      "Version: 2.165.0\n",
      "Summary: Open source library for training and deploying models on Amazon SageMaker.\n",
      "Home-page: https://github.com/aws/sagemaker-python-sdk/\n",
      "Author: Amazon Web Services\n",
      "Author-email: \n",
      "License: Apache License 2.0\n",
      "Location: /usr/local/lib/python3.8/site-packages\n",
      "Requires: attrs, boto3, cloudpickle, google-pasta, importlib-metadata, jsonschema, numpy, packaging, pandas, pathos, platformdirs, protobuf, protobuf3-to-dict, PyYAML, schema, smdebug-rulesconfig, tblib\n",
      "Required-by: \n"
     ]
    }
   ],
   "source": [
    "!pip install -q --upgrade pip\n",
    "!pip install -q --upgrade awscli boto3\n",
    "!pip install -q --upgrade scikit-learn==0.23.2\n",
    "!pip install -q --upgrade PyYAML==6.0\n",
    "!pip install -q --upgrade sagemaker==2.165.0\n",
    "!pip show sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "212a0538-62b5-4875-a59c-1a4e1a833a07",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86227bb7-0351-4d59-9576-c881130389f3",
   "metadata": {},
   "source": [
    "# Initial Setup\n",
    "\n",
    "Let's start by preparing the S3 bucket where we will organize every resource we are going to use during the program. Make sure you set `BUCKET` to the bucket name you want to use. This name has to be unique. The [command line interface](https://docs.aws.amazon.com/cli/latest/index.html) is a simple way to interact with the AWS services. You can combine Python code with bash commands in the same notebook cell, which makes notebooks a very flexible tool.\n",
    "\n",
    "If you want to create a bucket in a region other than `us-east-1`, use this command instead:\n",
    "\n",
    "```\n",
    "!aws s3api create-bucket --bucket $BUCKET --create-bucket-configuration LocationConstraint=$region\n",
    "```\n",
    "\n",
    "The `LocationConstraint` argument should specify the region where you want to create the bucket.\n",
    "\n",
    "After we have a bucket, we can download the [Penguins dataset](https://www.kaggle.com/parulpandey/palmer-archipelago-antarctica-penguin-data) and store it in a folder inside the bucket. Our SageMaker Pipeline will use this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3ea8d2fd-7dd4-43c8-b9d9-5372448d72a6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"Location\": \"/mlschool\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "BUCKET = \"mlschool\"\n",
    "\n",
    "!aws s3api create-bucket --bucket $BUCKET"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "882a8be9-dc31-4df1-9009-3b2d0284cb70",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset S3 location: s3://mlschool/penguins/data.csv\n"
     ]
    }
   ],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import urllib.request\n",
    "import pandas as pd\n",
    "\n",
    "from pathlib import Path\n",
    "\n",
    "\n",
    "PENGUINS_FOLDER = Path(\"penguins\")\n",
    "S3_FILEPATH = f\"s3://{BUCKET}/{PENGUINS_FOLDER}\"\n",
    "LOCAL_FILEPATH = Path(PENGUINS_FOLDER)/ \"data.csv\"\n",
    "\n",
    "# Create the local folder if it doesn't exist.\n",
    "PENGUINS_FOLDER.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "# Download the official Penguins dataset and store it locally.\n",
    "urllib.request.urlretrieve(\n",
    "    \"https://storage.googleapis.com/download.tensorflow.org/data/palmer_penguins/penguins_size.csv\", \n",
    "    LOCAL_FILEPATH\n",
    ")\n",
    "\n",
    "# Upload the dataset to S3. We need to do this to make it available to \n",
    "# the preprocessing step.\n",
    "INPUT_DATA_URI = sagemaker.s3.S3Uploader.upload(\n",
    "    local_path=str(LOCAL_FILEPATH), \n",
    "    desired_s3_uri=S3_FILEPATH,\n",
    ")\n",
    "\n",
    "print(f\"Dataset S3 location: {INPUT_DATA_URI}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59e1efbc-5f66-4b44-9555-68a4731e8e7b",
   "metadata": {},
   "source": [
    "We can now load and display the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a19b546a-6884-483b-8cda-0521e2656a99",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>species</th>\n",
       "      <th>island</th>\n",
       "      <th>culmen_length_mm</th>\n",
       "      <th>culmen_depth_mm</th>\n",
       "      <th>flipper_length_mm</th>\n",
       "      <th>body_mass_g</th>\n",
       "      <th>sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>39.1</td>\n",
       "      <td>18.7</td>\n",
       "      <td>181.0</td>\n",
       "      <td>3750.0</td>\n",
       "      <td>MALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>39.5</td>\n",
       "      <td>17.4</td>\n",
       "      <td>186.0</td>\n",
       "      <td>3800.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>40.3</td>\n",
       "      <td>18.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>3250.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>36.7</td>\n",
       "      <td>19.3</td>\n",
       "      <td>193.0</td>\n",
       "      <td>3450.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>339</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>340</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>46.8</td>\n",
       "      <td>14.3</td>\n",
       "      <td>215.0</td>\n",
       "      <td>4850.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>341</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>50.4</td>\n",
       "      <td>15.7</td>\n",
       "      <td>222.0</td>\n",
       "      <td>5750.0</td>\n",
       "      <td>MALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>45.2</td>\n",
       "      <td>14.8</td>\n",
       "      <td>212.0</td>\n",
       "      <td>5200.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>343</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>49.9</td>\n",
       "      <td>16.1</td>\n",
       "      <td>213.0</td>\n",
       "      <td>5400.0</td>\n",
       "      <td>MALE</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>344 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    species     island  culmen_length_mm  culmen_depth_mm  flipper_length_mm  \\\n",
       "0    Adelie  Torgersen              39.1             18.7              181.0   \n",
       "1    Adelie  Torgersen              39.5             17.4              186.0   \n",
       "2    Adelie  Torgersen              40.3             18.0              195.0   \n",
       "3    Adelie  Torgersen               NaN              NaN                NaN   \n",
       "4    Adelie  Torgersen              36.7             19.3              193.0   \n",
       "..      ...        ...               ...              ...                ...   \n",
       "339  Gentoo     Biscoe               NaN              NaN                NaN   \n",
       "340  Gentoo     Biscoe              46.8             14.3              215.0   \n",
       "341  Gentoo     Biscoe              50.4             15.7              222.0   \n",
       "342  Gentoo     Biscoe              45.2             14.8              212.0   \n",
       "343  Gentoo     Biscoe              49.9             16.1              213.0   \n",
       "\n",
       "     body_mass_g     sex  \n",
       "0         3750.0    MALE  \n",
       "1         3800.0  FEMALE  \n",
       "2         3250.0  FEMALE  \n",
       "3            NaN     NaN  \n",
       "4         3450.0  FEMALE  \n",
       "..           ...     ...  \n",
       "339          NaN     NaN  \n",
       "340       4850.0  FEMALE  \n",
       "341       5750.0    MALE  \n",
       "342       5200.0  FEMALE  \n",
       "343       5400.0    MALE  \n",
       "\n",
       "[344 rows x 7 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(LOCAL_FILEPATH)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "843c0a29-57b3-4bb0-bb12-0c0225938ba7",
   "metadata": {},
   "source": [
    "## Permissions\n",
    "\n",
    "To run this notebook you need to update the Execution Policy assigned to SageMaker's Execution Role with the following permissions:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Sid\": \"IAM0\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"iam:CreateServiceLinkedRole\"\n",
    "            ],\n",
    "            \"Resource\": \"*\",\n",
    "            \"Condition\": {\n",
    "                \"StringEquals\": {\n",
    "                    \"iam:AWSServiceName\": [\n",
    "                        \"autoscaling.amazonaws.com\",\n",
    "                        \"ec2scheduled.amazonaws.com\",\n",
    "                        \"elasticloadbalancing.amazonaws.com\",\n",
    "                        \"spot.amazonaws.com\",\n",
    "                        \"spotfleet.amazonaws.com\",\n",
    "                        \"transitgateway.amazonaws.com\"\n",
    "                    ]\n",
    "                }\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"Sid\": \"IAM1\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"iam:CreateRole\",\n",
    "                \"iam:PassRole\",\n",
    "                \"iam:AttachRolePolicy\"\n",
    "            ],\n",
    "            \"Resource\": \"*\"\n",
    "        },\n",
    "        {\n",
    "            \"Sid\": \"Lambda\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"lambda:CreateFunction\",\n",
    "                \"lambda:DeleteFunction\",\n",
    "                \"lambda:InvokeFunctionUrl\",\n",
    "                \"lambda:InvokeFunction\",\n",
    "                \"lambda:UpdateFunctionCode\",\n",
    "                \"lambda:InvokeAsync\"\n",
    "            ],\n",
    "            \"Resource\": \"*\"\n",
    "        },\n",
    "        {\n",
    "            \"Sid\": \"SageMaker\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"sagemaker:UpdateDomain\",\n",
    "                \"sagemaker:UpdateUserProfile\"\n",
    "            ],\n",
    "            \"Resource\": \"*\"\n",
    "        },\n",
    "        {\n",
    "            \"Sid\": \"CloudWatch\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"cloudwatch:PutMetricData\",\n",
    "                \"cloudwatch:GetMetricData\",\n",
    "                \"cloudwatch:DescribeAlarmsForMetric\",\n",
    "                \"logs:CreateLogStream\",\n",
    "                \"logs:PutLogEvents\",\n",
    "                \"logs:CreateLogGroup\",\n",
    "                \"logs:DescribeLogStreams\"\n",
    "            ],\n",
    "            \"Resource\": \"*\"\n",
    "        },\n",
    "        {\n",
    "            \"Sid\": \"ECR\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"ecr:GetAuthorizationToken\",\n",
    "                \"ecr:BatchCheckLayerAvailability\",\n",
    "                \"ecr:GetDownloadUrlForLayer\",\n",
    "                \"ecr:BatchGetImage\"\n",
    "            ],\n",
    "            \"Resource\": \"*\"\n",
    "        },\n",
    "        {\n",
    "            \"Sid\": \"S3\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"s3:CreateBucket\",\n",
    "                \"s3:ListBucket\",\n",
    "                \"s3:GetBucketLocation\",\n",
    "                \"s3:PutObject\",\n",
    "                \"s3:GetObject\",\n",
    "                \"s3:DeleteObject\"\n",
    "            ],\n",
    "            \"Resource\": \"arn:aws:s3:::*\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "237d3498-3c01-41de-97af-2dbf47630435",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SageMaker Execution Role: arn:aws:iam::325223348818:role/service-role/AmazonSageMaker-ExecutionRole-20230312T160501\n"
     ]
    }
   ],
   "source": [
    "role = sagemaker.get_execution_role()\n",
    "print(f\"SageMaker Execution Role: {role}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9992fc69-4868-4e50-a50e-76da237f70ac",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Session 1 - Data Preprocessing\n",
    "\n",
    "This session aims to build a simple [SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) with one step to preprocess the [Penguins dataset](https://www.kaggle.com/parulpandey/palmer-archipelago-antarctica-penguin-data). We'll use a [Processing Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing) with a [SKLearnProcessor](https://sagemaker.readthedocs.io/en/stable/frameworks/sklearn/sagemaker.sklearn.html#scikit-learn-processor) to execute a preprocessing script. Check the [SageMaker Pipelines Overview](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) for an introduction to the fundamental components of a SageMaker Pipeline.\n",
    "\n",
    "Here is what the Pipeline will look like at the end of this session:\n",
    "\n",
    "<img src='penguins/images/session1-pipeline.png' alt='Session 1 Pipeline' width=\"600\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "915b1d0b-d9da-4529-aca5-fd1c08a36f48",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import json\n",
    "import numpy as np\n",
    "import argparse\n",
    "import tempfile\n",
    "\n",
    "from botocore.exceptions import ClientError\n",
    "from sagemaker.inputs import FileSystemInput\n",
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "from sagemaker.processing import ScriptProcessor\n",
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "from sagemaker.workflow.steps import ProcessingStep\n",
    "from sagemaker.workflow.model_step import ModelStep\n",
    "from sagemaker.workflow.pipeline_context import PipelineSession\n",
    "from sagemaker.workflow.parameters import ParameterInteger, ParameterString, ParameterFloat\n",
    "from sagemaker.workflow.pipeline import Pipeline\n",
    "from sagemaker.workflow.steps import CacheConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef66efe2-6a01-41b5-ba8a-9c82c480994c",
   "metadata": {},
   "source": [
    "Let's start by defining a few variables we'll use throughout this notebook:\n",
    "\n",
    "* `sagemaker_client`: We'll use a [boto3 SageMaker Client](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html) instance to access SageMaker.\n",
    "* `iam_client`: We'll use a [boto3 IAM Client](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam.html) instance to access IAM.\n",
    "* `role`: This is the execution role attached to this notebook. We can use this role with any of the SageMaker services that need it to ensure they run with the appropriate permissions.\n",
    "* `region`: The current region attached to our session. \n",
    "* `sagemaker_session`: The current SageMaker session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7bb1e9bf-25d8-401b-a7d9-0455b1a7434d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "iam_client = boto3.client(\"iam\")\n",
    "sagemaker_client = boto3.client(\"sagemaker\")\n",
    "region = boto3.Session().region_name\n",
    "sagemaker_session = sagemaker.session.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "587d06e0-b711-4e3d-b424-6fa611a51f94",
   "metadata": {},
   "source": [
    "## Step 1 - Preprocessing the Dataset\n",
    "\n",
    "Let's create a script to do feature engineering on the original dataset. We will run this script using a SageMaker Processing Job later in this session.\n",
    "\n",
    "The script should split the data into train, validation, and test sets so we can later train and evaluate a model. We will also save the Scikit-Learn pipeline that we use to preprocess the data to use it during inference time and the list of target classes.\n",
    "\n",
    "The script uses the [np.split()](https://numpy.org/doc/stable/reference/generated/numpy.split.html) function to split the dataset into three sets in the following way:\n",
    "\n",
    "1. The train set will use the top 70% of the data.\n",
    "2. The validation set will use 15% of the data, starting with the sample after the 70% used for the train set.\n",
    "3. Finally, the test set will use the remaining 15% of the data.\n",
    "\n",
    "Pay special attention to the way the Scikit-Learn pipeline `preprocessor` is used to process the three sets:\n",
    "\n",
    "* First, we use the `fit_transform()` to fit the pipeline on the train set.\n",
    "* Then, we consecutively transform the validation and test sets using `transform()`.\n",
    "\n",
    "Always use `fit_transform()` on the training data to fit the scaling parameters we need to transform the data. For example, `fit_transform()` will learn the mean and variance of the features of the training set. It can then use these same parameters to scale the validation and test sets.\n",
    "\n",
    "That's why we want to save this Scikit-Learn pipeline to use later to scale production data using the same parameters we learned on the train set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "fb6ba7c0-1bd6-4fe5-8b7f-f6cbdfd3846c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguins/preprocessor.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {PENGUINS_FOLDER}/preprocessor.py\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tempfile\n",
    "\n",
    "from pathlib import Path\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, LabelEncoder, StandardScaler\n",
    "from pickle import dump\n",
    "\n",
    "\n",
    "# This is the location where the SageMaker Processing job\n",
    "# will save the input dataset.\n",
    "BASE_DIRECTORY = \"/opt/ml/processing\"\n",
    "DATA_FILEPATH = Path(BASE_DIRECTORY) / \"input\" / \"data.csv\"\n",
    "\n",
    "\n",
    "def _save_splits(base_directory, train, validation, test):\n",
    "    \"\"\"\n",
    "    One of the goals of this script is to output the three\n",
    "    dataset splits. This function will save each of these\n",
    "    splits to disk.\n",
    "    \"\"\"\n",
    "    \n",
    "    train_path = Path(base_directory) / \"train\" \n",
    "    validation_path = Path(base_directory) / \"validation\" \n",
    "    test_path = Path(base_directory) / \"test\"\n",
    "    \n",
    "    train_path.mkdir(parents=True, exist_ok=True)\n",
    "    validation_path.mkdir(parents=True, exist_ok=True)\n",
    "    test_path.mkdir(parents=True, exist_ok=True)\n",
    "    \n",
    "    pd.DataFrame(train).to_csv(train_path / \"train.csv\", header=False, index=False)\n",
    "    pd.DataFrame(validation).to_csv(validation_path / \"validation.csv\", header=False, index=False)\n",
    "    pd.DataFrame(test).to_csv(test_path / \"test.csv\", header=False, index=False)\n",
    "    \n",
    "\n",
    "def _save_pipeline(base_directory, pipeline):\n",
    "    \"\"\"\n",
    "    Saves the Scikit-Learn pipeline that we used to\n",
    "    preprocess the data.\n",
    "    \"\"\"\n",
    "    pipeline_path = Path(base_directory) / \"pipeline\"\n",
    "    pipeline_path.mkdir(parents=True, exist_ok=True)\n",
    "    dump(pipeline, open(pipeline_path / \"pipeline.pkl\", 'wb'))\n",
    "    \n",
    "\n",
    "def _save_classes(base_directory, classes):\n",
    "    \"\"\"\n",
    "    Saves the list of classes from the dataset.\n",
    "    \"\"\"\n",
    "    path = Path(base_directory) / \"classes\"\n",
    "    path.mkdir(parents=True, exist_ok=True)\n",
    "    \n",
    "    print(\"CLASSES\", np.asarray(classes))\n",
    "\n",
    "    np.asarray(classes).tofile(path / \"classes.csv\", sep = \",\") \n",
    "    \n",
    "    \n",
    "def _save_baseline(base_directory, X_train, X_test):\n",
    "    \"\"\"\n",
    "    During the data and quality monitoring steps, we will need a baseline\n",
    "    to compute constraints and statistics. This function will save that\n",
    "    baseline to the disk.\n",
    "    \"\"\"\n",
    "    \n",
    "    for split, data in [(\"train\", X_train), (\"test\", X_test)]:\n",
    "        baseline_path = Path(base_directory) / f\"{split}-baseline\" \n",
    "        baseline_path.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "        df = data.copy().dropna()\n",
    "        df.to_json(baseline_path / f\"{split}-baseline.json\", orient='records', lines=True)\n",
    "    \n",
    "    \n",
    "def preprocess(base_directory, data_filepath):\n",
    "    \"\"\"\n",
    "    Preprocesses the supplied raw dataset and splits it into a train, validation,\n",
    "    and a test set.\n",
    "    \"\"\"\n",
    "    \n",
    "    df = pd.read_csv(data_filepath)\n",
    "    \n",
    "    numerical_columns = [column for column in df.columns if df[column].dtype in [\"int64\", \"float64\"]]\n",
    "    \n",
    "    numerical_preprocessor = Pipeline(steps=[\n",
    "        (\"imputer\", SimpleImputer(strategy=\"mean\")),\n",
    "        (\"scaler\", StandardScaler())\n",
    "    ])\n",
    "\n",
    "    categorical_preprocessor = Pipeline(steps=[\n",
    "        (\"imputer\", SimpleImputer(strategy=\"most_frequent\")),\n",
    "        (\"onehot\", OneHotEncoder(handle_unknown=\"ignore\"))\n",
    "    ])\n",
    "\n",
    "    preprocessor = ColumnTransformer(\n",
    "        transformers=[\n",
    "            (\"numerical\", numerical_preprocessor, numerical_columns),\n",
    "            (\"categorical\", categorical_preprocessor, [\"island\"]),\n",
    "        ]\n",
    "    )\n",
    "    \n",
    "\n",
    "    X = df.drop([\"sex\"], axis=1)\n",
    "    columns = list(X.columns)\n",
    "    \n",
    "    X = X.to_numpy()\n",
    "    \n",
    "    np.random.shuffle(X)\n",
    "    train, validation, test = np.split(X, [int(.7 * len(X)), int(.85 * len(X))])\n",
    "    \n",
    "    X_train = pd.DataFrame(train, columns=columns)\n",
    "    X_validation = pd.DataFrame(validation, columns=columns)\n",
    "    X_test = pd.DataFrame(test, columns=columns)\n",
    "    \n",
    "    _save_baseline(base_directory, X_train, X_test)\n",
    "    \n",
    "    y_train = X_train.species\n",
    "    y_validation = X_validation.species\n",
    "    y_test = X_test.species\n",
    "    \n",
    "    label_encoder = LabelEncoder()\n",
    "    \n",
    "    y_train = label_encoder.fit_transform(y_train)\n",
    "    y_validation = label_encoder.transform(y_validation)\n",
    "    y_test = label_encoder.transform(y_test)\n",
    "    \n",
    "    X_train.drop([\"species\"], axis=1, inplace=True)\n",
    "    X_validation.drop([\"species\"], axis=1, inplace=True)\n",
    "    X_test.drop([\"species\"], axis=1, inplace=True)\n",
    "\n",
    "    X_train = preprocessor.fit_transform(X_train)\n",
    "    X_validation = preprocessor.transform(X_validation)\n",
    "    X_test = preprocessor.transform(X_test)\n",
    "    \n",
    "    train = np.concatenate((X_train, np.expand_dims(y_train, axis=1)), axis=1)\n",
    "    validation = np.concatenate((X_validation, np.expand_dims(y_validation, axis=1)), axis=1)\n",
    "    test = np.concatenate((X_test, np.expand_dims(y_test, axis=1)), axis=1)\n",
    "    \n",
    "    _save_splits(base_directory, train, validation, test)\n",
    "    _save_pipeline(base_directory, pipeline=preprocessor)\n",
    "    _save_classes(base_directory, label_encoder.classes_)\n",
    "        \n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    preprocess(BASE_DIRECTORY, DATA_FILEPATH)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "214fbdcb-8ee8-4104-bcd3-d7329693299e",
   "metadata": {},
   "source": [
    "## Step 2 - Testing the Preprocessing Script\n",
    "\n",
    "We can now load the script we just created and run it locally to ensure it outputs every file we need.\n",
    "\n",
    "We will set up a SageMaker Processing Job to run this script, but we always want to test the code locally. In this case, we can call the `preprocess()` function with the local directory and the local copy of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "d1f122a4-acff-4687-91b9-bfef13567d88",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CLASSES ['Adelie' 'Chinstrap' 'Gentoo']\n",
      "Folders: ['train-baseline', 'test-baseline', 'train', 'validation', 'test', 'pipeline', 'classes']\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/4554.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/4624.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/4654.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/4725.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/4755.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/4825.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/4856.jsonl...\n",
      "Generating 334 predictions...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/4926.jsonl...\n",
      "Generating 334 predictions...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/4956.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/5026.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/5056.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/5127.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/5157.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/5227.jsonl...\n"
     ]
    }
   ],
   "source": [
    "from penguins.preprocessor import preprocess\n",
    "\n",
    "with tempfile.TemporaryDirectory() as directory:\n",
    "    preprocess(\n",
    "        base_directory=directory, \n",
    "        data_filepath=LOCAL_FILEPATH\n",
    "    )\n",
    "    \n",
    "    print(f\"Folders: {os.listdir(directory)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3938a1d-d5bd-4967-ad65-a945d5df1f61",
   "metadata": {},
   "source": [
    "## Step 3 - Pipeline Configuration\n",
    "\n",
    "When creating a SageMaker Pipeline, we can specify a list of parameters we can use on individual pipeline steps. To read more about these parameters, check [Pipeline Parameters](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-parameters.html).\n",
    "\n",
    "These are the parameters that we need right now:\n",
    "\n",
    "* `dataset_location`: This parameter represents the dataset's location in S3. We will use this parameter to indicate the SageMaker Processing Job where to find the dataset. The Processing Job will download the dataset from S3 and make it available on the instance running the script.\n",
    "* `preprocessor_destination`: We need to define the location where the SageMaker Processing Job will store the output. When it finishes, the Processing Job will copy the script's output to the S3 location specified by this parameter. By default, SageMaker uploads the output of a job to a custom location in S3, but unfortunately, if we rely on that functionality, we can't cache the Processing Step in the Pipeline.\n",
    "* `train_dataset_baseline_destination`: This parameter represents the location where we will store the train dataset to compute constraints and statistic baselines in Session 6.\n",
    "* `test_dataset_baseline_destination`: This parameter represents the location where we will store the test dataset to compute constraints and statistic baselines in Session 6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "86de7edd-18b0-40d1-ac8e-0f3ef4469be6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset_location = ParameterString(\n",
    "    name=\"dataset_location\",\n",
    "    default_value=INPUT_DATA_URI,\n",
    ")\n",
    "\n",
    "preprocessor_destination = ParameterString(\n",
    "    name=\"preprocessor_destination\",\n",
    "    default_value=f\"{S3_FILEPATH}/preprocessing\",\n",
    ")\n",
    "\n",
    "train_dataset_baseline_destination = ParameterString(\n",
    "    name=\"train_dataset_baseline_destination\",\n",
    "    default_value=f\"{S3_FILEPATH}/preprocessing/baselines/train\",\n",
    ")\n",
    "\n",
    "test_dataset_baseline_destination = ParameterString(\n",
    "    name=\"test_dataset_baseline_destination\",\n",
    "    default_value=f\"{S3_FILEPATH}/preprocessing/baselines/test\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0b42440-b3c0-4ded-b578-52e21f5fbfdd",
   "metadata": {},
   "source": [
    "## Step 4 - Caching Pipeline Steps\n",
    "\n",
    "While building a pipeline, you only want to rerun every step if you expect a different result. To accomplish this, you can instruct SageMaker to reuse the result of a previous successful run of a pipeline step. You can find more information about this topic in [Caching Pipeline Steps](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-caching.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "c7e0a332-e54e-4e57-97c7-9b54f4058738",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cache_config = CacheConfig(\n",
    "    enable_caching=True, \n",
    "    expire_after=\"15d\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9771e08-5560-492a-845e-f06988b6ae1b",
   "metadata": {},
   "source": [
    "## Step 5 - Setting up a Processing Step\n",
    "\n",
    "The first step we need in the pipeline is a [Processing Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing) to run the preprocessing script. This Processing Step will create a SageMaker Processing Job in the background, run the script, and upload the output to S3. You can use Processing Jobs to perform data preprocessing, post-processing, feature engineering, data validation, and model evaluation. Check the [ProcessingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.ProcessingStep) SageMaker's SDK documentation for more information.\n",
    "\n",
    "A processor gives the Processing Step information about the hardware and software that SageMaker should use to launch the Processing Job. To run the script, we need access to Scikit-Learn, so we can use the [SKLearnProcessor](https://sagemaker.readthedocs.io/en/stable/frameworks/sklearn/sagemaker.sklearn.html#scikit-learn-processor) processor that comes out-of-the-box with the SageMaker's Python SDK. The [Data Processing with Framework Processors](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job-frameworks.html) page discusses other built-in processors you can use. The [Docker Registry Paths and Example Code](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html) page contains information about the available framework versions for each region.\n",
    "\n",
    "The [ProcessingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.ProcessingStep) requires a list of inputs that we need on the preprocessing script. In this case, the input is the dataset we stored in S3. We also have a few outputs that we want SageMaker to capture when the Processing Job finishes. SageMaker will upload every one of these outputs to the location specified by the `preprocessor_destination` parameter except the baseline data, which we will upload to the location specified by the `baseline_destination` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "2187b65e-e504-40a5-ad05-82f54885805c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:Defaulting to only available Python version: py3\n"
     ]
    }
   ],
   "source": [
    "sklearn_processor = SKLearnProcessor(\n",
    "    base_job_name=\"penguins-preprocessing\",\n",
    "    framework_version=\"0.23-1\",\n",
    "    instance_type=\"ml.t3.medium\",\n",
    "    instance_count=1,\n",
    "    role=role,\n",
    ")\n",
    "\n",
    "preprocess_data_step = ProcessingStep(\n",
    "    name=\"preprocess-data\",\n",
    "    processor=sklearn_processor,\n",
    "    inputs=[\n",
    "        ProcessingInput(source=dataset_location, destination=\"/opt/ml/processing/input\"),  \n",
    "    ],\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name=\"train\", source=\"/opt/ml/processing/train\", destination=preprocessor_destination),\n",
    "        ProcessingOutput(output_name=\"validation\", source=\"/opt/ml/processing/validation\", destination=preprocessor_destination),\n",
    "        ProcessingOutput(output_name=\"test\", source=\"/opt/ml/processing/test\", destination=preprocessor_destination),\n",
    "        ProcessingOutput(output_name=\"pipeline\", source=\"/opt/ml/processing/pipeline\", destination=preprocessor_destination),\n",
    "        ProcessingOutput(output_name=\"classes\", source=\"/opt/ml/processing/classes\", destination=preprocessor_destination),\n",
    "        ProcessingOutput(output_name=\"train-baseline\", source=\"/opt/ml/processing/train-baseline\", destination=train_dataset_baseline_destination),\n",
    "        ProcessingOutput(output_name=\"test-baseline\", source=\"/opt/ml/processing/test-baseline\", destination=test_dataset_baseline_destination),\n",
    "    ],\n",
    "    code=f\"{PENGUINS_FOLDER}/preprocessor.py\",\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1465690a-67b5-4d68-a026-dd874ee569e8",
   "metadata": {},
   "source": [
    "## Step 6 - Running the Pipeline\n",
    "\n",
    "Let's define and run the SageMaker Pipeline. Check [Pipeline Structure and Execution](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-pipeline.html) for more information about how to define a pipeline and [Run a Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/run-pipeline.html) for information about how to run it.\n",
    "\n",
    "The pipeline uses the parameters we defined before and the Preprocess Step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "564fe6fb-00aa-480f-a58f-3144b2e33a9d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session1_pipeline = Pipeline(\n",
    "    name=\"penguins-session1-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location, \n",
    "        preprocessor_destination,\n",
    "        train_dataset_baseline_destination,\n",
    "        test_dataset_baseline_destination\n",
    "    ],\n",
    "    steps=[\n",
    "        preprocess_data_step, \n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e16d0ac8-1439-4329-9a1c-c43eafacd0da",
   "metadata": {},
   "source": [
    "Submit the pipeline definition to the SageMaker Pipelines service to create a pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "id": "f4410f2f-a17b-4272-abd2-e092e134174c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n"
     ]
    }
   ],
   "source": [
    "session1_pipeline.upsert(role_arn=role)\n",
    "execution = session1_pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63c190c5-52b5-4ccc-8d42-847a694b8e66",
   "metadata": {},
   "source": [
    "# Session 2 - Model Training and Tuning\n",
    "\n",
    "This session extends the [SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) we built in the previous session with a step to train a model. We'll explore the [Training](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-training) and the [Tuning](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-tuning) steps. \n",
    "\n",
    "Here is what the Pipeline will look like at the end of this session:\n",
    "\n",
    "<img src='penguins/images/session2-pipeline.png' alt='Session 2 Pipeline' width=\"600\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4756a3d8-d47b-4a88-a7f2-65ed7d4c1175",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.tuner import HyperparameterTuner\n",
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker.workflow.steps import TuningStep\n",
    "from sagemaker.parameter import IntegerParameter\n",
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker.tensorflow import TensorFlow\n",
    "from sagemaker.workflow.steps import TrainingStep\n",
    "from sagemaker.workflow.pipeline_context import PipelineSession"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8608092-7aab-4fd2-aa99-47c2db27bdb7",
   "metadata": {},
   "source": [
    "## Step 1 - Training the Model\n",
    "\n",
    "This script is responsible for training a simple neural network on the train data, validating the model, and saving it so we can later use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d92b121d-dcb9-43e8-9ee3-3ececb583e7e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguins/train.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {PENGUINS_FOLDER}/train.py\n",
    "\n",
    "import os\n",
    "import argparse\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "\n",
    "from pathlib import Path\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.optimizers import SGD\n",
    "\n",
    "\n",
    "def train(base_directory, train_path, validation_path, epochs=50, batch_size=32):\n",
    "    X_train = pd.read_csv(Path(train_path) / \"train.csv\")\n",
    "    y_train = X_train[X_train.columns[-1]]\n",
    "    X_train.drop(X_train.columns[-1], axis=1, inplace=True)\n",
    "    \n",
    "    X_validation = pd.read_csv(Path(validation_path) / \"validation.csv\")\n",
    "    y_validation = X_validation[X_validation.columns[-1]]\n",
    "    X_validation.drop(X_validation.columns[-1], axis=1, inplace=True)\n",
    "    \n",
    "    model = Sequential([\n",
    "        Dense(10, input_shape=(X_train.shape[1],), activation=\"relu\"),\n",
    "        Dense(8, activation=\"relu\"),\n",
    "        Dense(3, activation=\"softmax\"),\n",
    "    ])\n",
    "\n",
    "    model.compile(\n",
    "        optimizer=SGD(learning_rate=0.01),\n",
    "        loss=\"sparse_categorical_crossentropy\",\n",
    "        metrics=[\"accuracy\"]\n",
    "    )\n",
    "\n",
    "    model.fit(\n",
    "        X_train, \n",
    "        y_train, \n",
    "        validation_data=(X_validation, y_validation),\n",
    "        epochs=epochs, \n",
    "        batch_size=batch_size,\n",
    "        verbose=2,\n",
    "    )\n",
    "\n",
    "    predictions = np.argmax(model.predict(X_validation), axis=-1)\n",
    "    print(f\"Validation accuracy: {accuracy_score(y_validation, predictions)}\")\n",
    "    \n",
    "    model_filepath = Path(base_directory) / \"model\" / \"001\"\n",
    "    model.save(model_filepath)\n",
    "    \n",
    "if __name__ == \"__main__\":\n",
    "    # Any hyperparameters provided by the training job are passed to the entry point\n",
    "    # as script arguments. SageMaker will also provide a list of special parameters\n",
    "    # that you can capture here. Here is the full list: \n",
    "    # https://github.com/aws/sagemaker-training-toolkit/blob/master/src/sagemaker_training/params.py\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\"--base_directory\", type=str, default=\"/opt/ml/\")\n",
    "    parser.add_argument(\"--train_path\", type=str, default=os.environ.get(\"SM_CHANNEL_TRAIN\", None))\n",
    "    parser.add_argument(\"--validation_path\", type=str, default=os.environ.get(\"SM_CHANNEL_VALIDATION\", None))\n",
    "    parser.add_argument(\"--epochs\", type=int, default=50)\n",
    "    parser.add_argument(\"--batch_size\", type=int, default=32)\n",
    "    args, _ = parser.parse_known_args()\n",
    "    \n",
    "    train(\n",
    "        base_directory=args.base_directory,\n",
    "        train_path=args.train_path,\n",
    "        validation_path=args.validation_path,\n",
    "        epochs=args.epochs,\n",
    "        batch_size=args.batch_size\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50f0a4fa-ce70-4882-b9f5-8253df03d890",
   "metadata": {},
   "source": [
    "## Step 2 - Testing the Training Script\n",
    "\n",
    "Let's test the script we just created by running it locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "14ea27ce-c453-4cb0-b309-dbecd732957e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CLASSES ['Adelie' 'Chinstrap' 'Gentoo']\n",
      "Epoch 1/10\n",
      "Extension horovod.torch has not been built: /usr/local/lib/python3.8/site-packages/horovod/torch/mpi_lib/_mpi_lib.cpython-38-x86_64-linux-gnu.so not found\n",
      "If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error.\n",
      "Warning! MPI libs are missing, but python applications are still avaiable.\n",
      "[2023-06-28 06:55:48.927 tensorflow-2-6-cpu-py-ml-t3-medium-9169b2e75617c45c79c40579f6a8:76 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\n",
      "[2023-06-28 06:55:48.985 tensorflow-2-6-cpu-py-ml-t3-medium-9169b2e75617c45c79c40579f6a8:76 INFO profiler_config_parser.py:111] Unable to find config at /opt/ml/input/config/profilerconfig.json. Profiler is disabled.\n",
      "8/8 - 1s - loss: 1.1367 - accuracy: 0.4603 - val_loss: 1.0562 - val_accuracy: 0.4706\n",
      "Epoch 2/10\n",
      "8/8 - 0s - loss: 1.0693 - accuracy: 0.4937 - val_loss: 1.0011 - val_accuracy: 0.5098\n",
      "Epoch 3/10\n",
      "8/8 - 0s - loss: 1.0105 - accuracy: 0.5272 - val_loss: 0.9516 - val_accuracy: 0.5686\n",
      "Epoch 4/10\n",
      "8/8 - 0s - loss: 0.9586 - accuracy: 0.5900 - val_loss: 0.9088 - val_accuracy: 0.5882\n",
      "Epoch 5/10\n",
      "8/8 - 0s - loss: 0.9142 - accuracy: 0.6151 - val_loss: 0.8719 - val_accuracy: 0.6078\n",
      "Epoch 6/10\n",
      "8/8 - 0s - loss: 0.8773 - accuracy: 0.6653 - val_loss: 0.8396 - val_accuracy: 0.6471\n",
      "Epoch 7/10\n",
      "8/8 - 0s - loss: 0.8435 - accuracy: 0.6904 - val_loss: 0.8089 - val_accuracy: 0.6863\n",
      "Epoch 8/10\n",
      "8/8 - 0s - loss: 0.8124 - accuracy: 0.7113 - val_loss: 0.7782 - val_accuracy: 0.6863\n",
      "Epoch 9/10\n",
      "8/8 - 0s - loss: 0.7827 - accuracy: 0.7238 - val_loss: 0.7504 - val_accuracy: 0.7059\n",
      "Epoch 10/10\n",
      "8/8 - 0s - loss: 0.7554 - accuracy: 0.7490 - val_loss: 0.7242 - val_accuracy: 0.7451\n",
      "Validation accuracy: 0.7450980392156863\n",
      "INFO:tensorflow:Assets written to: /tmp/tmp7sl0c48r/model/001/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /tmp/tmp7sl0c48r/model/001/assets\n"
     ]
    }
   ],
   "source": [
    "from penguins.preprocessor import preprocess\n",
    "from penguins.train import train\n",
    "\n",
    "\n",
    "with tempfile.TemporaryDirectory() as directory:\n",
    "    # First, we preprocess the data and create the \n",
    "    # dataset splits.\n",
    "    preprocess(\n",
    "        base_directory=directory, \n",
    "        data_filepath=LOCAL_FILEPATH\n",
    "    )\n",
    "\n",
    "    # Then, we train a model using the train and \n",
    "    # validation splits.\n",
    "    train(\n",
    "        base_directory=directory, \n",
    "        train_path=Path(directory) / \"train\", \n",
    "        validation_path=Path(directory) / \"validation\",\n",
    "        epochs=10\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27cff4c1-6510-4d99-8ae1-cb14927b87c7",
   "metadata": {},
   "source": [
    "## Step 3 - Setting up a Training Step\n",
    "\n",
    "We can now create a [Training Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-training) that we can add to the pipeline. This Training Step will create a SageMaker Training Job in the background, run the training script, and upload the output to S3. Check the [TrainingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TrainingStep) SageMaker's SDK documentation for more information. \n",
    "\n",
    "SageMaker uses the concept of an [Estimator](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html) to handle end-to-end training and deployment tasks. For this example, we will use the built-in [TensorFlow Estimator](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-estimator) to run the training script we wrote before. The [Docker Registry Paths and Example Code](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html) page contains information about the available framework versions for each region. Here, you can also check the available SageMaker [Deep Learning Container images](https://github.com/aws/deep-learning-containers/blob/master/available_images.md).\n",
    "\n",
    "Notice the list of hyperparameters defined below. SageMaker will pass these hyperparameters as arguments to the entry point of the training script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "90fe82ae-6a2c-4461-bc83-bb52d8871e3b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "estimator = TensorFlow(\n",
    "    entry_point=f\"{PENGUINS_FOLDER}/train.py\",\n",
    "    \n",
    "    hyperparameters={\n",
    "        \"epochs\": 50,\n",
    "        \"batch_size\": 32\n",
    "    },\n",
    "    \n",
    "    framework_version=\"2.6\",\n",
    "    instance_type=\"ml.m5.large\",\n",
    "    py_version=\"py38\",\n",
    "    instance_count=1,\n",
    "    script_mode=True,\n",
    "    \n",
    "    # The default profiler rule includes a timestamp which will change each time\n",
    "    # the pipeline is upserted, causing cache misses. Since we don't need\n",
    "    # profiling, we can disable it to take advantage of caching.\n",
    "    disable_profiler=True,\n",
    "\n",
    "    role=role,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "545d2b43-3bb5-4fe9-b3e4-cb8eb55c8a21",
   "metadata": {},
   "source": [
    "We can now create the [TrainingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TrainingStep) using the estimator we defined before.\n",
    "\n",
    "This step will receive the train and validation split from the preprocessing step as inputs. Notice how we reference both splits using preprocess step. This creates a dependency between the Training and Processing Steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "99e4850c-83d6-4f4e-a813-d5a3f4bb7486",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_model_step = TrainingStep(\n",
    "    name=\"train-model\",\n",
    "    estimator=estimator,\n",
    "    inputs={\n",
    "        \"train\": TrainingInput(\n",
    "            s3_data=preprocess_data_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"train\"\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type=\"text/csv\"\n",
    "        ),\n",
    "        \"validation\": TrainingInput(\n",
    "            s3_data=preprocess_data_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"validation\"\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type=\"text/csv\"\n",
    "        )\n",
    "    },\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5814e258-c633-4e9a-85c5-6ed0f168b503",
   "metadata": {},
   "source": [
    "## Step 4 - Setting up a Tuning Step\n",
    "\n",
    "Let's now create a [Tuning Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-tuning). This Tuning Step will create a SageMaker Hyperparameter Tuning Job in the background and use the training script to train different model variants and choose the best one. Check the [TuningStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TuningStep) SageMaker's SDK documentation for more information.\n",
    "\n",
    "The Tuning Step requires a [HyperparameterTuner](https://sagemaker.readthedocs.io/en/stable/api/training/tuner.html) reference to configure the Hyperparameter Tuning Job.\n",
    "\n",
    "Here is the configuration that we'll use to find the best model:\n",
    "\n",
    "1. `objective_metric_name`: This is the name of the metric the tuner will use to determine the best model.\n",
    "2. `objective_type`: This is the objective of the tuner. Should it \"Minimize\" the metric or \"Maximize\" it? In this example, since we are using the validation accuracy of the model, we want the objective to be \"Maximize.\" If we were using the loss of the model, we would set the objective to \"Minimize.\"\n",
    "3. `metric_definitions`: Defines how the tuner will determine the metric's value by looking at the output logs of the training process.\n",
    "\n",
    "The tuner expects the list of the hyperparameters you want to explore. You can use subclasses of the [Parameter](https://sagemaker.readthedocs.io/en/stable/api/training/parameter.html#sagemaker.parameter.ParameterRange) class to specify different types of hyperparameters. This example explores different values for the `epochs` hyperparameter.\n",
    "\n",
    "Finally, you can control the number of jobs and how many of them will run in parallel using the following two arguments:\n",
    "\n",
    "* `max_jobs`: Defines the maximum total number of training jobs to start for the hyperparameter tuning job.\n",
    "* `max_parallel_jobs`: Defines the maximum number of parallel training jobs to start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "038ff2e5-ed28-445b-bc03-4e996ec2286f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "objective_metric_name = \"val_accuracy\"\n",
    "objective_type = \"Maximize\"\n",
    "metric_definitions = [{\"Name\": objective_metric_name, \"Regex\": \"val_accuracy: ([0-9\\\\.]+)\"}]\n",
    "    \n",
    "hyperparameter_ranges = {\n",
    "    \"epochs\": IntegerParameter(10, 50),\n",
    "}\n",
    "\n",
    "tuner = HyperparameterTuner(\n",
    "    estimator,\n",
    "    objective_metric_name,\n",
    "    hyperparameter_ranges,\n",
    "    metric_definitions,\n",
    "    objective_type=objective_type,\n",
    "    max_jobs=3,\n",
    "    max_parallel_jobs=3,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59de0743-6c9e-4895-b060-f58a6d60c50d",
   "metadata": {},
   "source": [
    "We can now create the [TuningStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TuningStep). \n",
    "\n",
    "This step will use the tuner we configured before and will receive the train and validation split from the preprocessing step as inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "cf5dd9a7-8643-4fbb-8eb4-40f39011e27b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tune_model_step = TuningStep(\n",
    "    name = \"tune-model\",\n",
    "    tuner=tuner,\n",
    "    inputs={\n",
    "        \"train\": TrainingInput(\n",
    "            s3_data=preprocess_data_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"train\"\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type=\"text/csv\"\n",
    "        ),\n",
    "        \"validation\": TrainingInput(\n",
    "            s3_data=preprocess_data_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"validation\"\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type=\"text/csv\"\n",
    "        )\n",
    "    },\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf9c9362-ef17-4d68-b8c8-cefe21326ba2",
   "metadata": {},
   "source": [
    "## Step 5 - Switching Between Training and Tuning\n",
    "\n",
    "We could use a [Training Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-training) or use a [Tuning Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-tuning) to create the model.\n",
    "\n",
    "In this notebook, we will alternate between both methods and use the `USE_TUNING_STEP` flag to indicate which approach we want to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c3df0f7a-e43a-4a7b-9809-0a639fa178d9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "USE_TUNING_STEP = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4babe38c-1682-42d2-8442-101d17aa89b5",
   "metadata": {},
   "source": [
    "## Step 6 - Running the Pipeline\n",
    "\n",
    "We can now define and run the SageMaker Pipeline using the Training or Tuning Step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "9799ab39-fcae-41f4-a68b-85ab71b3ba9a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session2_pipeline = Pipeline(\n",
    "    name=\"penguins-session2-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location, \n",
    "        preprocessor_destination,\n",
    "        train_dataset_baseline_destination,\n",
    "        test_dataset_baseline_destination,\n",
    "    ],\n",
    "    steps=[\n",
    "        preprocess_data_step, \n",
    "        tune_model_step if USE_TUNING_STEP else train_model_step\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0e0708c-3692-4d09-a269-bbd14f7b0226",
   "metadata": {},
   "source": [
    "Submit the pipeline definition to the SageMaker Pipelines service to create a pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "id": "77c6d523-6810-4bc7-8fc4-64ae978cf245",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n",
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n"
     ]
    }
   ],
   "source": [
    "session2_pipeline.upsert(role_arn=role)\n",
    "execution = session2_pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21d40fe8-ba74-4c12-9555-d8ea33d1c8b4",
   "metadata": {},
   "source": [
    "# Session 3 - Model Registration\n",
    "\n",
    "This session extends the [SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) with a step to evaluate and register the model if it reaches a predefined accuracy threshold. We'll use a [Processing Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing) with a [ScriptProcessor](https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.processing.ScriptProcessor) running TensorFlow to execute an evaluation script. We'll use a [Condition Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-condition) to determine whether the model's accuracy is above a threshold and a [Model Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-model) to register the model. After we register the model, we'll deploy it manually. To learn more about the Model Registry, check [Register and Deploy Models with Model Registry](https://docs.aws.amazon.com/sagemaker/latest/dg/model-registry.html).\n",
    "\n",
    "Here is what the Pipeline will look like at the end of this session:\n",
    "\n",
    "<img src='penguins/images/session3-pipeline.png' alt='Session 3 Pipeline' width=\"600\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "34fc262f-a1cf-4f94-9c60-e7c8e83cfdfb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "import tarfile\n",
    "\n",
    "from sagemaker import ModelPackage\n",
    "from sagemaker.tensorflow import TensorFlowProcessor\n",
    "from sagemaker.model import Model\n",
    "from sagemaker.tensorflow.model import TensorFlowModel\n",
    "from sagemaker.model_metrics import MetricsSource, ModelMetrics \n",
    "from sagemaker.predictor import Predictor\n",
    "from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo\n",
    "from sagemaker.workflow.condition_step import ConditionStep\n",
    "from sagemaker.workflow.fail_step import FailStep\n",
    "from sagemaker.workflow.functions import JsonGet\n",
    "from sagemaker.workflow.functions import Join\n",
    "from sagemaker.workflow.properties import PropertyFile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eaa9691-f49f-48af-b272-3d4d17563b01",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Step 1 - Evaluating the Model\n",
    "\n",
    "This script is responsible for loading the model we created and evaluating it on the test set. Before finishing, this script will generate an evaluation report of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3ee3ab26-afa5-4ceb-9f7a-005d5fdea646",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguins/evaluation.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {PENGUINS_FOLDER}/evaluation.py\n",
    "\n",
    "import os\n",
    "import json\n",
    "import tarfile\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from pathlib import Path\n",
    "from tensorflow import keras\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "\n",
    "MODEL_PATH = \"/opt/ml/processing/model/\"\n",
    "TEST_PATH = \"/opt/ml/processing/test/\"\n",
    "OUTPUT_PATH = \"/opt/ml/processing/evaluation/\"\n",
    "\n",
    "\n",
    "def evaluate(model_path, test_path, output_path):\n",
    "    # The first step is to extract the model package so we can load \n",
    "    # it in memory.\n",
    "    with tarfile.open(Path(model_path) / \"model.tar.gz\") as tar:\n",
    "        tar.extractall(path=Path(model_path))\n",
    "        \n",
    "    model = keras.models.load_model(Path(model_path) / \"001\")\n",
    "    \n",
    "    X_test = pd.read_csv(Path(test_path) / \"test.csv\")\n",
    "    y_test = X_test[X_test.columns[-1]]\n",
    "    X_test.drop(X_test.columns[-1], axis=1, inplace=True)\n",
    "    \n",
    "    predictions = np.argmax(model.predict(X_test), axis=-1)\n",
    "    accuracy = accuracy_score(y_test, predictions)\n",
    "    print(f\"Test accuracy: {accuracy}\")\n",
    "\n",
    "    # Let's create an evaluation report using the model accuracy.\n",
    "    evaluation_report = {\n",
    "        \"metrics\": {\n",
    "            \"accuracy\": {\n",
    "                \"value\": accuracy\n",
    "            },\n",
    "        },\n",
    "    }\n",
    "    \n",
    "    Path(output_path).mkdir(parents=True, exist_ok=True)\n",
    "    with open(Path(output_path) / \"evaluation.json\", \"w\") as f:\n",
    "        f.write(json.dumps(evaluation_report))\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    evaluate(\n",
    "        model_path=MODEL_PATH, \n",
    "        test_path=TEST_PATH,\n",
    "        output_path=OUTPUT_PATH\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dcc79a0-adfd-4ce9-8580-5cd228c3c2d9",
   "metadata": {},
   "source": [
    "## Step 2 - Testing the Evaluation Script\n",
    "\n",
    "Let's test the script we just created by running it locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "9a2540d8-278a-4953-bc54-0469d154427d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CLASSES ['Adelie' 'Chinstrap' 'Gentoo']\n",
      "Epoch 1/10\n",
      "8/8 - 0s - loss: 1.0382 - accuracy: 0.4519 - val_loss: 1.0016 - val_accuracy: 0.5294\n",
      "Epoch 2/10\n",
      "8/8 - 0s - loss: 0.9754 - accuracy: 0.5523 - val_loss: 0.9467 - val_accuracy: 0.6471\n",
      "Epoch 3/10\n",
      "8/8 - 0s - loss: 0.9213 - accuracy: 0.6444 - val_loss: 0.9014 - val_accuracy: 0.6667\n",
      "Epoch 4/10\n",
      "8/8 - 0s - loss: 0.8774 - accuracy: 0.6862 - val_loss: 0.8605 - val_accuracy: 0.7059\n",
      "Epoch 5/10\n",
      "8/8 - 0s - loss: 0.8396 - accuracy: 0.7071 - val_loss: 0.8232 - val_accuracy: 0.7255\n",
      "Epoch 6/10\n",
      "8/8 - 0s - loss: 0.8045 - accuracy: 0.7490 - val_loss: 0.7895 - val_accuracy: 0.7843\n",
      "Epoch 7/10\n",
      "8/8 - 0s - loss: 0.7734 - accuracy: 0.7657 - val_loss: 0.7591 - val_accuracy: 0.7843\n",
      "Epoch 8/10\n",
      "8/8 - 0s - loss: 0.7446 - accuracy: 0.7699 - val_loss: 0.7317 - val_accuracy: 0.7843\n",
      "Epoch 9/10\n",
      "8/8 - 0s - loss: 0.7182 - accuracy: 0.7741 - val_loss: 0.7061 - val_accuracy: 0.8039\n",
      "Epoch 10/10\n",
      "8/8 - 0s - loss: 0.6941 - accuracy: 0.7699 - val_loss: 0.6831 - val_accuracy: 0.8039\n",
      "Validation accuracy: 0.803921568627451\n",
      "INFO:tensorflow:Assets written to: /tmp/tmpukn0f2j9/model/001/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /tmp/tmpukn0f2j9/model/001/assets\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.7254901960784313\n"
     ]
    }
   ],
   "source": [
    "from penguins.preprocessor import preprocess\n",
    "from penguins.train import train\n",
    "from penguins.evaluation import evaluate\n",
    "\n",
    "\n",
    "with tempfile.TemporaryDirectory() as directory:\n",
    "    preprocess(\n",
    "        base_directory=directory, \n",
    "        data_filepath=LOCAL_FILEPATH\n",
    "    )\n",
    "\n",
    "    train(\n",
    "        base_directory=directory, \n",
    "        train_path=Path(directory) / \"train\", \n",
    "        validation_path=Path(directory) / \"validation\",\n",
    "        epochs=10\n",
    "    )\n",
    "    \n",
    "    # After training a model, we need to prepare a package just like\n",
    "    # SageMaker would. This package is what the evaluation script is\n",
    "    # expecting as an input.\n",
    "    with tarfile.open(Path(directory) / \"model.tar.gz\", \"w:gz\") as tar:\n",
    "        tar.add(Path(directory) / \"model\" / \"001\", arcname=\"001\")\n",
    "        \n",
    "    \n",
    "    # We can now call the evaluation script.\n",
    "    evaluate(\n",
    "        model_path=directory, \n",
    "        test_path=Path(directory) / \"test\",\n",
    "        output_path=Path(directory) / \"evaluation\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bec1109a-6c26-4464-8338-94960729d212",
   "metadata": {},
   "source": [
    "## Step 3 - Setting up a Processing Step\n",
    "\n",
    "To run the evaluation script, we can use a [Processing Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing). In this case, we'll use the [TensorFlowProcessor](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job-frameworks-tensorflow.html) because we need access to TensorFlow. \n",
    "\n",
    "The inputs of this Processing Step will be the model we created and the test set. The output will be the evaluation report file.\n",
    "\n",
    "We can use the `USE_TUNING_STEP` flag to determine whether we created the model using a Training Step or a Tuning Step. In case we are using the Tuning Step, we can use the [TuningStep.get_top_model_s3_uri()](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TuningStep.get_top_model_s3_uri) function to get the model artifacts from the top performing training job of the Hyperparameter Tuning Job.\n",
    "\n",
    "The [ProcessingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.ProcessingStep) lets us specify a list of [PropertyFile](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.properties.PropertyFile) instances from the output of the job. We can use this to map the evaluation report generated in the evaluation script. Check [How to Build and Manage Property Files](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-propertyfile.html) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "48139a07-5c8e-4bc6-b666-bf9531f7f520",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    }
   ],
   "source": [
    "tensorflow_processor = TensorFlowProcessor(\n",
    "    base_job_name=\"penguins-evaluation-processor\",\n",
    "    framework_version=\"2.6\",\n",
    "    py_version=\"py38\",\n",
    "    instance_type=\"ml.m5.large\",\n",
    "    instance_count=1,\n",
    "    role=role\n",
    ")\n",
    "\n",
    "# This is a workaround to a problem with the SageMaker SDK: \n",
    "# By default, the TensorFlowProcessor runs the script using\n",
    "# /bin/bash as its entrypoint. We want to ensure we run it \n",
    "# using python3.\n",
    "tensorflow_processor.framework_entrypoint_command = [\"python3\"]\n",
    "\n",
    "\n",
    "# We want to map the evaluation report that we generate inside\n",
    "# the evaluation script so we can later reference it.\n",
    "evaluation_report = PropertyFile(\n",
    "    name=\"evaluation-report\",\n",
    "    output_name=\"evaluation\",\n",
    "    path=\"evaluation.json\"\n",
    ")\n",
    "\n",
    "\n",
    "# Notice how this step uses the model generated by the tuning or training\n",
    "# step, and the test set generated by the preprocessing step.\n",
    "evaluate_model_step = ProcessingStep(\n",
    "    name=\"evaluate-model\",\n",
    "    processor=tensorflow_processor,\n",
    "    inputs=[\n",
    "        ProcessingInput(\n",
    "            source=preprocess_data_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"test\"\n",
    "            ].S3Output.S3Uri,\n",
    "            destination=\"/opt/ml/processing/test\"\n",
    "        ),\n",
    "        ProcessingInput(\n",
    "            source=(\n",
    "                tune_model_step.get_top_model_s3_uri(top_k=0, s3_bucket=sagemaker_session.default_bucket()) \n",
    "                if USE_TUNING_STEP \n",
    "                else train_model_step.properties.ModelArtifacts.S3ModelArtifacts\n",
    "            ),\n",
    "            destination=\"/opt/ml/processing/model\",\n",
    "        )\n",
    "    ],\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name=\"evaluation\", source=\"/opt/ml/processing/evaluation\", destination=f\"{S3_FILEPATH}/evaluation\"),\n",
    "    ],\n",
    "    code=f\"{PENGUINS_FOLDER}/evaluation.py\",\n",
    "    property_files=[evaluation_report],\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "789a48e6-507d-42fb-8abc-ebde997e3d43",
   "metadata": {},
   "source": [
    "## Step 4 - Configuring the Model Metrics\n",
    "\n",
    "When we register a model, we can specify a set of [ModelMetrics](https://sagemaker.readthedocs.io/en/stable/api/inference/model_monitor.html#sagemaker.model_metrics.ModelMetrics). We can use the evaluation report we generated during the Evaluation step to populate these statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "4982146f-0c0f-4938-b5d0-06db45a58531",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_metrics = ModelMetrics(\n",
    "    model_statistics=MetricsSource(\n",
    "        s3_uri=Join(on=\"/\", values=[\n",
    "            evaluate_model_step.arguments['ProcessingOutputConfig']['Outputs'][0]['S3Output']['S3Uri'],\n",
    "            \"evaluation.json\"]\n",
    "        ),\n",
    "        content_type=\"application/json\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "441fac3a-5ecc-441f-84c3-717c4c7ba290",
   "metadata": {},
   "source": [
    "## Step 5 - Registering the Model\n",
    "\n",
    "We can now create a [Model Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-model) to register the model. Check the [ModelStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.model_step.ModelStep) SageMaker's SDK documentation for more information. We want to create a new version of the model and register it in the Model Registry. Check [Register a Model Version](https://docs.aws.amazon.com/sagemaker/latest/dg/model-registry-version.html) for more information about model registration.\n",
    "\n",
    "The model we trained uses TensorFlow, so we can use the built-in [TensorFlowModel](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-serving-model) class to create an instance of the model.\n",
    "\n",
    "Notice that we use an instance of the [PipelineSession](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.pipeline_context.PipelineSession) class to create the model. This special session does not register the model immediately when you call `model.register()`, instead, it captures the arguments required to register a model, and delegate it to the `ModelStep` to register the model later during pipeline execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "52a48cef-fb78-412b-a5c6-977eafe98e27",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.tensorflow.model:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n",
      "/usr/local/lib/python3.8/site-packages/sagemaker/workflow/pipeline_context.py:286: UserWarning: Running within a PipelineSession, there will be No Wait, No Logs, and No Job being started.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "model_package_group_name = \"penguins\"\n",
    "\n",
    "model = TensorFlowModel(\n",
    "    model_data=(\n",
    "        tune_model_step.get_top_model_s3_uri(top_k=0, s3_bucket=sagemaker_session.default_bucket())\n",
    "        if USE_TUNING_STEP\n",
    "        else train_model_step.properties.ModelArtifacts.S3ModelArtifacts\n",
    "    ),\n",
    "    framework_version=\"2.6\",\n",
    "    sagemaker_session=PipelineSession(),\n",
    "    role=role,\n",
    ")\n",
    "\n",
    "register_model_step = ModelStep(\n",
    "    name=\"register-model\",\n",
    "    step_args=model.register(\n",
    "        model_package_group_name=model_package_group_name,\n",
    "        model_metrics=model_metrics,\n",
    "        approval_status=\"Approved\",\n",
    "        \n",
    "        content_types=[\"text/csv\"],\n",
    "        response_types=[\"text/csv\"],\n",
    "        inference_instances=[\"ml.m5.large\"],\n",
    "        transform_instances=[\"ml.m5.large\"],\n",
    "        domain=\"MACHINE_LEARNING\",\n",
    "        task=\"CLASSIFICATION\",\n",
    "        framework=\"TENSORFLOW\",\n",
    "        framework_version=\"2.6\",\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c110f7-fe72-4db8-9d06-cfb9a0f2bfbd",
   "metadata": {},
   "source": [
    "## Step 6 - Setting up a Condition Step\n",
    "\n",
    "We only want to register a new model if its accuracy exceeds a predefined threshold. We can use a [Condition Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-condition) together with the evaluation report we generated to accomplish this. Check the [ConditionStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#conditionstep) SageMaker's SDK documentation for more information.\n",
    "\n",
    "In this example, we will use a [ConditionGreaterThanOrEqualTo](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.conditions.ConditionGreaterThanOrEqualTo) condition to compare the model's accuracy with the threshold. Look at the [Conditions](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_building_pipeline.html#conditions) section in the documentation for more information about the types of supported conditions.\n",
    "\n",
    "If the model's accuracy is not greater than or equal our threshold, we will send the pipeline to a [Fail Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-fail) with the appropriate error message. Check the [FailStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.fail_step.FailStep) SageMaker's SDK documentation for more information.\n",
    "\n",
    "We are going to use a new [Pipeline Parameter](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-parameters.html) in our pipeline to specify the minimum accuracy that the model should reach for it to be registered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "36e2a2b1-6711-4266-95d8-d2aebd52e199",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "accuracy_threshold = ParameterFloat(\n",
    "    name=\"accuracy_threshold\", \n",
    "    default_value=0.70\n",
    ")\n",
    "\n",
    "condition_gte = ConditionGreaterThanOrEqualTo(\n",
    "    left=JsonGet(\n",
    "        step_name=evaluate_model_step.name,\n",
    "        property_file=evaluation_report,\n",
    "        json_path=\"metrics.accuracy.value\"\n",
    "    ),\n",
    "    right=accuracy_threshold\n",
    ")\n",
    "\n",
    "fail_step = FailStep(\n",
    "    name=\"fail\",\n",
    "    error_message=Join(\n",
    "        on=\" \", \n",
    "        values=[\n",
    "            \"Execution failed because the model's accuracy was lower than\", \n",
    "            accuracy_threshold\n",
    "        ]\n",
    "    ),\n",
    ")\n",
    "\n",
    "condition_step = ConditionStep(\n",
    "    name=\"check-model-accuracy\",\n",
    "    conditions=[condition_gte],\n",
    "    if_steps=[register_model_step],\n",
    "    else_steps=[fail_step], \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2309b8fa-f03e-4959-853f-dc2416f82bdd",
   "metadata": {},
   "source": [
    "## Step 7 - Running the Pipeline\n",
    "\n",
    "We can now add the registration of the model to the pipeline. Notice how we add the Condition Step, which will call the Model Step if the condition passes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "f70bcd33-b499-4e2b-953e-94d1ed96c10a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session3_pipeline = Pipeline(\n",
    "    name=\"penguins-session3-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location, \n",
    "        preprocessor_destination,\n",
    "        train_dataset_baseline_destination,\n",
    "        test_dataset_baseline_destination,\n",
    "        accuracy_threshold,\n",
    "    ],\n",
    "    steps=[\n",
    "        preprocess_data_step, \n",
    "        tune_model_step if USE_TUNING_STEP else train_model_step, \n",
    "        evaluate_model_step,\n",
    "        condition_step\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04e98fb1-fdff-4ba4-aea3-0d9f271fb24c",
   "metadata": {},
   "source": [
    "Submit the pipeline definition to the SageMaker Pipelines service to create a pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "4864ab08-7acc-4719-970e-fcce66c5fef5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker.workflow._utils:Popping out 'CertifyForMarketplace' from the pipeline definition since it will be overridden in pipeline execution time.\n",
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    }
   ],
   "source": [
    "session3_pipeline.upsert(role_arn=role)\n",
    "execution = session3_pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "565cf77e-7fc7-406e-a2e2-40c553f459f7",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Session 4 - Model Deployment\n",
    "\n",
    "This session extends the [SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) with a step to deploy the model to an endpoint. We'll use a [Lambda Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda) to create an endpoint and deploy the model. To control the endpoint's inputs and outputs, we'll modify the model's assets to include code that customizes the processing of a request. \n",
    "\n",
    "At the end of this session, our Pipeline will look like this:\n",
    "\n",
    "<img src='penguins/images/session4-pipeline.png' alt='Session 4 Pipeline' width=\"600\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "8d585140-7940-4543-9954-b74352e8ff3e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow.model import TensorFlowModel\n",
    "from sagemaker.tensorflow.model import TensorFlowPredictor\n",
    "from sagemaker.workflow.lambda_step import LambdaStep, LambdaOutput, LambdaOutputTypeEnum\n",
    "from sagemaker.workflow.parameters import ParameterBoolean\n",
    "from sagemaker.lambda_helper import Lambda\n",
    "from sagemaker.serializers import JSONSerializer\n",
    "from sagemaker.deserializers import JSONDeserializer\n",
    "from sagemaker.s3 import S3Downloader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eca780cf-d2f2-4de7-91c5-ce0dcc829ee9",
   "metadata": {},
   "source": [
    "## Step 1 - Deploy Latest Model From Registry\n",
    "\n",
    "Let's get the latest approved model from the Model Registry and deploy it to an endpoint.\n",
    "\n",
    "We can use `boto3` to query the list of approved models and get the latest one. Check the [boto3 SageMaker Client API](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html) for a list of everything you can do using the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "7804ee05-c2de-48ac-96b5-dbb2ebe9836f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ModelPackageGroupName': 'penguins',\n",
       " 'ModelPackageVersion': 10,\n",
       " 'ModelPackageArn': 'arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/10',\n",
       " 'CreationTime': datetime.datetime(2023, 6, 27, 10, 14, 23, 111000, tzinfo=tzlocal()),\n",
       " 'ModelPackageStatus': 'Completed',\n",
       " 'ModelApprovalStatus': 'Approved'}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = sagemaker_client.list_model_packages(\n",
    "    ModelPackageGroupName=model_package_group_name,\n",
    "    ModelApprovalStatus=\"Approved\",\n",
    "    SortBy=\"CreationTime\",\n",
    "    MaxResults=1,\n",
    ")\n",
    "\n",
    "package = response[\"ModelPackageSummaryList\"][0]\n",
    "package"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02b4ff58-235f-4c86-8134-4cfacc45e459",
   "metadata": {},
   "source": [
    "Let's define the name of the endpoint where we'll deploy the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "3149cb25-20cb-4561-a29a-cf04f655cf41",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "endpoint_name = \"penguins-endpoint\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "119c2a8f-b164-4fd3-8e3d-a084cd3219b7",
   "metadata": {},
   "source": [
    "Using the ARN of the model package from the Model Registry, we can deploy the model by creating a [ModelPackage](https://sagemaker.readthedocs.io/en/stable/api/inference/model.html#sagemaker.model.ModelPackage) instance and calling its `deploy()` function. The model information lives in the Model Registry, so we don't need to specify anything else."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "bdf25ea5-6c5b-4455-9198-9f0aad06ad51",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model with name: penguins-2023-06-28-06-56-15-748\n",
      "INFO:sagemaker:Creating endpoint-config with name penguins-endpoint\n",
      "INFO:sagemaker:Creating endpoint with name penguins-endpoint\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----!"
     ]
    }
   ],
   "source": [
    "model_package = ModelPackage(\n",
    "    model_package_arn=package[\"ModelPackageArn\"], \n",
    "    sagemaker_session=sagemaker_session,\n",
    "    role=role, \n",
    ")\n",
    "\n",
    "model_package.deploy(\n",
    "    endpoint_name=endpoint_name,\n",
    "    initial_instance_count=1, \n",
    "    instance_type=\"ml.m5.large\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76f29a26-4016-435c-8ec3-7639cb90399b",
   "metadata": {},
   "source": [
    "We can create a [Predictor](https://sagemaker.readthedocs.io/en/stable/api/inference/predictors.html#sagemaker.predictor.Predictor) using the endpoint name and test our model.\n",
    "\n",
    "The payload we need to provide the model is in CSV format. Notice how the model expects data that's already transformed. We can't provide the original data from our dataset because the model will not work with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c9fc46e-d807-434f-a980-93b054337b88",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "predictor = Predictor(endpoint_name=endpoint_name)\n",
    "\n",
    "payload = \"\"\"\n",
    "0.6569590202313976, -1.0813829646495108, 1.2097102831892812, 0.9226343641317372, 1.0, 0.0, 0.0\n",
    "-0.7751048801481084, 0.8822689351285553,  -1.2168066120762704, 0.9226343641317372, 0.0, 1.0, 0.0\n",
    "-0.837387834894918, 0.3386660813829646, -0.26237731892812, -1.92351941317372, 0.0, 0.0, 1.0\n",
    "\"\"\"\n",
    "\n",
    "response = predictor.predict(payload, initial_args={\"ContentType\": \"text/csv\"})\n",
    "response = json.loads(response.decode(\"utf-8\"))\n",
    "\n",
    "print(json.dumps(response, indent=2))\n",
    "print(f\"\\nSpecies: {np.argmax(response['predictions'], axis=1)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75271c1c-b661-4569-aeb6-93f2c6621bdb",
   "metadata": {},
   "source": [
    "Let's now delete the endpoint.predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "283ce0a7-ca8f-46c0-bcdf-2347b0d2f1fc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Deleting endpoint configuration with name: penguins-endpoint\n",
      "INFO:sagemaker:Deleting endpoint with name: penguins-endpoint\n"
     ]
    }
   ],
   "source": [
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07fc27c6-31d4-454d-ae5b-1aeba25f0ac0",
   "metadata": {},
   "source": [
    "## Step 2 - Preparing the Inference Code\n",
    "\n",
    "Deploying the model we trained directly to an endpoint doesn't lets us control the data that goes in and comes out of the endpoint. Fortunately, SageMaker allows us to include an `inference.py` file with the model assets from where we can control how the endpoint works. You can see more information about how this works by checking the [SageMaker TensorFlow Serving Container](https://github.com/aws/sagemaker-tensorflow-serving-container) documentation.\n",
    "\n",
    "We want our endpoint to handle unprocessed data in JSON format and return the penguin's species. Here is an example of the payload we want the endpoint to support:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"island\": \"Biscoe\",\n",
    "    \"culmen_length_mm\": 48.6,\n",
    "    \"culmen_depth_mm\": 16.0,\n",
    "    \"flipper_length_mm\": 230.0,\n",
    "    \"body_mass_g\": 5800.0,\n",
    "}\n",
    "```\n",
    "\n",
    "And here is an example of the output we'd like to get from the endpoint:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"species\": \"Adelie\", \n",
    "    \"prediction\": 0, \n",
    "    \"confidence\": 0.402672\n",
    "}\n",
    "```\n",
    "\n",
    "Let's start by setting up a local folder where we will create the `inference.py` script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "ac9cec59-c812-4ca9-9f71-d6725de03c3a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "CODE_FOLDER = PENGUINS_FOLDER / \"code\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a34c1c1-66af-4cf2-b103-8612bd60ce0e",
   "metadata": {},
   "source": [
    "We will include the inference code as part of the model assets to control the inference process on the SageMaker endpoint. SageMaker will automatically call the `handler()` function for every request to the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "57e7ad9e-598b-4e28-9a20-a93ad2bb5f7d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguins/code/inference.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $CODE_FOLDER/inference.py\n",
    "\n",
    "import os\n",
    "import json\n",
    "import boto3\n",
    "import requests\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from pickle import load\n",
    "from pathlib import Path\n",
    "\n",
    "\n",
    "PIPELINE_FILE = Path(\"/tmp\") / \"pipeline.pkl\"\n",
    "CLASSES_FILE = Path(\"/tmp\") / \"classes.csv\"\n",
    "\n",
    "s3 = boto3.resource(\"s3\")\n",
    "\n",
    "\n",
    "def handler(data, context):\n",
    "    \"\"\"\n",
    "    This is the entrypoint that will be called by SageMaker when the endpoint\n",
    "    receives a request. You can see more information at \n",
    "    https://github.com/aws/sagemaker-tensorflow-serving-container.\n",
    "    \"\"\"\n",
    "    print(\"Handling endpoint request\")\n",
    "    \n",
    "    data = _process_input(data, context)\n",
    "    output = _predict(data, context)\n",
    "    return _process_output(output, context)\n",
    "\n",
    "\n",
    "def _process_input(data, context):\n",
    "    print(\"Processing input data...\")\n",
    "    \n",
    "    if context is None:\n",
    "        # The context will be None when we are testing the code\n",
    "        # directly from a notebook. In that case, we can use the\n",
    "        # data directly.\n",
    "        endpoint_input = data\n",
    "    elif context.request_content_type in (\"application/json\", \"application/octet-stream\"):\n",
    "        # When the endpoint is running, we will receive a context\n",
    "        # object. We need to parse the input and turn it into \n",
    "        # JSON in that case.\n",
    "        endpoint_input = json.loads(data.read().decode(\"utf-8\"))\n",
    "\n",
    "        if endpoint_input is None:\n",
    "            raise ValueError(\"There was an error parsing the input request.\")\n",
    "    else:\n",
    "        raise ValueError(f\"Unsupported content type: {context.request_content_type or 'unknown'}\")\n",
    "        \n",
    "    pipeline = _get_pipeline()\n",
    "\n",
    "    df = pd.json_normalize(endpoint_input)\n",
    "    result = pipeline.transform(df)\n",
    "    \n",
    "    return result[0].tolist()\n",
    "\n",
    "\n",
    "def _predict(instance, context):\n",
    "    print(\"Sending input data to model to make a prediction...\")\n",
    "    \n",
    "    model_input = json.dumps({\"instances\": [instance]})\n",
    "    \n",
    "    if context is None:\n",
    "        # The context will be None when we are testing the code\n",
    "        # directly from a notebook. In that case, we want to return\n",
    "        # a fake prediction back.\n",
    "        result = {\n",
    "            \"predictions\": [\n",
    "                [0.2, 0.5, 0.3]\n",
    "            ]\n",
    "        }\n",
    "    else:\n",
    "        # When the endpoint is running, we will receive a context\n",
    "        # object. In that case we need to send the instance to the\n",
    "        # model to get a prediction back.\n",
    "        response = requests.post(context.rest_uri, data=model_input)\n",
    "        \n",
    "        if response.status_code != 200:\n",
    "            raise ValueError(response.content.decode('utf-8'))\n",
    "            \n",
    "        result = json.loads(response.content)\n",
    "    \n",
    "    print(f\"Response: {result}\")\n",
    "    return result\n",
    "\n",
    "\n",
    "def _process_output(output, context):\n",
    "    print(\"Processing prediction received from the model...\")\n",
    "    \n",
    "    response_content_type = \"application/json\" if context is None else context.accept_header\n",
    "    \n",
    "    prediction = np.argmax(output[\"predictions\"][0])\n",
    "    confidence = output[\"predictions\"][0][prediction]\n",
    "    \n",
    "    print(f\"Prediction: {prediction}. Confidence: {confidence}\")\n",
    "    \n",
    "    result = json.dumps({\n",
    "        \"species\": _get_class(prediction),\n",
    "        \"prediction\": int(prediction),\n",
    "        \"confidence\": confidence\n",
    "    }), response_content_type\n",
    "    \n",
    "    return result\n",
    "\n",
    "\n",
    "def _get_pipeline():\n",
    "    \"\"\"\n",
    "    This function returns the Scikit-Learn pipeline we used to transform the\n",
    "    dataset.\n",
    "    \"\"\"\n",
    "    \n",
    "    _download(PIPELINE_FILE)\n",
    "    return load(open(PIPELINE_FILE, 'rb'))\n",
    "\n",
    "\n",
    "def _get_class(prediction):\n",
    "    \"\"\"\n",
    "    This function returns the class name of a given prediction. \n",
    "    \"\"\"\n",
    "    \n",
    "    _download(CLASSES_FILE)\n",
    "    \n",
    "    with open(CLASSES_FILE) as f:\n",
    "        file = f.readlines()\n",
    "        \n",
    "    classes = list(map(lambda x: x.replace(\"'\", \"\"), file[0].split(',')))\n",
    "    return classes[prediction]\n",
    "\n",
    "\n",
    "def _download(file):\n",
    "    \"\"\"\n",
    "    This function will download a file from S3 if it doesn't already exist. The\n",
    "    function will use the `S3_LOCATION` environment variable to determine the\n",
    "    location of the file.\n",
    "    \"\"\"\n",
    "    if CLASSES_FILE.exists():\n",
    "        return\n",
    "        \n",
    "    s3_uri = os.environ.get(\"S3_LOCATION\", None)\n",
    "        \n",
    "    s3_parts = s3_uri.split('/', 3)\n",
    "    bucket = s3_parts[2]\n",
    "    key = s3_parts[3]\n",
    "\n",
    "    s3.Bucket(bucket).download_file(f\"{key}/{file.name}\", str(file))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "656e7e63-39a8-4859-af1e-3e7b1573f9a2",
   "metadata": {},
   "source": [
    "## Step 3 - Testing the Inference Code\n",
    "\n",
    "Let's test the inference code locally to ensure it works before deploying it. The `handler()` function is the entry point that will be called by SageMaker whenever the endpoint receives a request.\n",
    "\n",
    "When testing the inference code, we want to set the `context` to `None` so the function recognizes we are calling it locally. We also want to set the `S3_LOCATION` environment variable to the S3 location of the Scikit-Learn pipeline and the list of supported classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "a7aca49b-2080-4068-80e6-8b3f10e54177",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: S3_LOCATION=s3://mlschool/penguins/preprocessing\n",
      "Handling endpoint request\n",
      "Processing input data...\n",
      "Sending input data to model to make a prediction...\n",
      "Response: {'predictions': [[0.2, 0.5, 0.3]]}\n",
      "Processing prediction received from the model...\n",
      "Prediction: 1. Confidence: 0.5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('{\"species\": \"Chinstrap\", \"prediction\": 1, \"confidence\": 0.5}',\n",
       " 'application/json')"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from penguins.code.inference import handler\n",
    "\n",
    "# This sets the environment variable indicating the location of the\n",
    "# pipeline and the classes files we need to download from S3.\n",
    "%env S3_LOCATION=$preprocessor_destination.default_value\n",
    "\n",
    "handler(\n",
    "    data={\n",
    "        \"island\": \"Biscoe\",\n",
    "        \"culmen_length_mm\": 48.6,\n",
    "        \"culmen_depth_mm\": 16.0,\n",
    "        \"flipper_length_mm\": 230.0,\n",
    "        \"body_mass_g\": 5800.0,\n",
    "    }, \n",
    "    context=None\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed63f158-06f3-438b-b489-bb0496f2eddd",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Step 4 - Registering the Model\n",
    "\n",
    "We can now register a new [TensorFlowModel](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-serving-model). We must also ensure SageMaker repackages the model assets to include the `inference.py` file.\n",
    "\n",
    "SageMaker triggers a repack whenever we specify the `source_dir` attribute. We want that attribute to point to the local folder containing the `inference.py` file. SageMaker will automatically modify the original `model.tar.gz` package to include a `/code` folder containing the file. Since we need access to Scikit-Learn in our script, we can include a `requirements.txt` file in the same `/code` folder, and SageMaker will install everything in it. To repack the model assets, SageMaker will automatically include a new step in the pipeline right before registering the model.\n",
    "\n",
    "Here is what the new `model.tar.gz` package will look like:\n",
    "\n",
    "```\n",
    "model/\n",
    "    |--[model_version_number]\n",
    "        |--assets/\n",
    "        |--variables/\n",
    "        |--saved_model.pb\n",
    "code/\n",
    "    |--inference.py\n",
    "    |--requirements.txt\n",
    "```\n",
    "\n",
    "Let's use a [ModelStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.model_step.ModelStep) to register the model. Notice the following:\n",
    "\n",
    "* `model_data`: We use the model assets we generated during the Training or Tuning Step. We determined which assets to use back in Session 4 and stored them in the `model_data` variable.\n",
    "* `source_dir`: This points to the local folder containing the `inference.py` file. SageMaker will trigger a repack to include the `/code` folder in the model assets.\n",
    "* `env`: Our custom inference code expects an environment variable `S3_LOCATION` to point to the location of the Scikit-Learn pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a27e2c6d-977e-43ec-98d4-ec66781af582",
   "metadata": {},
   "source": [
    "SageMaker's default TensorFlow inference container doesn't come with Scikit-Learn installed, so we need to provide a `requirements.txt` file with the libraries we want SageMaker to install in our endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "ed40d098-d553-4d56-b2eb-f80cec420ca5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguins/code/requirements.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile $CODE_FOLDER/requirements.txt\n",
    "\n",
    "numpy==1.19.5\n",
    "pandas==1.2.5\n",
    "scikit-learn==0.23.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "07513884-c7bd-4710-9730-f8ca7fe904a5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.tensorflow.model:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n",
      "/usr/local/lib/python3.8/site-packages/sagemaker/workflow/pipeline_context.py:286: UserWarning: Running within a PipelineSession, there will be No Wait, No Logs, and No Job being started.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "model = TensorFlowModel(\n",
    "    model_data=train_model_step.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "    entry_point=\"inference.py\",\n",
    "    source_dir=str(CODE_FOLDER),\n",
    "    env={\n",
    "        \"S3_LOCATION\": preprocessor_destination,\n",
    "    },\n",
    "    framework_version=\"2.6\",\n",
    "    sagemaker_session=PipelineSession(),\n",
    "    role=role,\n",
    ")\n",
    "\n",
    "register_model_step = ModelStep(\n",
    "    name=\"register-model\",\n",
    "    step_args=model.register(\n",
    "        model_package_group_name=model_package_group_name,\n",
    "        model_metrics=model_metrics,\n",
    "        approval_status=\"Approved\",\n",
    "        \n",
    "        content_types=[\"application/json\"],\n",
    "        response_types=[\"application/json\"],\n",
    "        inference_instances=[\"ml.m5.large\"],\n",
    "        domain=\"MACHINE_LEARNING\",\n",
    "        task=\"CLASSIFICATION\",\n",
    "        framework=\"TENSORFLOW\",\n",
    "        framework_version=\"2.6\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "436b60d5-6be3-428f-9979-4fe9eebfc5eb",
   "metadata": {},
   "source": [
    "## Step 5 - Deploying the Model\n",
    "\n",
    "Let's use a [Lambda Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda) to deploy the model automatically.\n",
    "\n",
    "Let's start by writing the Lambda function to take the model information and create a new hosting endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "28c75afd-d0d0-47f4-b7b6-9d590c5b600e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguins/lambda.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $PENGUINS_FOLDER/lambda.py\n",
    "\n",
    "import os\n",
    "import json\n",
    "import boto3\n",
    "import time\n",
    "\n",
    "sagemaker = boto3.client(\"sagemaker\")\n",
    "\n",
    "def lambda_handler(event, context):\n",
    "    model_package_arn = event[\"model_package_arn\"]\n",
    "    endpoint_name = event[\"endpoint_name\"]\n",
    "    data_capture_percentage = event[\"data_capture_percentage\"]\n",
    "    data_capture_destination = event[\"data_capture_destination\"]\n",
    "    role = event[\"role\"]\n",
    "    \n",
    "    timestamp = time.strftime(\"%m%d%H%M%S\", time.localtime())\n",
    "    model_name = f\"penguins-model-{timestamp}\"\n",
    "    endpoint_config_name = f\"penguins-endpoint-config-{timestamp}\"\n",
    "\n",
    "    sagemaker.create_model(\n",
    "        ModelName=model_name, \n",
    "        ExecutionRoleArn=role, \n",
    "        Containers=[{\n",
    "            \"ModelPackageName\": model_package_arn\n",
    "        }] \n",
    "    )\n",
    "\n",
    "    sagemaker.create_endpoint_config(\n",
    "        EndpointConfigName=endpoint_config_name,\n",
    "        ProductionVariants=[\n",
    "            {\n",
    "                \"ModelName\": model_name,\n",
    "                \"InstanceType\": \"ml.m5.large\",\n",
    "                \"InitialVariantWeight\": 1,\n",
    "                \"InitialInstanceCount\": 1,\n",
    "                \"VariantName\": \"AllTraffic\",\n",
    "            }\n",
    "        ],\n",
    "        DataCaptureConfig={\n",
    "            \"EnableCapture\": True,\n",
    "            \"InitialSamplingPercentage\": data_capture_percentage,\n",
    "            \"DestinationS3Uri\": data_capture_destination,\n",
    "            \"CaptureOptions\": [\n",
    "                {\n",
    "                    'CaptureMode': \"Input\"\n",
    "                },\n",
    "                {\n",
    "                    'CaptureMode': \"Output\"\n",
    "                },\n",
    "            ],\n",
    "            \"CaptureContentTypeHeader\": {\n",
    "                \"JsonContentTypes\": [\n",
    "                    \"application/json\",\n",
    "                    \"application/octect-stream\"\n",
    "                ]\n",
    "            }\n",
    "        },\n",
    "    )\n",
    "\n",
    "    sagemaker.create_endpoint(\n",
    "        EndpointName=endpoint_name, \n",
    "        EndpointConfigName=endpoint_config_name,\n",
    "    )\n",
    "    \n",
    "    return {\n",
    "        \"statusCode\": 200,\n",
    "        \"body\": json.dumps(\"Endpoint deployed successfully\")\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c66a4446-6db4-4515-bb1d-88ee21013bb2",
   "metadata": {},
   "source": [
    "We need to ensure our Lambda function has permission to interact with SageMaker, so let's create a new role to run the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "f9941979-c085-4ef9-8c66-908378d89a91",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def create_lambda_role(role_name):\n",
    "    try:\n",
    "        response = iam_client.create_role(\n",
    "            RoleName = role_name,\n",
    "            AssumeRolePolicyDocument = json.dumps({\n",
    "                \"Version\": \"2012-10-17\",\n",
    "                \"Statement\": [\n",
    "                    {\n",
    "                        \"Effect\": \"Allow\",\n",
    "                        \"Principal\": {\n",
    "                            \"Service\": \"lambda.amazonaws.com\"\n",
    "                        },\n",
    "                        \"Action\": \"sts:AssumeRole\"\n",
    "                    }\n",
    "                ]\n",
    "            }),\n",
    "            Description=\"Lambda Pipeline Role\"\n",
    "        )\n",
    "\n",
    "        role_arn = response['Role']['Arn']\n",
    "\n",
    "        iam_client.attach_role_policy(\n",
    "            RoleName=role_name,\n",
    "            PolicyArn='arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'\n",
    "        )\n",
    "\n",
    "        iam_client.attach_role_policy(\n",
    "            PolicyArn='arn:aws:iam::aws:policy/AmazonSageMakerFullAccess',\n",
    "            RoleName=role_name\n",
    "        )\n",
    "\n",
    "        return role_arn\n",
    "\n",
    "    except iam_client.exceptions.EntityAlreadyExistsException:\n",
    "        response = iam_client.get_role(RoleName=role_name)\n",
    "        return response['Role']['Arn']\n",
    "\n",
    "\n",
    "lambda_role = create_lambda_role(\"lambda-pipeline-role\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad4dcc96-2890-43ff-a6cd-f2a9969adc52",
   "metadata": {},
   "source": [
    "## Step 6 - Setting up the Lambda Step\n",
    "\n",
    "Let's define the [LambdaStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.lambda_step.LambdaStep) that will run the function to deploy the model.\n",
    "\n",
    "We can use [Data Capture](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-data-capture.html) to record the inputs and outputs of the endpoint to use them later for monitoring the model. We'll enable Data Capture using the following settings:\n",
    "\n",
    "* `data_capture_percentage`: Represents the percentage of information that flows through the endpoint that we want to capture. For this example, we'll set that to 100%.\n",
    "* `data_capture_destination`: Specifies the S3 location where we want to store the captured data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "7c6eda89-70f3-49b4-8983-90f4db72fd02",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_capture_percentage = ParameterInteger(\n",
    "    name=\"data_capture_percentage\",\n",
    "    default_value=100,\n",
    ")\n",
    "\n",
    "data_capture_destination = ParameterString(\n",
    "    name=\"data_capture_destination\",\n",
    "    default_value=f\"{S3_FILEPATH}/monitoring/data-capture\",\n",
    ")\n",
    "\n",
    "deploy_fn = Lambda(\n",
    "    function_name=\"deploy_fn\",\n",
    "    execution_role_arn=lambda_role,\n",
    "    script=str(PENGUINS_FOLDER / \"lambda.py\"),\n",
    "    handler=\"lambda.lambda_handler\",\n",
    "    timeout=600\n",
    ")\n",
    "\n",
    "deploy_fn.upsert()\n",
    "\n",
    "deploy_step = LambdaStep(\n",
    "    name=\"deploy\",\n",
    "    lambda_func=deploy_fn,\n",
    "    inputs={\n",
    "        # We use the ARN of the model we registered to\n",
    "        # deploy it to the endpoint.\n",
    "        \"model_package_arn\": register_model_step.properties.ModelPackageArn,\n",
    "\n",
    "        \"endpoint_name\": endpoint_name,\n",
    "        \n",
    "        \"data_capture_percentage\": data_capture_percentage,\n",
    "        \"data_capture_destination\": data_capture_destination,\n",
    "        \n",
    "        \"role\": role,\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cfe2299-be2a-46a7-809e-6174d44abddb",
   "metadata": {},
   "source": [
    "## Step 7 - Modifying the Condition Step\n",
    "\n",
    "We need to modify the [Condition Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-condition) to include the new Deploy Step we just created. If the condition succeeds, we will register and deploy the custom model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "caacdde0-a65f-4225-8303-2b0106da1ed8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "condition_step = ConditionStep(\n",
    "    name=\"check-model-accuracy\",\n",
    "    conditions=[condition_gte],\n",
    "    if_steps=[\n",
    "        register_model_step, deploy_step\n",
    "    ],\n",
    "    else_steps=[fail_step], \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ade6d67c-6906-434d-9c41-17a4efeb38d2",
   "metadata": {},
   "source": [
    "## Step 8 - Running the Pipeline\n",
    "\n",
    "We can now run the pipeline. If the pipeline succeeds, there will be a new running endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "d1969a5e-2ebf-474f-a275-ae0946c2fab8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session4_pipeline = Pipeline(\n",
    "    name=\"penguins-session4-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location, \n",
    "        preprocessor_destination,\n",
    "        train_dataset_baseline_destination,\n",
    "        test_dataset_baseline_destination,\n",
    "        accuracy_threshold,\n",
    "        data_capture_percentage,\n",
    "        data_capture_destination,\n",
    "    ],\n",
    "    steps=[\n",
    "        preprocess_data_step, \n",
    "        train_model_step, \n",
    "        evaluate_model_step,\n",
    "        condition_step\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de309b9-6011-40fe-b03d-ff991700c517",
   "metadata": {},
   "source": [
    "Submit the pipeline definition to the SageMaker Pipelines service to create a pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "9e01390f-31a5-44b0-a9ea-2c4697a57aa1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker.workflow._utils:Popping out 'CertifyForMarketplace' from the pipeline definition since it will be overridden in pipeline execution time.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n",
      "Using provided s3_resource\n"
     ]
    }
   ],
   "source": [
    "session4_pipeline.upsert(role_arn=role)\n",
    "execution = session4_pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "536ca1b1-fdc6-4260-b6f7-ee3c6aab4e76",
   "metadata": {},
   "source": [
    "## Step 9 - Testing the Endpoint\n",
    "\n",
    "We can now create a [Predictor](https://sagemaker.readthedocs.io/en/stable/api/inference/predictors.html) to test the endpoint with a few examples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96e32865-705d-4a2e-a7df-7df198aa8c96",
   "metadata": {},
   "source": [
    "First, let's wait for the endpoint to be ready to service traffic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "17098e7e-fc61-4027-b98f-7f192ffdf9fd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "waiter = sagemaker_client.get_waiter(\"endpoint_in_service\")\n",
    "waiter.wait(\n",
    "    EndpointName=endpoint_name,\n",
    "    WaiterConfig={\n",
    "        \"Delay\": 10,\n",
    "        \"MaxAttempts\": 30\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "016bcfc1-32ba-4ec8-93f6-0c04b00cc4f8",
   "metadata": {},
   "source": [
    "Now that the endpoint is in service, we can create a [Predictor](https://sagemaker.readthedocs.io/en/stable/api/inference/predictors.html) using a JSON serializer and a deserializer to have it automatically serialize and deserialize the information to and from the endpoint. Check [Serializers](https://sagemaker.readthedocs.io/en/stable/api/inference/serializers.html) and [Deserializers](https://sagemaker.readthedocs.io/en/stable/api/inference/deserializers.html) for a list of supported serializers and deserializers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "862118a0-c00b-4f5a-9374-68a94dbfbc0c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "predictor = Predictor(\n",
    "    endpoint_name=endpoint_name,\n",
    "    serializer=JSONSerializer(),\n",
    "    deserializer=JSONDeserializer()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "529b98a7-2f36-4e4e-a277-c5172e5fbe51",
   "metadata": {
    "tags": []
   },
   "source": [
    "Running one example through the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "8148ef38-92ed-4f9a-8024-69e2d9d6b5d4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'species': 'Adelie', 'prediction': 0, 'confidence': 0.46396777}"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.predict({\n",
    "    \"island\": \"Dream\",\n",
    "    \"culmen_length_mm\": 46.4,\n",
    "    \"culmen_depth_mm\": 18.6,\n",
    "    \"flipper_length_mm\": 190.0,\n",
    "    \"body_mass_g\": 3450.0,\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9075ef4c-36a8-4c5d-88ef-2a3e49abdd51",
   "metadata": {},
   "source": [
    "Running another example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "2476337f-a39b-45d5-ab57-63567c3f0438",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'species': 'Gentoo', 'prediction': 2, 'confidence': 0.997359216}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.predict({\n",
    "    \"island\": \"Biscoe\",\n",
    "    \"culmen_length_mm\": 48.6,\n",
    "    \"culmen_depth_mm\": 16.0,\n",
    "    \"flipper_length_mm\": 230.0,\n",
    "    \"body_mass_g\": 5800.0,\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d34fcdcd-cc9e-4cf5-8da1-3dc73cb6b59a",
   "metadata": {},
   "source": [
    "## Step 10 - Cleaning up\n",
    "\n",
    "Before you finish, don't forget to clean up after yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "0cf877e9-e1bc-40f0-acaa-64b6a5e6650f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Deleting endpoint configuration with name: penguins-endpoint-config-0626140836\n",
      "INFO:sagemaker:Deleting endpoint with name: penguins-endpoint\n"
     ]
    }
   ],
   "source": [
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e544ae36-00b3-4bde-b133-c3a59bb7f1d8",
   "metadata": {},
   "source": [
    "# Session 5 - Data Monitoring\n",
    "\n",
    "In this session we'll set up a monitoring process to analyze the quality of the data our endpoint receives in production. For this, we will have SageMaker capture and evaluate the data observed by the endpoint.\n",
    "\n",
    "To enable this functionality, we need a couple of steps:\n",
    "\n",
    "1. Create a baseline to compare the real-time traffic.\n",
    "2. Set up a schedule to continuously evaluate and compare against the baseline.\n",
    "\n",
    "Notice that the Data Quality process uses the baseline dataset we generated during preprocessing. This baseline dataset is the same unprocessed train set in JSON format. We do this because we transformed the train data during the preprocessing step, but we need raw data because that's what the endpoint expects.\n",
    "\n",
    "Check [Amazon SageMaker Model Monitor](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_monitoring.html) for a brief explanation of how to use SageMaker's Model Monitoring functionality. [Monitor models for data and model quality, bias, and explainability](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html) is a much more extensive guide to monitoring in Amazon SageMaker.\n",
    "\n",
    "Here is what the Pipeline will look like at the end of this session:\n",
    "\n",
    "<img src='penguins/images/session5-pipeline.png' alt='Session 5 Pipeline' width=\"600\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "2c7e2f9d-cc75-46bc-8700-f7123292fac5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "from time import sleep\n",
    "from datetime import datetime\n",
    "from threading import Thread, Event\n",
    "\n",
    "from IPython.display import JSON\n",
    "\n",
    "from sagemaker.workflow.check_job_config import CheckJobConfig\n",
    "from sagemaker.workflow.quality_check_step import DataQualityCheckConfig, QualityCheckStep\n",
    "from sagemaker.workflow.execution_variables import ExecutionVariables\n",
    "\n",
    "from sagemaker.drift_check_baselines import DriftCheckBaselines\n",
    "from sagemaker.workflow.parameters import ParameterBoolean\n",
    "from sagemaker.model import Model\n",
    "from sagemaker.model_monitor import (\n",
    "    CronExpressionGenerator, EndpointInput, DefaultModelMonitor, MonitoringExecution\n",
    ")\n",
    "from sagemaker.model_monitor.dataset_format import DatasetFormat\n",
    "from sagemaker.s3 import S3Uploader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24c26ac4-5d30-41e9-8952-e4deb39de819",
   "metadata": {},
   "source": [
    "## Step 1 - Generating a Baseline\n",
    "\n",
    "Let's now configure the [Quality Check Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-quality-check) and feed it the train set we generated in the preprocessing step.\n",
    "\n",
    "We can configure the instance that will run the quality check using the [CheckJobConfig](https://sagemaker.readthedocs.io/en/v2.73.0/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.check_job_config.CheckJobConfig) class, and we can use the `DataQualityCheckConfig` class to configure the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "0b80bcab-d2c5-437c-a1c8-8eea208c0e29",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:Defaulting to the only supported framework/algorithm version: .\n",
      "INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.\n"
     ]
    }
   ],
   "source": [
    "data_quality_location = f\"{S3_FILEPATH}/monitoring/data-quality\"\n",
    "\n",
    "data_quality_baseline_step = QualityCheckStep(\n",
    "    name=\"generate-data-quality-baseline\",\n",
    "    \n",
    "    check_job_config = CheckJobConfig(\n",
    "        instance_type=\"ml.t3.xlarge\",\n",
    "        instance_count=1,\n",
    "        volume_size_in_gb=20,\n",
    "        sagemaker_session=sagemaker_session,\n",
    "        role=role,\n",
    "    ),\n",
    "    \n",
    "    quality_check_config = DataQualityCheckConfig(\n",
    "        # We will use the train dataset we generated during the preprocessing \n",
    "        # step to generate the data quality baseline.\n",
    "        baseline_dataset=preprocess_data_step.properties.ProcessingOutputConfig.Outputs[\"train-baseline\"].S3Output.S3Uri,\n",
    "\n",
    "        dataset_format=DatasetFormat.json(lines=True),\n",
    "        output_s3_uri=data_quality_location\n",
    "    ),\n",
    "    \n",
    "    skip_check=True,\n",
    "    register_new_baseline=True,\n",
    "    model_package_group_name=model_package_group_name,\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08936035-1a36-4a0c-9529-76cc60b7850d",
   "metadata": {},
   "source": [
    "## Step 2 - Running the Pipeline\n",
    "\n",
    "We can now run the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "8c926da4-043a-4237-ac71-90ef18cf806b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session5_pipeline = Pipeline(\n",
    "    name=\"penguins-session5-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location, \n",
    "        preprocessor_destination,\n",
    "        train_dataset_baseline_destination,\n",
    "        test_dataset_baseline_destination,\n",
    "        data_capture_percentage,\n",
    "        data_capture_destination,       \n",
    "        accuracy_threshold,\n",
    "    ],\n",
    "    steps=[\n",
    "        preprocess_data_step, \n",
    "        data_quality_baseline_step,\n",
    "        train_model_step, \n",
    "        evaluate_model_step,\n",
    "        condition_step\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34b9aeb7-1c63-470a-80dc-d5932715c088",
   "metadata": {},
   "source": [
    "Submit the pipeline definition to the SageMaker Pipelines service to create a pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0eb61f6e-2e07-4b11-9c50-3df8ab2164bb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n",
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n",
      "Using provided s3_resource\n"
     ]
    }
   ],
   "source": [
    "session5_pipeline.upsert(role_arn=role)\n",
    "execution = session5_pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60810a83-edba-464b-8b41-d96a89991769",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Step 3 - Statistics and Constraints\n",
    "\n",
    "Our pipeline generated baseline statistics and constraints using our train set. We can take a look at what these values look like by downloading them from S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "4c6670fa-8560-4c12-9400-235bf1b10a7a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/json": {
       "dataset": {
        "item_count": 239
       },
       "features": [
        {
         "inferred_type": "Fractional",
         "name": "body_mass_g",
         "numerical_statistics": {
          "common": {
           "num_missing": 0,
           "num_present": 239
          },
          "distribution": {
           "kll": {
            "buckets": [
             {
              "count": 9,
              "lower_bound": 2700,
              "upper_bound": 3030
             },
             {
              "count": 25,
              "lower_bound": 3030,
              "upper_bound": 3360
             },
             {
              "count": 41,
              "lower_bound": 3360,
              "upper_bound": 3690
             },
             {
              "count": 45,
              "lower_bound": 3690,
              "upper_bound": 4020
             },
             {
              "count": 24,
              "lower_bound": 4020,
              "upper_bound": 4350
             },
             {
              "count": 26,
              "lower_bound": 4350,
              "upper_bound": 4680
             },
             {
              "count": 25,
              "lower_bound": 4680,
              "upper_bound": 5010
             },
             {
              "count": 15,
              "lower_bound": 5010,
              "upper_bound": 5340
             },
             {
              "count": 17,
              "lower_bound": 5340,
              "upper_bound": 5670
             },
             {
              "count": 12,
              "lower_bound": 5670,
              "upper_bound": 6000
             }
            ],
            "sketch": {
             "data": [
              [
               4750,
               3625,
               3900,
               4400,
               3500,
               3900,
               3400,
               3675,
               4275,
               5050,
               5100,
               3700,
               3600,
               4300,
               3275,
               3950,
               3975,
               5750,
               5150,
               3650,
               5200,
               2900,
               4500,
               5550,
               3150,
               3500,
               5650,
               4750,
               3950,
               5800,
               6000,
               5550,
               3650,
               3300,
               5700,
               4000,
               3775,
               4625,
               3475,
               5350,
               4050,
               3700,
               4200,
               4475,
               2850,
               5850,
               4750,
               5500,
               3600,
               2900,
               4850,
               3300,
               3750,
               4750,
               5700,
               5000,
               4400,
               3500,
               4300,
               4725,
               4575,
               3250,
               4200,
               5200,
               4200,
               4925,
               3700,
               4400,
               3550,
               3600,
               5300,
               4650,
               3800,
               5600,
               4400,
               3325,
               5300,
               5250,
               3200,
               3900,
               3350,
               3450,
               5000,
               4150,
               3675,
               5450,
               4000,
               5700,
               3450,
               5400,
               4725,
               3950,
               4100,
               3400,
               5550,
               3950,
               3250,
               3725,
               3900,
               3450,
               3175,
               3425,
               3400,
               4000,
               3700,
               5500,
               4400,
               4850,
               4650,
               4100,
               4625,
               4300,
               6000,
               4775,
               3250,
               3800,
               4000,
               3650,
               3900,
               3800,
               5150,
               4250,
               3800,
               4500,
               4450,
               3850,
               4800,
               5050,
               4250,
               4300,
               3650,
               3450,
               4675,
               3950,
               2975,
               5400,
               5800,
               4600,
               3325,
               3400,
               5350,
               3450,
               3400,
               2850,
               3725,
               4075,
               2700,
               3600,
               4500,
               2900,
               3700,
               4050,
               4700,
               5400,
               3200,
               3325,
               5050,
               3500,
               3000,
               5550,
               5200,
               3725,
               3450,
               4100,
               4150,
               3800,
               4375,
               4300,
               4550,
               4000,
               4800,
               3200,
               3325,
               4100,
               4700,
               3700,
               5000,
               3000,
               3175,
               3950,
               3950,
               5000,
               3300,
               4250,
               3900,
               5850,
               3050,
               3600,
               3900,
               4400,
               4600,
               3900,
               3800,
               3600,
               3525,
               4900,
               5950,
               4650,
               4600,
               3525,
               4800,
               5250,
               3450,
               3400,
               3700,
               4150,
               4400,
               3550,
               4925,
               5650,
               3300,
               4450,
               5300,
               3350,
               3550,
               3300,
               5700,
               3575,
               3475,
               3750,
               4950,
               5500,
               5300,
               3875,
               4250,
               3075,
               5650,
               4875,
               3150,
               4950,
               3600,
               3900,
               3150,
               3750,
               4875,
               4050,
               3450,
               4450,
               3750
              ]
             ],
             "parameters": {
              "c": 0.64,
              "k": 2048
             }
            }
           }
          },
          "max": 6000,
          "mean": 4204.07949790795,
          "min": 2700,
          "std_dev": 799.6348473381913,
          "sum": 1004775
         }
        },
        {
         "inferred_type": "Fractional",
         "name": "culmen_depth_mm",
         "numerical_statistics": {
          "common": {
           "num_missing": 0,
           "num_present": 239
          },
          "distribution": {
           "kll": {
            "buckets": [
             {
              "count": 13,
              "lower_bound": 13.2,
              "upper_bound": 14.03
             },
             {
              "count": 28,
              "lower_bound": 14.03,
              "upper_bound": 14.86
             },
             {
              "count": 18,
              "lower_bound": 14.86,
              "upper_bound": 15.69
             },
             {
              "count": 24,
              "lower_bound": 15.69,
              "upper_bound": 16.52
             },
             {
              "count": 40,
              "lower_bound": 16.52,
              "upper_bound": 17.35
             },
             {
              "count": 32,
              "lower_bound": 17.35,
              "upper_bound": 18.18
             },
             {
              "count": 42,
              "lower_bound": 18.18,
              "upper_bound": 19.009999999999998
             },
             {
              "count": 22,
              "lower_bound": 19.009999999999998,
              "upper_bound": 19.84
             },
             {
              "count": 11,
              "lower_bound": 19.84,
              "upper_bound": 20.67
             },
             {
              "count": 9,
              "lower_bound": 20.67,
              "upper_bound": 21.5
             }
            ],
            "sketch": {
             "data": [
              [
               13.8,
               17.8,
               16.8,
               21.1,
               17,
               17.5,
               18.1,
               17.9,
               19.5,
               15.4,
               15.6,
               16.8,
               17,
               18.3,
               17.6,
               19.2,
               20.1,
               15.7,
               14.8,
               20.6,
               14.8,
               17,
               13.2,
               17,
               17.2,
               17.7,
               15,
               13.8,
               19,
               16,
               16.3,
               15,
               19.2,
               17.1,
               16.8,
               19,
               20.3,
               14.4,
               18.4,
               15.9,
               19.7,
               17.3,
               13.9,
               18.5,
               16.6,
               15.7,
               14.5,
               15,
               17.3,
               16.6,
               15,
               20,
               19.1,
               15,
               15.2,
               15.6,
               19.7,
               17.9,
               20.8,
               20,
               14,
               18.7,
               19.4,
               15.7,
               21.5,
               15.8,
               17.3,
               13.4,
               18.6,
               18.8,
               15.8,
               13.7,
               19,
               17.3,
               14.4,
               17,
               14.2,
               17.3,
               17.6,
               20,
               17.3,
               19.3,
               15.2,
               21.2,
               17.3,
               15.6,
               19.1,
               16.3,
               17.9,
               16.1,
               14.6,
               19.5,
               18.8,
               17.9,
               15.3,
               20,
               18.2,
               17.9,
               18.4,
               18.8,
               17.5,
               17.6,
               17.1,
               19.8,
               17.5,
               15.8,
               21.1,
               14.3,
               14.4,
               19.1,
               14.5,
               13.9,
               16.2,
               19,
               16.6,
               19.2,
               18.5,
               16.5,
               19.4,
               17.2,
               15.4,
               17.9,
               18.8,
               19.9,
               18.1,
               18.3,
               20.7,
               15.8,
               20,
               18.9,
               18.7,
               18.6,
               19.6,
               18.1,
               18.9,
               17.1,
               16.2,
               14.2,
               18.6,
               17.5,
               15.7,
               19,
               17.1,
               17.1,
               19.8,
               18.3,
               16.6,
               17.3,
               20.7,
               18.1,
               17.8,
               19.5,
               14.2,
               14.5,
               17.9,
               17.9,
               14.5,
               19.2,
               16.9,
               16.1,
               14.8,
               17,
               17.9,
               14.3,
               13.5,
               18.8,
               14.1,
               19.6,
               20,
               18,
               14.6,
               17.2,
               16.4,
               19,
               17.6,
               17.1,
               15,
               16.8,
               17,
               18,
               18.9,
               14.5,
               17.8,
               20.2,
               18.9,
               14.6,
               17.1,
               18.6,
               20.7,
               14.5,
               18.8,
               18.1,
               21.2,
               18.7,
               19.4,
               14.4,
               16.4,
               15.2,
               14.3,
               18.7,
               14.5,
               16.5,
               17,
               16.5,
               18.6,
               18.9,
               13.9,
               18.5,
               13.7,
               16.1,
               19.3,
               18.5,
               14.1,
               16.7,
               17.2,
               17.9,
               14.3,
               17.1,
               18.1,
               18.7,
               15,
               15.1,
               14.9,
               18.5,
               18.5,
               16,
               15.7,
               14,
               16.9,
               13.7,
               19,
               19.5,
               18.6,
               17.2,
               14.6,
               18.2,
               17.8,
               14.5,
               18.1
              ]
             ],
             "parameters": {
              "c": 0.64,
              "k": 2048
             }
            }
           }
          },
          "max": 21.5,
          "mean": 17.220083682008358,
          "min": 13.2,
          "std_dev": 1.964117013197759,
          "sum": 4115.599999999998
         }
        },
        {
         "inferred_type": "Fractional",
         "name": "culmen_length_mm",
         "numerical_statistics": {
          "common": {
           "num_missing": 0,
           "num_present": 239
          },
          "distribution": {
           "kll": {
            "buckets": [
             {
              "count": 12,
              "lower_bound": 33.5,
              "upper_bound": 35.73
             },
             {
              "count": 29,
              "lower_bound": 35.73,
              "upper_bound": 37.96
             },
             {
              "count": 27,
              "lower_bound": 37.96,
              "upper_bound": 40.19
             },
             {
              "count": 33,
              "lower_bound": 40.19,
              "upper_bound": 42.42
             },
             {
              "count": 21,
              "lower_bound": 42.42,
              "upper_bound": 44.65
             },
             {
              "count": 40,
              "lower_bound": 44.65,
              "upper_bound": 46.879999999999995
             },
             {
              "count": 29,
              "lower_bound": 46.879999999999995,
              "upper_bound": 49.11
             },
             {
              "count": 32,
              "lower_bound": 49.11,
              "upper_bound": 51.339999999999996
             },
             {
              "count": 13,
              "lower_bound": 51.339999999999996,
              "upper_bound": 53.57
             },
             {
              "count": 3,
              "lower_bound": 53.57,
              "upper_bound": 55.8
             }
            ],
            "sketch": {
             "data": [
              [
               45.2,
               38.9,
               47.5,
               34.6,
               45.5,
               41.1,
               43.5,
               50.9,
               42.2,
               45,
               48.2,
               40.9,
               37.6,
               41.5,
               38.8,
               38.3,
               40.2,
               50.4,
               49.1,
               39.3,
               45.2,
               38.6,
               46.1,
               52.1,
               36.2,
               39.6,
               47.8,
               44.9,
               51.4,
               48.6,
               51.1,
               50.7,
               51.3,
               37.8,
               49.8,
               40.6,
               51.7,
               48.4,
               36.6,
               50,
               50.7,
               37.8,
               45.5,
               37.5,
               36.5,
               49.3,
               45.5,
               49.1,
               42.4,
               43.2,
               48.5,
               50.3,
               37.6,
               49.6,
               50,
               46.4,
               44.1,
               46.5,
               54.2,
               41,
               43.3,
               51.5,
               35.1,
               50.8,
               46,
               49.4,
               47,
               43.3,
               40.6,
               38.9,
               45.2,
               45.5,
               49.5,
               50.8,
               45.1,
               36.4,
               51.3,
               44.4,
               41.1,
               38.2,
               42.5,
               36.7,
               50.5,
               42.3,
               49.8,
               52.5,
               42.1,
               50,
               36,
               49.9,
               47.4,
               51.9,
               51,
               50.1,
               50,
               38.8,
               46.1,
               35,
               40.8,
               52.2,
               35.6,
               38.1,
               40.2,
               55.8,
               35.5,
               53.4,
               41.3,
               46.8,
               46.2,
               41.1,
               49.1,
               43.8,
               48.8,
               43.2,
               45.2,
               35.9,
               41.5,
               46.8,
               37.2,
               38.8,
               46.8,
               39.7,
               36.7,
               53.5,
               39.6,
               47.6,
               52,
               46.3,
               37.8,
               40.8,
               39,
               46.4,
               39.2,
               38.2,
               37.5,
               52.2,
               49.5,
               47.5,
               41.1,
               48.5,
               48.7,
               38.7,
               34,
               36.4,
               52.7,
               42.7,
               46.9,
               45.7,
               42.5,
               34.5,
               36.6,
               42,
               42.8,
               47.6,
               40.5,
               38.5,
               45.1,
               43.1,
               37,
               49,
               46.5,
               40.7,
               35,
               44.5,
               42,
               40.6,
               46.2,
               49,
               52.8,
               44.1,
               45.4,
               34.6,
               48.1,
               50.8,
               42.9,
               36,
               45.5,
               37.3,
               38.1,
               41.6,
               40.5,
               45.1,
               39.5,
               42,
               40.9,
               48.4,
               39,
               49.7,
               39.6,
               46.5,
               39.6,
               37.2,
               38.6,
               37.7,
               45.6,
               46.5,
               45.2,
               43.5,
               48.2,
               45.4,
               46.2,
               51.1,
               40.2,
               37,
               41.4,
               46,
               45.7,
               42.2,
               47.2,
               49.5,
               37.6,
               50.8,
               48.5,
               42.5,
               39.6,
               46.7,
               50.2,
               45.9,
               34.1,
               39.1,
               47.5,
               48.1,
               46.2,
               41.4,
               42.8,
               37.7,
               54.3,
               47.5,
               35.7,
               42.6,
               33.5,
               50,
               37.9,
               38.6,
               46.9,
               41.1,
               36,
               43.2,
               37.8
              ]
             ],
             "parameters": {
              "c": 0.64,
              "k": 2048
             }
            }
           }
          },
          "max": 55.8,
          "mean": 43.89874476987454,
          "min": 33.5,
          "std_dev": 5.25806398954227,
          "sum": 10491.800000000016
         }
        },
        {
         "inferred_type": "Fractional",
         "name": "flipper_length_mm",
         "numerical_statistics": {
          "common": {
           "num_missing": 0,
           "num_present": 239
          },
          "distribution": {
           "kll": {
            "buckets": [
             {
              "count": 2,
              "lower_bound": 172,
              "upper_bound": 177.9
             },
             {
              "count": 17,
              "lower_bound": 177.9,
              "upper_bound": 183.8
             },
             {
              "count": 35,
              "lower_bound": 183.8,
              "upper_bound": 189.7
             },
             {
              "count": 59,
              "lower_bound": 189.7,
              "upper_bound": 195.6
             },
             {
              "count": 28,
              "lower_bound": 195.6,
              "upper_bound": 201.5
             },
             {
              "count": 11,
              "lower_bound": 201.5,
              "upper_bound": 207.4
             },
             {
              "count": 31,
              "lower_bound": 207.4,
              "upper_bound": 213.3
             },
             {
              "count": 28,
              "lower_bound": 213.3,
              "upper_bound": 219.2
             },
             {
              "count": 18,
              "lower_bound": 219.2,
              "upper_bound": 225.1
             },
             {
              "count": 10,
              "lower_bound": 225.1,
              "upper_bound": 231
             }
            ],
            "sketch": {
             "data": [
              [
               215,
               181,
               199,
               198,
               196,
               190,
               202,
               196,
               197,
               220,
               221,
               191,
               185,
               195,
               191,
               189,
               200,
               222,
               220,
               190,
               212,
               188,
               211,
               230,
               187,
               186,
               215,
               212,
               201,
               230,
               220,
               223,
               193,
               186,
               230,
               199,
               194,
               203,
               184,
               224,
               203,
               180,
               210,
               199,
               181,
               217,
               212,
               228,
               181,
               187,
               219,
               197,
               194,
               216,
               218,
               221,
               196,
               192,
               201,
               203,
               208,
               187,
               193,
               226,
               194,
               216,
               185,
               209,
               183,
               190,
               215,
               214,
               200,
               228,
               210,
               195,
               218,
               219,
               182,
               190,
               187,
               193,
               216,
               191,
               198,
               221,
               195,
               230,
               190,
               213,
               212,
               206,
               203,
               190,
               220,
               190,
               178,
               192,
               195,
               197,
               191,
               187,
               193,
               207,
               190,
               219,
               195,
               215,
               214,
               188,
               212,
               208,
               222,
               197,
               191,
               189,
               201,
               189,
               184,
               180,
               215,
               193,
               187,
               205,
               186,
               195,
               210,
               215,
               190,
               208,
               185,
               190,
               195,
               185,
               179,
               228,
               229,
               209,
               189,
               191,
               208,
               195,
               185,
               184,
               197,
               196,
               192,
               193,
               197,
               187,
               185,
               200,
               209,
               215,
               187,
               190,
               207,
               197,
               185,
               216,
               217,
               190,
               190,
               216,
               210,
               193,
               217,
               212,
               205,
               210,
               211,
               189,
               199,
               210,
               196,
               187,
               220,
               192,
               181,
               192,
               180,
               215,
               188,
               190,
               184,
               213,
               191,
               195,
               191,
               213,
               190,
               178,
               191,
               180,
               194,
               217,
               223,
               213,
               210,
               188,
               209,
               225,
               176,
               185,
               191,
               195,
               214,
               180,
               214,
               224,
               181,
               201,
               220,
               187,
               196,
               195,
               218,
               190,
               193,
               181,
               218,
               209,
               221,
               202,
               195,
               183,
               231,
               212,
               185,
               213,
               190,
               196,
               172,
               199,
               222,
               192,
               195,
               208,
               193
              ]
             ],
             "parameters": {
              "c": 0.64,
              "k": 2048
             }
            }
           }
          },
          "max": 231,
          "mean": 200.55648535564853,
          "min": 172,
          "std_dev": 13.91771807266565,
          "sum": 47933
         }
        },
        {
         "inferred_type": "String",
         "name": "island",
         "string_statistics": {
          "common": {
           "num_missing": 0,
           "num_present": 239
          },
          "distinct_count": 3,
          "distribution": {
           "categorical": {
            "buckets": [
             {
              "count": 82,
              "value": "Dream"
             },
             {
              "count": 39,
              "value": "Torgersen"
             },
             {
              "count": 118,
              "value": "Biscoe"
             }
            ]
           }
          }
         }
        },
        {
         "inferred_type": "String",
         "name": "species",
         "string_statistics": {
          "common": {
           "num_missing": 0,
           "num_present": 239
          },
          "distinct_count": 3,
          "distribution": {
           "categorical": {
            "buckets": [
             {
              "count": 108,
              "value": "Adelie"
             },
             {
              "count": 47,
              "value": "Chinstrap"
             },
             {
              "count": 84,
              "value": "Gentoo"
             }
            ]
           }
          }
         }
        }
       ],
       "version": 0
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 84,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "statistics = f\"{data_quality_location}/statistics.json\"\n",
    "JSON(json.loads(S3Downloader.read_file(statistics)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "c17076de-e3ba-4b0d-9472-d4d2dab542bd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/json": {
       "features": [
        {
         "completeness": 1,
         "inferred_type": "Fractional",
         "name": "body_mass_g",
         "num_constraints": {
          "is_non_negative": true
         }
        },
        {
         "completeness": 1,
         "inferred_type": "Fractional",
         "name": "culmen_depth_mm",
         "num_constraints": {
          "is_non_negative": true
         }
        },
        {
         "completeness": 1,
         "inferred_type": "Fractional",
         "name": "culmen_length_mm",
         "num_constraints": {
          "is_non_negative": true
         }
        },
        {
         "completeness": 1,
         "inferred_type": "Fractional",
         "name": "flipper_length_mm",
         "num_constraints": {
          "is_non_negative": true
         }
        },
        {
         "completeness": 1,
         "inferred_type": "String",
         "name": "island",
         "string_constraints": {
          "domains": [
           "Dream",
           "Torgersen",
           "Biscoe"
          ]
         }
        },
        {
         "completeness": 1,
         "inferred_type": "String",
         "name": "species",
         "string_constraints": {
          "domains": [
           "Adelie",
           "Chinstrap",
           "Gentoo"
          ]
         }
        }
       ],
       "monitoring_config": {
        "datatype_check_threshold": 1,
        "distribution_constraints": {
         "categorical_comparison_threshold": 0.1,
         "categorical_drift_method": "LInfinity",
         "comparison_method": "Robust",
         "comparison_threshold": 0.1,
         "perform_comparison": "Enabled"
        },
        "domain_content_threshold": 1,
        "emit_metrics": "Enabled",
        "evaluate_constraints": "Enabled"
       },
       "version": 0
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 85,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "constraints = f\"{data_quality_location}/constraints.json\"\n",
    "JSON(json.loads(S3Downloader.read_file(constraints)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16bc1535-7854-4901-b9f0-3c3250b8252a",
   "metadata": {},
   "source": [
    "## Step 4 - Generating Endpoint Traffic\n",
    "\n",
    "Let's generate some traffic for our endpoint so we can test the monitoring functionality. We will repeatedly send every sample from the dataset to the endpoint to simulate real prediction requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "480e5feb-915a-4e38-9b96-1f092fa09507",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def generate_traffic(predictor):\n",
    "    \n",
    "    def _predict(data, predictor, stop_traffic_thread):\n",
    "        for index, row in data.iterrows():\n",
    "            predictor.predict(row.to_dict(), inference_id=str(index))\n",
    "            \n",
    "            sleep(1)\n",
    "\n",
    "            if stop_traffic_thread.is_set():\n",
    "                break\n",
    "\n",
    "    def _generate_prediction_data(data, predictor, stop_traffic_thread):\n",
    "        while True:\n",
    "            print(f\"Generating {data.shape[0]} predictions...\")\n",
    "            _predict(data, predictor, stop_traffic_thread)\n",
    "            \n",
    "            if stop_traffic_thread.is_set():\n",
    "                break\n",
    "\n",
    "                \n",
    "    stop_traffic_thread = Event()\n",
    "    \n",
    "    data = pd.read_csv(LOCAL_FILEPATH).dropna()\n",
    "    data.drop([\"sex\", \"species\"], axis=1, inplace=True)\n",
    "    \n",
    "    traffic_thread = Thread(\n",
    "        target=_generate_prediction_data,\n",
    "        args=(data, predictor, stop_traffic_thread,)\n",
    "    )\n",
    "    \n",
    "    traffic_thread.start()\n",
    "    \n",
    "    return stop_traffic_thread, traffic_thread\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e3c5fb0-0580-4472-bb9e-e1b60aa01693",
   "metadata": {},
   "source": [
    "Let's start generating the traffic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "fa8bb3ca-acdc-49cd-a77d-8894fec8e01e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating 334 predictions...\n"
     ]
    }
   ],
   "source": [
    "waiter = sagemaker_client.get_waiter(\"endpoint_in_service\")\n",
    "waiter.wait(\n",
    "    EndpointName=endpoint_name,\n",
    "    WaiterConfig={\n",
    "        \"Delay\": 10,\n",
    "        \"MaxAttempts\": 30\n",
    "    }\n",
    ")\n",
    "\n",
    "predictor = Predictor(\n",
    "    endpoint_name=endpoint_name,\n",
    "    serializer=JSONSerializer(),\n",
    "    deserializer=JSONDeserializer()\n",
    ")\n",
    "\n",
    "stop_traffic_thread, traffic_thread = generate_traffic(predictor)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "184a24f8-9fac-47ef-b707-4919d2c14ecd",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Step 5 - Checking the Captured Data\n",
    "\n",
    "Let's check the S3 location where the endpoint stores the requests and responses that it receives.\n",
    "\n",
    "Notice that it make take a few minutes for the first few files to show up in S3. Keep running the following line until you get some."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "3ba479ee-aefc-4480-9ebf-cd3e679edbe0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['s3://mlschool/penguins/monitoring/data-capture/penguins-endpoint/AllTraffic/2023/06/27/10/58-55-069-5a80dc2b-dafe-4be7-920a-eaece978479b.jsonl',\n",
       " 's3://mlschool/penguins/monitoring/data-capture/penguins-endpoint/AllTraffic/2023/06/27/10/59-56-068-4bdc305d-5965-490e-b0e5-5e687dac7702.jsonl',\n",
       " 's3://mlschool/penguins/monitoring/data-capture/penguins-endpoint/AllTraffic/2023/06/27/11/00-56-384-6decb274-dfde-459c-9bf9-e53036a12711.jsonl']"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files = S3Downloader.list(data_capture_destination.default_value)[:3]\n",
    "files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52455abe-3eea-46e5-a4b3-95a83ba5f926",
   "metadata": {},
   "source": [
    "These files contain the data captured by the endpoint in a SageMaker-specific JSON-line format. Each inference request is captured in a single line in the `jsonl` file. The line contains both the input and output merged together.\n",
    "\n",
    "Let's read the first line from the first file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "ea31c263-f941-4177-bdc7-842e4b6172b9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"captureData\": {\n",
      "    \"endpointInput\": {\n",
      "      \"observedContentType\": \"application/json\",\n",
      "      \"mode\": \"INPUT\",\n",
      "      \"data\": \"{\\\"island\\\": \\\"Torgersen\\\", \\\"culmen_length_mm\\\": 39.1, \\\"culmen_depth_mm\\\": 18.7, \\\"flipper_length_mm\\\": 181.0, \\\"body_mass_g\\\": 3750.0}\",\n",
      "      \"encoding\": \"JSON\"\n",
      "    },\n",
      "    \"endpointOutput\": {\n",
      "      \"observedContentType\": \"application/json\",\n",
      "      \"mode\": \"OUTPUT\",\n",
      "      \"data\": \"{\\\"species\\\": \\\"Adelie\\\", \\\"prediction\\\": 0, \\\"confidence\\\": 0.809994876}\",\n",
      "      \"encoding\": \"JSON\"\n",
      "    }\n",
      "  },\n",
      "  \"eventMetadata\": {\n",
      "    \"eventId\": \"705ebc5b-70c7-4eab-977a-807320b1b589\",\n",
      "    \"inferenceId\": \"0\",\n",
      "    \"inferenceTime\": \"2023-06-27T10:58:55Z\"\n",
      "  },\n",
      "  \"eventVersion\": \"0\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "if len(files):\n",
    "    lines = S3Downloader.read_file(files[0])\n",
    "    print(json.dumps(json.loads(lines.split(\"\\n\")[0]), indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c7a58ae-d3aa-4f69-bd52-66b05458bee7",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Step 6 - Scheduling the Monitoring Job\n",
    "\n",
    "We can now set up a schedule to continuously monitor data going into the endpoint and compare it to the baseline we generated before. This monitoring job will use the baseline statistics and constraints we generated during the Data Quality Check Step. Check [Schedule Data Quality Monitoring Jobs](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-schedule-data-monitor.html) for more information.\n",
    "\n",
    "SageMaker looks for violations in the data captured by the endpoint. By default, it combines the input data with the endpoint output and compare the result with the baseline we generated. If we let SageMaker do this, we will get a few violations, for example an \"extra column check\" violation because the fields `confidence` and `prediction` don't exist in the baseline data.\n",
    "\n",
    "We can fix these violations by creating a preprocessing script configuring the data we want the monitoring job to use.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "345efa6b-40bd-4084-83fd-654847bf22c3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DATA_QUALITY_PREPROCESSOR = \"data_quality_preprocessor.py\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a22c83ca-9b2c-40cc-81f9-bf85c0f17b5e",
   "metadata": {},
   "source": [
    "Here is the preprocessing script for the Data Quality Monitoring Job. Check [Preprocessing and Postprocessing](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-pre-and-post-processing.html) for more information about how to configure these scripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "2efcc106-b86d-4998-8fae-095c52a2b56b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguins/data_quality_preprocessor.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {PENGUINS_FOLDER}/{DATA_QUALITY_PREPROCESSOR}\n",
    "import json\n",
    "\n",
    "def preprocess_handler(inference_record):\n",
    "    input_data = inference_record.endpoint_input.data\n",
    "    output_data = json.loads(inference_record.endpoint_output.data)\n",
    "    \n",
    "    response = json.loads(input_data)\n",
    "    response[\"species\"] = output_data[\"species\"]\n",
    "\n",
    "    # The `response` variable contains the data that we want the\n",
    "    # monitoring job to use to compare with the baseline.\n",
    "    return response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4fa310b-2443-4a61-a09a-ea59df21f5c6",
   "metadata": {},
   "source": [
    "The monitoring schedule expects an S3 location pointing to the preprocessing script. Let's upload the script to the default bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "f501b907-6844-412e-bb79-e04fce790e7c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://sagemaker-us-east-1-325223348818/penguins-monitoring/data_quality_preprocessor.py'"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bucket = boto3.Session().resource(\"s3\").Bucket(sagemaker_session.default_bucket())\n",
    "prefix = \"penguins-monitoring\"\n",
    "bucket.Object(os.path.join(prefix, DATA_QUALITY_PREPROCESSOR)).upload_file(str(PENGUINS_FOLDER / DATA_QUALITY_PREPROCESSOR))\n",
    "data_quality_preprocessor = f\"s3://{os.path.join(bucket.name, prefix, DATA_QUALITY_PREPROCESSOR)}\"\n",
    "data_quality_preprocessor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b59c8199-5c13-4284-9ca9-1b695e3ff122",
   "metadata": {},
   "source": [
    "We can now set up the Data Quality Monitoring Job using the [DefaultModelMonitor](https://sagemaker.readthedocs.io/en/stable/api/inference/model_monitor.html#sagemaker.model_monitor.model_monitoring.DefaultModelMonitor) class. Notice how we specify the `record_preprocessor_script` using the S3 location where we uploaded our script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "5ded982d-d858-4ca6-b754-196917251a60",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:Defaulting to the only supported framework/algorithm version: .\n",
      "INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.\n",
      "INFO:sagemaker.model_monitor.model_monitoring:Creating Monitoring Schedule with name: penguins-data-monitoring-schedule\n"
     ]
    }
   ],
   "source": [
    "data_monitor = DefaultModelMonitor(\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    "    instance_count=1,\n",
    "    max_runtime_in_seconds=3600,\n",
    "    role=role,\n",
    ")\n",
    "\n",
    "data_monitor.create_monitoring_schedule(\n",
    "    monitor_schedule_name=\"penguins-data-monitoring-schedule\",\n",
    "    endpoint_input=predictor.endpoint_name,\n",
    "    record_preprocessor_script=data_quality_preprocessor,\n",
    "    statistics=f\"{data_quality_location}/statistics.json\",\n",
    "    constraints=f\"{data_quality_location}/constraints.json\",\n",
    "    schedule_cron_expression=CronExpressionGenerator.hourly(),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3705ec7-e7fb-4eca-8290-53df06361e77",
   "metadata": {},
   "source": [
    "You can describe the schedule to see more information about the Data Quality Monitoring Job."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a43cb2fa-0947-479a-b6be-969b0878cb6f",
   "metadata": {},
   "source": [
    "## Step 7 - Introducing a Violation\n",
    "\n",
    "Let's make a prediction for a penguin and include extra fields in the request. This should be flagged by the monitoring job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "c4f88fd4-ceae-4b1f-9085-b7cf54bd899f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'species': 'Chinstrap', 'prediction': 1, 'confidence': 0.561856747}"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.predict({\n",
    "    \"island\": \"Dream\",\n",
    "    \"culmen_length_mm\": 46.4,\n",
    "    \"culmen_depth_mm\": 18.6,\n",
    "    \"flipper_length_mm\": 190.0,\n",
    "    \"body_mass_g\": 5608.0,\n",
    "    \n",
    "    # These two columns are not in the baseline data,\n",
    "    # so they will be reported by the monitoring job\n",
    "    # as a violation.\n",
    "    \"name\": \"Johnny\",\n",
    "    \"height\": 28.0\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8863d029-90ff-4626-8e40-af153bb7d6b3",
   "metadata": {},
   "source": [
    "## Step 8 - Checking Monitoring Violations\n",
    "\n",
    "We can check the results of the monitoring job by looking at whether it generated any violations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "2e6ce5ec-a4c4-4f79-a78b-d27f678f6db9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'MonitoringScheduleArn': 'arn:aws:sagemaker:us-east-1:325223348818:monitoring-schedule/penguins-data-monitoring-schedule',\n",
       " 'MonitoringScheduleName': 'penguins-data-monitoring-schedule',\n",
       " 'MonitoringScheduleStatus': 'Scheduled',\n",
       " 'MonitoringType': 'DataQuality',\n",
       " 'CreationTime': datetime.datetime(2023, 6, 28, 10, 25, 47, 459000, tzinfo=tzlocal()),\n",
       " 'LastModifiedTime': datetime.datetime(2023, 6, 28, 11, 18, 42, 954000, tzinfo=tzlocal()),\n",
       " 'MonitoringScheduleConfig': {'ScheduleConfig': {'ScheduleExpression': 'cron(0 * ? * * *)'},\n",
       "  'MonitoringJobDefinitionName': 'data-quality-job-definition-2023-06-28-10-25-46-656',\n",
       "  'MonitoringType': 'DataQuality'},\n",
       " 'EndpointName': 'penguins-endpoint',\n",
       " 'LastMonitoringExecutionSummary': {'MonitoringScheduleName': 'penguins-data-monitoring-schedule',\n",
       "  'ScheduledTime': datetime.datetime(2023, 6, 28, 11, 0, tzinfo=tzlocal()),\n",
       "  'CreationTime': datetime.datetime(2023, 6, 28, 11, 5, 38, 386000, tzinfo=tzlocal()),\n",
       "  'LastModifiedTime': datetime.datetime(2023, 6, 28, 11, 18, 42, 944000, tzinfo=tzlocal()),\n",
       "  'MonitoringExecutionStatus': 'Completed',\n",
       "  'ProcessingJobArn': 'arn:aws:sagemaker:us-east-1:325223348818:processing-job/model-monitoring-202306281100-84c05d03db8bb211e269b6b1',\n",
       "  'EndpointName': 'penguins-endpoint'},\n",
       " 'ResponseMetadata': {'RequestId': '9a24b30b-cc95-49d6-85cb-a8de1f1155e0',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '9a24b30b-cc95-49d6-85cb-a8de1f1155e0',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '957',\n",
       "   'date': 'Wed, 28 Jun 2023 12:00:21 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n"
     ]
    }
   ],
   "source": [
    "description = data_monitor.describe_schedule()\n",
    "description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "56bd9ea6-e0e0-43b2-b1e6-05d18a131e8d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status: Completed\n",
      "Generating 334 predictions...\n"
     ]
    }
   ],
   "source": [
    "status = description[\"LastMonitoringExecutionSummary\"][\"MonitoringExecutionStatus\"]\n",
    "print(f\"Status: {status}\")\n",
    "\n",
    "if status == \"CompletedWithViolations\":\n",
    "    processing_job_arn = description[\"LastMonitoringExecutionSummary\"][\"ProcessingJobArn\"]\n",
    "    execution = MonitoringExecution.from_processing_arn(sagemaker_session=sagemaker_session, processing_job_arn=processing_job_arn)\n",
    "    execution_destination = execution.output.destination\n",
    "    \n",
    "    violations_filepath = os.path.join(execution_destination, \"constraint_violations.json\")\n",
    "    violations = json.loads(S3Downloader.read_file(violations_filepath))[\"violations\"]\n",
    "    \n",
    "    print(json.dumps(violations, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13d1f841-8962-436c-9766-0359a709747b",
   "metadata": {},
   "source": [
    "## Step 9 - Cleaning up\n",
    "\n",
    "Let's stop the monitoring jobs by deleting the monitoring schedule we created before. The following function waits for the job to finish before deleting it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "id": "4a85cf8e-0d56-483c-ae68-385cf79ac0ad",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def delete_monitoring_schedule(schedule):\n",
    "    attempts = 30\n",
    "    \n",
    "    try:\n",
    "        status = schedule.describe_schedule()[\"MonitoringScheduleStatus\"]\n",
    "    except Exception:\n",
    "        print(\"Monitoring schedule deleted.\")\n",
    "        return\n",
    "        \n",
    "    while status in (\"Pending\", \"InProgress\") and attempts > 0:\n",
    "        attempts -= 1\n",
    "        print(f\"Monitoring schedule status: {status}. Waiting for it to finish.\")\n",
    "        time.sleep(30)\n",
    "        status = schedule.describe_schedule()[\"MonitoringScheduleStatus\"]\n",
    "\n",
    "    if status not in (\"Pending\", \"InProgress\"):\n",
    "        schedule.delete_monitoring_schedule()\n",
    "        print(\"Monitoring schedule deleted.\")\n",
    "    else:\n",
    "        print(\"Waiting for monitoring schedule timed out\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7874ccff-3f43-4aea-ab13-ef41b4d5bae6",
   "metadata": {},
   "outputs": [],
   "source": [
    "delete_monitoring_schedule(data_monitor)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91729f95-4966-4741-aa74-2d90382710ea",
   "metadata": {},
   "source": [
    "Let's now stop the thread generating traffic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "id": "b9ec787f-4ee7-4620-ac2b-a77c977a2014",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "stop_traffic_thread.set()\n",
    "traffic_thread.join()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1776d472-323f-4205-9e6d-076375323280",
   "metadata": {},
   "source": [
    "Finally, we can delete the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "63614880-de4b-4cf9-bf1b-3f141d881559",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Deleting endpoint configuration with name: penguins-endpoint-config-0628070933\n",
      "INFO:sagemaker:Deleting endpoint with name: penguins-endpoint\n"
     ]
    }
   ],
   "source": [
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "917fe920-172d-4b07-bd7d-b2ecea536319",
   "metadata": {},
   "source": [
    "# Session 6 - Model Monitoring\n",
    "\n",
    "This session aims to set up a monitoring process to analyze the quality of the model predictions. For this, we need to generate ground truth for the data captured by the endpoint and compare it with a baseline performance.\n",
    "\n",
    "Check [Amazon SageMaker Model Monitor](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_monitoring.html) for a brief explanation of how to use SageMaker's Model Monitoring functionality. [Monitor models for data and model quality, bias, and explainability](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html) is a much more extensive guide to Model Monitoring in Amazon SageMaker.\n",
    "\n",
    "Here is what the Pipeline will look like at the end of this session:\n",
    "\n",
    "<img src='penguins/images/session6-pipeline.png' alt='Session 6 Pipeline' width=\"600\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "748c47c5-71e6-4f76-9a8b-9a5f4716e806",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.quality_check_step import ModelQualityCheckConfig\n",
    "\n",
    "from sagemaker.inputs import CreateModelInput, TransformInput\n",
    "from sagemaker.transformer import Transformer\n",
    "from sagemaker.workflow.steps import CreateModelStep, TransformStep\n",
    "from sagemaker.model_monitor import ModelQualityMonitor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81430dfd-2524-43e4-bfe9-c6545316005d",
   "metadata": {},
   "source": [
    "## Step 1 - Creating Test Predictions\n",
    "\n",
    "To create a baseline to compare the model performance, we must create predictions for the test set and compare them with the predictions from the model. We can do this by running a Batch Transform Job to predict every sample from the test dataset. We can use a [Transform Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-transform) as part of the pipeline to run this job. You can check [Batch Transform](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html) for more information about Batch Transform Jobs.\n",
    "\n",
    "The Transform Step requires a model to generate predictions, so we need a Model Step that creates a model.\n",
    "\n",
    "We also need to configure the [Batch Transform Job](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html) using a [Transform Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-transform). This Batch Transform Job will run every sample from the training dataset through the model so we can compute the baseline metrics. We can use an instance of the [Transformer](https://sagemaker.readthedocs.io/en/stable/api/inference/transformer.html) class to configure the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "1987a788-de7a-4f60-ac8d-819d9ffcdf8e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "create_model_step = ModelStep(\n",
    "    name=\"create-model\",\n",
    "    step_args=model.create(instance_type=\"ml.m5.large\"),\n",
    ")\n",
    "\n",
    "transformer = Transformer(\n",
    "    model_name=create_model_step.properties.ModelName,\n",
    "    \n",
    "    instance_type=\"ml.c5.xlarge\",\n",
    "    instance_count=1,\n",
    "    \n",
    "    accept=\"application/json\",\n",
    "    strategy=\"SingleRecord\",\n",
    "    assemble_with=\"Line\",\n",
    "    \n",
    "    output_path=f\"{S3_FILEPATH}/transform\",\n",
    ")\n",
    "\n",
    "generate_test_predictions_step = TransformStep(\n",
    "    name=\"generate_test_predictions\",\n",
    "    transformer=transformer,\n",
    "    inputs=TransformInput(\n",
    "        \n",
    "        # We will use the test dataset we generated during the preprocessing \n",
    "        # step to run it through the model and generate predictions.\n",
    "        data=preprocess_data_step.properties.ProcessingOutputConfig.Outputs[\"test-baseline\"].S3Output.S3Uri,\n",
    "\n",
    "        join_source=\"Input\",\n",
    "        content_type=\"application/json\",\n",
    "        split_type=\"Line\",\n",
    "    ),\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fafc7c4-6fef-4832-8b99-8c45d078fdd2",
   "metadata": {},
   "source": [
    "## Step 2 - Generating a Baseline\n",
    "\n",
    "Let's now configure the [Quality Check Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-quality-check) and feed it the data we generated in the Transform Step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "9aa3a284-8763-4000-a263-70314b530652",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:Defaulting to the only supported framework/algorithm version: .\n",
      "INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.\n"
     ]
    }
   ],
   "source": [
    "model_quality_location = f\"{S3_FILEPATH}/monitoring/model-quality\"\n",
    "\n",
    "model_quality_baseline_step = QualityCheckStep(\n",
    "    name=\"generate-model-quality-baseline\",\n",
    "    \n",
    "    check_job_config = CheckJobConfig(\n",
    "        instance_type=\"ml.t3.xlarge\",\n",
    "        instance_count=1,\n",
    "        volume_size_in_gb=20,\n",
    "        sagemaker_session=sagemaker_session,\n",
    "        role=role,\n",
    "    ),\n",
    "    \n",
    "    quality_check_config = ModelQualityCheckConfig(\n",
    "        # We are going to use the output of the Transform Step to generate\n",
    "        # the model quality baseline.\n",
    "        baseline_dataset=generate_test_predictions_step.properties.TransformOutput.S3OutputPath,\n",
    "\n",
    "        dataset_format=DatasetFormat.json(lines=True),\n",
    "\n",
    "        # We need to specify the problem type and the fields where the prediction\n",
    "        # and groundtruth are so the process knows how to interpret the results.\n",
    "        problem_type=\"MulticlassClassification\",\n",
    "        inference_attribute=\"$.SageMakerOutput.species\",\n",
    "        ground_truth_attribute=\"species\",\n",
    "\n",
    "        output_s3_uri=model_quality_location,\n",
    "    ),\n",
    "    \n",
    "    skip_check=True,\n",
    "    register_new_baseline=True,\n",
    "    model_package_group_name=model_package_group_name,\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "693535ba-fca7-4e89-a4cb-b4f333fa2d03",
   "metadata": {},
   "source": [
    "## Step 3 - Setting up Model Metrics\n",
    "\n",
    "We can configure a new set of [ModelMetrics](https://sagemaker.readthedocs.io/en/stable/api/inference/model_monitor.html#sagemaker.model_metrics.ModelMetrics) using the results of the Data and Model Quality Steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "a773f134-ac2f-4dba-976e-9b7f0b384b6e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_metrics = ModelMetrics(\n",
    "    model_data_statistics=MetricsSource(\n",
    "        s3_uri=data_quality_baseline_step.properties.CalculatedBaselineStatistics,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_data_constraints=MetricsSource(\n",
    "        s3_uri=data_quality_baseline_step.properties.CalculatedBaselineConstraints,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_statistics=MetricsSource(\n",
    "        s3_uri=model_quality_baseline_step.properties.CalculatedBaselineStatistics,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    \n",
    "    model_constraints=MetricsSource(\n",
    "        s3_uri=model_quality_baseline_step.properties.CalculatedBaselineConstraints,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    ")\n",
    "\n",
    "drift_check_baselines = DriftCheckBaselines(\n",
    "    model_data_statistics=MetricsSource(\n",
    "        s3_uri=data_quality_baseline_step.properties.BaselineUsedForDriftCheckStatistics,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_data_constraints=MetricsSource(\n",
    "        s3_uri=data_quality_baseline_step.properties.BaselineUsedForDriftCheckConstraints,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_statistics=MetricsSource(\n",
    "        s3_uri=model_quality_baseline_step.properties.BaselineUsedForDriftCheckStatistics,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_constraints=MetricsSource(\n",
    "        s3_uri=model_quality_baseline_step.properties.BaselineUsedForDriftCheckConstraints,\n",
    "        content_type=\"application/json\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba3487a0-05ad-4f3a-8f50-9884dc2aef64",
   "metadata": {},
   "source": [
    "## Step 4 - Registering the Model\n",
    "\n",
    "We need to redefine the Model Step to register the [TensorFlowModel](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-serving-model) so it takes into account the new metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "7056a009-91c0-4955-90dd-b90ef8cab149",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "register_model_step = ModelStep(\n",
    "    name=\"register-model\",\n",
    "    step_args=model.register(\n",
    "        model_package_group_name=model_package_group_name,\n",
    "        model_metrics=model_metrics,\n",
    "        drift_check_baselines=drift_check_baselines,\n",
    "        approval_status=\"Approved\",\n",
    "\n",
    "        content_types=[\"application/json\"],\n",
    "        response_types=[\"application/json\"],\n",
    "        inference_instances=[\"ml.m5.large\"],\n",
    "        domain=\"MACHINE_LEARNING\",\n",
    "        task=\"CLASSIFICATION\",\n",
    "        framework=\"TENSORFLOW\",\n",
    "        framework_version=\"2.6\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d00b5e6-9858-4acc-bbfe-a2ce24ec20e0",
   "metadata": {},
   "source": [
    "## Step 5 - Setting up the Condition Step\n",
    "\n",
    "We only want to compute the model quality baseline if the model's performance is above the predefined threshold. The [Condition Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-condition) will gate all necessary steps to compute the baseline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "bacaa9c6-22b0-48df-b138-95b6422fe834",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "condition_step = ConditionStep(\n",
    "    name=\"check-model-accuracy\",\n",
    "    conditions=[condition_gte],\n",
    "    if_steps=[\n",
    "        create_model_step, \n",
    "        generate_test_predictions_step, \n",
    "        model_quality_baseline_step, \n",
    "        register_model_step,\n",
    "        deploy_step\n",
    "    ],\n",
    "    else_steps=[fail_step], \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c95a7905-2550-4979-b885-f2daabb5d45e",
   "metadata": {},
   "source": [
    "## Step 6 - Running the Pipeline\n",
    "\n",
    "We can now run the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "4da5e453-acd8-47a0-a39f-264d05dd93d0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session6_pipeline = Pipeline(\n",
    "    name=\"penguins-session6-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location, \n",
    "        preprocessor_destination,\n",
    "        train_dataset_baseline_destination,\n",
    "        test_dataset_baseline_destination,\n",
    "        data_capture_percentage,\n",
    "        data_capture_destination,\n",
    "        accuracy_threshold,\n",
    "    ],\n",
    "    steps=[\n",
    "        preprocess_data_step, \n",
    "        data_quality_baseline_step,\n",
    "        train_model_step, \n",
    "        evaluate_model_step,\n",
    "        condition_step\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24713751-b109-47be-8d6f-5f4c7511f6d4",
   "metadata": {},
   "source": [
    "Submit the pipeline definition to the SageMaker Pipelines service to create a pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "b54dbf6e-9b52-41a6-bec3-91c56536b3a2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n",
      "Generating 334 predictions...\n",
      "Using provided s3_resource\n",
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker.workflow._utils:Popping out 'CertifyForMarketplace' from the pipeline definition since it will be overridden in pipeline execution time.\n",
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n",
      "Using provided s3_resource\n",
      "Using provided s3_resource\n",
      "Generating 334 predictions...\n",
      "Generating 334 predictions...\n"
     ]
    }
   ],
   "source": [
    "session6_pipeline.upsert(role_arn=role)\n",
    "execution = session6_pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b479872-58e2-4c61-8ccf-68d35185b282",
   "metadata": {},
   "source": [
    "## Step 7 - Generating Ground Truth Data\n",
    "\n",
    "To monitor our model, we need to generate ground truth data for the samples captured by the endpoint. We can simulate this by generating a random ground truth for every sample. Check [Ingest Ground Truth Labels and Merge Them With Predictions](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-merge.html) for more information about this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "f4ee5d6f-af93-41e8-a544-3a12dac99d8b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ground_truth_path = f\"{S3_FILEPATH}/monitoring/groundtruth\" \n",
    "\n",
    "def generate_ground_truth_data(predictor, ground_truth_path):\n",
    "    \n",
    "    def _generate_ground_truth_record(inference_id):\n",
    "        random.seed(inference_id)\n",
    "\n",
    "        return {\n",
    "            \"groundTruthData\": {\n",
    "                \"data\": random.choice([\"Adelie\", \"Chinstrap\", \"Gentoo\"]),\n",
    "                \"encoding\": \"CSV\",\n",
    "            },\n",
    "            \"eventMetadata\": {\n",
    "                \"eventId\": str(inference_id),\n",
    "            },\n",
    "            \"eventVersion\": \"0\",\n",
    "        }\n",
    "\n",
    "\n",
    "    def _upload_ground_truth(records, upload_time):\n",
    "        records = [json.dumps(r) for r in records]\n",
    "        data = \"\\n\".join(records)\n",
    "        uri = f\"{ground_truth_path}/{upload_time:%Y/%m/%d/%H/%M%S}.jsonl\"\n",
    "\n",
    "        print(f\"Uploading ground truth data to {uri}...\")\n",
    "\n",
    "        S3Uploader.upload_string_as_file_body(data, uri)    \n",
    "\n",
    "                \n",
    "    def _generate_ground_truth_data(max_records, stop_ground_truth_thread):\n",
    "        while True:\n",
    "            records = [_generate_ground_truth_record(i) for i in range(max_records)]\n",
    "            _upload_ground_truth(records, datetime.utcnow())\n",
    "\n",
    "            if stop_ground_truth_thread.is_set():\n",
    "                break\n",
    "\n",
    "            sleep(30)\n",
    "\n",
    "                \n",
    "    stop_ground_truth_thread = Event()\n",
    "    data = pd.read_csv(LOCAL_FILEPATH).dropna()\n",
    "    \n",
    "    groundtruth_thread = Thread(\n",
    "        target=_generate_ground_truth_data,\n",
    "        args=(len(data), stop_ground_truth_thread,)\n",
    "    )\n",
    "    \n",
    "    groundtruth_thread.start()\n",
    "    \n",
    "    return stop_ground_truth_thread, traffic_thread\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2beba47c-75bf-49fd-80e9-146c3ff815f1",
   "metadata": {},
   "source": [
    "Let's start generating traffic to the endpoint and create random ground truth data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "c9543c22-bfcc-427b-8c88-146a193e0009",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating 334 predictions...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/3750.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/3820.jsonl...\n",
      "Generating 334 predictions...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/3850.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/3920.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/12/3951.jsonl...\n"
     ]
    }
   ],
   "source": [
    "waiter = sagemaker_client.get_waiter(\"endpoint_in_service\")\n",
    "waiter.wait(\n",
    "    EndpointName=endpoint_name,\n",
    "    WaiterConfig={\n",
    "        \"Delay\": 10,\n",
    "        \"MaxAttempts\": 30\n",
    "    }\n",
    ")\n",
    "\n",
    "predictor = Predictor(\n",
    "    endpoint_name=endpoint_name,\n",
    "    serializer=JSONSerializer(),\n",
    "    deserializer=JSONDeserializer()\n",
    ")\n",
    "\n",
    "stop_traffic_thread, traffic_thread = generate_traffic(predictor)\n",
    "stop_ground_truth_thread, groundtruth_thread = generate_ground_truth_data(predictor, ground_truth_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75ecc695-535a-4556-89e6-082670746afa",
   "metadata": {},
   "source": [
    "## Step 8 - Scheduling the Monitoring Job\n",
    "\n",
    "Let's set up a schedule to continuously monitor the quality of the model and compare it to the baseline we generated before. This monitoring job will use the baseline constraints we generated during the Model Quality Check Step. Check [Schedule Model Quality Monitoring Jobs](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-schedule.html) for more information.\n",
    "\n",
    "To set up a Model Quality Monitoring Job, we can use the [ModelQualityMonitor](https://sagemaker.readthedocs.io/en/stable/api/inference/model_monitor.html#sagemaker.model_monitor.model_monitoring.ModelQualityMonitor) class. The [EndpointInput](https://sagemaker.readthedocs.io/en/v2.24.2/api/inference/model_monitor.html#sagemaker.model_monitor.model_monitoring.EndpointInput) instance configures the attribute the monitoring job should use to determine the prediction from the model.\n",
    "\n",
    "Check [Amazon SageMaker Model Quality Monitor](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker_model_monitor/model_quality/model_quality_churn_sdk.html) for a complete tutorial on how to run a Model Monitoring Job in SageMaker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "6b1bfce1-e812-4c6e-bade-8419d6c7ae5d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:Defaulting to the only supported framework/algorithm version: .\n",
      "INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.\n",
      "INFO:sagemaker.model_monitor.model_monitoring:Creating Monitoring Schedule with name: penguins-model-monitoring-schedule\n"
     ]
    }
   ],
   "source": [
    "model_monitor = ModelQualityMonitor(\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    "    instance_count=1,\n",
    "    max_runtime_in_seconds=1800,\n",
    "    role=role\n",
    ")\n",
    "\n",
    "model_monitor.create_monitoring_schedule(\n",
    "    monitor_schedule_name=\"penguins-model-monitoring-schedule\",\n",
    "    \n",
    "    endpoint_input = EndpointInput(\n",
    "        endpoint_name=predictor.endpoint_name,\n",
    "\n",
    "        # The endpoint returns an attribute `species` with the\n",
    "        # prediction from the model. That's the attribute we want to\n",
    "        # use to compare with the groundtruth.\n",
    "        inference_attribute=\"species\",\n",
    "\n",
    "        destination=\"/opt/ml/processing/input_data\",\n",
    "    ),\n",
    "    \n",
    "    problem_type=\"MulticlassClassification\",\n",
    "    ground_truth_input=ground_truth_path,\n",
    "    \n",
    "    constraints=f\"{model_quality_location}/constraints.json\",\n",
    "    \n",
    "    schedule_cron_expression=CronExpressionGenerator.hourly(),\n",
    "    output_s3_uri=f\"{S3_FILEPATH}/monitoring/model-quality\",\n",
    "    enable_cloudwatch_metrics=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ebb434e-89b1-4097-a397-acc2a6d01247",
   "metadata": {},
   "source": [
    "## Step 9 - Checking Monitoring Violations\n",
    "\n",
    "We can check the results of the monitoring job by looking at whether it generated any violations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "126bfac0-09a2-44a8-a6e3-b43abe9d9983",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'MonitoringScheduleArn': 'arn:aws:sagemaker:us-east-1:325223348818:monitoring-schedule/penguins-model-monitoring-schedule',\n",
       " 'MonitoringScheduleName': 'penguins-model-monitoring-schedule',\n",
       " 'MonitoringScheduleStatus': 'Scheduled',\n",
       " 'MonitoringType': 'ModelQuality',\n",
       " 'CreationTime': datetime.datetime(2023, 6, 28, 12, 40, 7, 350000, tzinfo=tzlocal()),\n",
       " 'LastModifiedTime': datetime.datetime(2023, 6, 28, 13, 32, 55, 997000, tzinfo=tzlocal()),\n",
       " 'MonitoringScheduleConfig': {'ScheduleConfig': {'ScheduleExpression': 'cron(0 * ? * * *)'},\n",
       "  'MonitoringJobDefinitionName': 'model-quality-job-definition-2023-06-28-12-40-06-773',\n",
       "  'MonitoringType': 'ModelQuality'},\n",
       " 'EndpointName': 'penguins-endpoint',\n",
       " 'LastMonitoringExecutionSummary': {'MonitoringScheduleName': 'penguins-model-monitoring-schedule',\n",
       "  'ScheduledTime': datetime.datetime(2023, 6, 28, 13, 0, tzinfo=tzlocal()),\n",
       "  'CreationTime': datetime.datetime(2023, 6, 28, 13, 8, 8, 325000, tzinfo=tzlocal()),\n",
       "  'LastModifiedTime': datetime.datetime(2023, 6, 28, 13, 32, 55, 973000, tzinfo=tzlocal()),\n",
       "  'MonitoringExecutionStatus': 'CompletedWithViolations',\n",
       "  'ProcessingJobArn': 'arn:aws:sagemaker:us-east-1:325223348818:processing-job/model-quality-monitoring-202306281300-3abdef542610bc2a9d721bd5',\n",
       "  'EndpointName': 'penguins-endpoint'},\n",
       " 'ResponseMetadata': {'RequestId': '35f8ed07-228b-4dad-ada0-a17e1b0bc510',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '35f8ed07-228b-4dad-ada0-a17e1b0bc510',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '986',\n",
       "   'date': 'Wed, 28 Jun 2023 13:39:37 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "description = model_monitor.describe_schedule()\n",
    "description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "4ba72770-0540-4019-88e6-1e8e7c783f89",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status: CompletedWithViolations\n",
      "[\n",
      "  {\n",
      "    \"constraint_check_type\": \"LessThanThreshold\",\n",
      "    \"description\": \"Metric weightedF2 with 0.35210111607474637 +/- 2.489284148228981E-5 was LessThanThreshold '0.9807900477958932'\",\n",
      "    \"metric_name\": \"weightedF2\"\n",
      "  },\n",
      "  {\n",
      "    \"constraint_check_type\": \"LessThanThreshold\",\n",
      "    \"description\": \"Metric accuracy with 0.35944503735325506 +/- 1.8368026615194536E-5 was LessThanThreshold '0.9807692307692307'\",\n",
      "    \"metric_name\": \"accuracy\"\n",
      "  },\n",
      "  {\n",
      "    \"constraint_check_type\": \"LessThanThreshold\",\n",
      "    \"description\": \"Metric weightedRecall with 0.359445037353255 +/- 1.8368026615206323E-5 was LessThanThreshold '0.9807692307692308'\",\n",
      "    \"metric_name\": \"weightedRecall\"\n",
      "  },\n",
      "  {\n",
      "    \"constraint_check_type\": \"LessThanThreshold\",\n",
      "    \"description\": \"Metric weightedPrecision with 0.35481246824881185 +/- 3.635184951719884E-5 was LessThanThreshold '0.9835164835164835'\",\n",
      "    \"metric_name\": \"weightedPrecision\"\n",
      "  },\n",
      "  {\n",
      "    \"constraint_check_type\": \"LessThanThreshold\",\n",
      "    \"description\": \"Metric weightedF1 with 0.3465051396530874 +/- 3.2275683236335196E-5 was LessThanThreshold '0.9813459031190452'\",\n",
      "    \"metric_name\": \"weightedF1\"\n",
      "  }\n",
      "]\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/13/3949.jsonl...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/13/4020.jsonl...\n",
      "Generating 334 predictions...\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/13/4050.jsonl...\n"
     ]
    }
   ],
   "source": [
    "status = description[\"LastMonitoringExecutionSummary\"][\"MonitoringExecutionStatus\"]\n",
    "print(f\"Status: {status}\")\n",
    "\n",
    "if status == \"CompletedWithViolations\":\n",
    "    processing_job_arn = description[\"LastMonitoringExecutionSummary\"][\"ProcessingJobArn\"]\n",
    "    execution = MonitoringExecution.from_processing_arn(sagemaker_session=sagemaker_session, processing_job_arn=processing_job_arn)\n",
    "    execution_destination = execution.output.destination\n",
    "    \n",
    "    violations_filepath = os.path.join(execution_destination, \"constraint_violations.json\")\n",
    "    violations = json.loads(S3Downloader.read_file(violations_filepath))[\"violations\"]\n",
    "    \n",
    "    print(json.dumps(violations, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41ad4f27-8ddb-44c8-b472-08fc69eab34e",
   "metadata": {},
   "source": [
    "## Step 10 - Cleaning up\n",
    "\n",
    "Let's stop the monitoring job by deleting the monitoring schedule we created before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "id": "7b1ada75-1606-4980-8fd5-620dfe9c6608",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Deleting Monitoring Schedule with name: penguins-model-monitoring-schedule\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/28/13/4150.jsonl...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.model_monitor.model_monitoring:Deleting Model Quality Job Definition with name: model-quality-job-definition-2023-06-28-12-40-06-773\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Monitoring schedule deleted.\n"
     ]
    }
   ],
   "source": [
    "delete_monitoring_schedule(model_monitor)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b87f0425-f0a4-4530-851b-7f665c55106b",
   "metadata": {},
   "source": [
    "We also need to stop the threads generating predictions and ground truth data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "id": "f265d3ab-384e-4e49-ab94-e7f87dea8f9e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "stop_traffic_thread.set()\n",
    "stop_ground_truth_thread.set()\n",
    "\n",
    "traffic_thread.join()\n",
    "groundtruth_thread.join()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "563fb066-a0a8-47dd-bbff-ca47d5ee967c",
   "metadata": {},
   "source": [
    "Finally, let's delete the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "id": "12ec02ec-6de1-4973-b9a0-525c81a5a56e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Deleting endpoint configuration with name: penguins-endpoint-config-0628100822\n",
      "INFO:sagemaker:Deleting endpoint with name: penguins-endpoint\n"
     ]
    }
   ],
   "source": [
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a0335ed-643d-4755-a1b2-ce129a8d7730",
   "metadata": {},
   "source": [
    "# Final Clean Up\n",
    "\n",
    "Here we can do a more deep clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "4cda93bb-29f8-47ad-a25f-f7b21df90325",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def delete_pipeline(pipeline):\n",
    "    if pipeline:\n",
    "        pipeline.delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b278e6f6-4001-4865-9ddb-6535c5dff961",
   "metadata": {},
   "outputs": [],
   "source": [
    "delete_pipeline(session1_pipeline)\n",
    "delete_pipeline(session2_pipeline)\n",
    "delete_pipeline(session3_pipeline)\n",
    "delete_pipeline(session4_pipeline)\n",
    "delete_pipeline(session5_pipeline)\n",
    "delete_pipeline(session6_pipeline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "id": "a8704c94-7a42-45f9-b006-74832f07cbc9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/15\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/14\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/13\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/12\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/11\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/10\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/9\n",
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/12/16/5021.jsonl...\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/8\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/7\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/6\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/5\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/4\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/3\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/2\n",
      "Deleting arn:aws:sagemaker:us-east-1:325223348818:model-package/penguins/1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'ResponseMetadata': {'RequestId': '1bcee64a-fbec-45d0-a996-5f07c99f0c15',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '1bcee64a-fbec-45d0-a996-5f07c99f0c15',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '0',\n",
       "   'date': 'Mon, 12 Jun 2023 16:50:33 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading ground truth data to s3://mlschool/penguins/monitoring/groundtruth/2023/06/12/16/5051.jsonl...\n"
     ]
    }
   ],
   "source": [
    "# Let's delete every model we registered under our model package group\n",
    "for mp in sagemaker_client.list_model_packages(ModelPackageGroupName=model_package_group_name)[\"ModelPackageSummaryList\"]:\n",
    "    print(f\"Deleting {mp['ModelPackageArn']}\")\n",
    "    sagemaker_client.delete_model_package(ModelPackageName=mp[\"ModelPackageArn\"])\n",
    "\n",
    "# We can now delete the model package group.    \n",
    "sagemaker_client.delete_model_package_group(ModelPackageGroupName=model_package_group_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "119eb67b-cd5e-458e-845a-6a07e512cf67",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (TensorFlow 2.6 Python 3.8 CPU Optimized)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/tensorflow-2.6-cpu-py38-ubuntu20.04-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}