08_deploy/02_deploy_autoscale_llama2_sagemaker_endpoint_deepspeed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a729a7fc-a1b6-4832-9155-d240ccd8ecc0",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Deploy LLama2 70b Model with high performance on SageMaker using Sagemaker LMI and Rolling batch\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2958bcf-767f-4cd7-826e-963f91565876",
   "metadata": {
    "tags": []
   },
   "source": [
    "\n",
    "In this notebook, we explore how to host a LLama2 large language model with FP16 precision on SageMaker using the DeepSpeed. We use DJLServing as the model serving solution in this example that is bundled in the LMI container. DJLServing is a high-performance universal model serving solution powered by the Deep Java Library (DJL) that is programming language agnostic. To learn more about DJL and DJLServing, you can refer to our recent blog post (https://aws.amazon.com/blogs/machine-learning/deploy-bloom-176b-and-opt-30b-on-amazon-sagemaker-with-large-model-inference-deep-learning-containers-and-deepspeed/).\n",
    "\n",
    "\n",
    "Model parallelism can help deploy large models that would normally be too large for a single GPU. With model parallelism, we partition and distribute a model across multiple GPUs. Each GPU holds a different part of the model, resolving the memory capacity issue for the largest deep learning models with billions of parameters. \n",
    "\n",
    "SageMaker has rolled out DeepSpeed container which now provides users with the ability to leverage the managed serving capabilities and help to provide the un-differentiated heavy lifting.\n",
    "\n",
    "In this notebook, we deploy https://huggingface.co/TheBloke/Llama-2-70b-fp16 model on a ml.g5.48xlarge instance. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7532764e-76e5-4f2b-8393-0cebd99056c8",
   "metadata": {},
   "source": [
    "# Licence agreement\n",
    " - View license information https://huggingface.co/meta-llama before using the model.\n",
    " - This notebook is a sample notebook and not intended for production use. Please refer to the licence at https://github.com/aws/mit-0. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8dd6a36f-ee95-453d-a127-c8a7de6a026d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sagemaker in /opt/conda/lib/python3.10/site-packages (2.203.1)\n",
      "Requirement already satisfied: boto3 in /opt/conda/lib/python3.10/site-packages (1.33.9)\n",
      "Collecting boto3\n",
      "  Downloading boto3-1.34.19-py3-none-any.whl.metadata (6.6 kB)\n",
      "Requirement already satisfied: attrs<24,>=23.1.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker) (23.1.0)\n",
      "Requirement already satisfied: cloudpickle==2.2.1 in /opt/conda/lib/python3.10/site-packages (from sagemaker) (2.2.1)\n",
      "Requirement already satisfied: google-pasta in /opt/conda/lib/python3.10/site-packages (from sagemaker) (0.2.0)\n",
      "Requirement already satisfied: numpy<2.0,>=1.9.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker) (1.26.2)\n",
      "Requirement already satisfied: protobuf<5.0,>=3.12 in /opt/conda/lib/python3.10/site-packages (from sagemaker) (4.25.1)\n",
      "Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /opt/conda/lib/python3.10/site-packages (from sagemaker) (1.0.1)\n",
      "Requirement already satisfied: importlib-metadata<7.0,>=1.4.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker) (4.11.3)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker) (21.3)\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.10/site-packages (from sagemaker) (2.1.3)\n",
      "Requirement already satisfied: pathos in /opt/conda/lib/python3.10/site-packages (from sagemaker) (0.3.1)\n",
      "Requirement already satisfied: schema in /opt/conda/lib/python3.10/site-packages (from sagemaker) (0.7.5)\n",
      "Requirement already satisfied: PyYAML~=6.0 in /opt/conda/lib/python3.10/site-packages/PyYAML-6.0-py3.10-linux-x86_64.egg (from sagemaker) (6.0)\n",
      "Requirement already satisfied: jsonschema in /opt/conda/lib/python3.10/site-packages (from sagemaker) (4.20.0)\n",
      "Requirement already satisfied: platformdirs in /opt/conda/lib/python3.10/site-packages (from sagemaker) (2.5.2)\n",
      "Requirement already satisfied: tblib<3,>=1.7.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker) (1.7.0)\n",
      "Requirement already satisfied: urllib3<1.27 in /opt/conda/lib/python3.10/site-packages (from sagemaker) (1.26.18)\n",
      "Requirement already satisfied: uvicorn==0.22.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker) (0.22.0)\n",
      "Requirement already satisfied: fastapi==0.95.2 in /opt/conda/lib/python3.10/site-packages (from sagemaker) (0.95.2)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.10/site-packages (from sagemaker) (2.31.0)\n",
      "Requirement already satisfied: docker in /opt/conda/lib/python3.10/site-packages (from sagemaker) (6.1.3)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.10/site-packages (from sagemaker) (4.64.1)\n",
      "Requirement already satisfied: psutil in /opt/conda/lib/python3.10/site-packages (from sagemaker) (5.9.0)\n",
      "Requirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /opt/conda/lib/python3.10/site-packages (from fastapi==0.95.2->sagemaker) (1.10.13)\n",
      "Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /opt/conda/lib/python3.10/site-packages (from fastapi==0.95.2->sagemaker) (0.27.0)\n",
      "Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.10/site-packages (from uvicorn==0.22.0->sagemaker) (8.1.7)\n",
      "Requirement already satisfied: h11>=0.8 in /opt/conda/lib/python3.10/site-packages (from uvicorn==0.22.0->sagemaker) (0.14.0)\n",
      "Collecting botocore<1.35.0,>=1.34.19 (from boto3)\n",
      "  Downloading botocore-1.34.19-py3-none-any.whl.metadata (5.6 kB)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.10/site-packages (from boto3) (0.10.0)\n",
      "Collecting s3transfer<0.11.0,>=0.10.0 (from boto3)\n",
      "  Downloading s3transfer-0.10.0-py3-none-any.whl.metadata (1.7 kB)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/lib/python3.10/site-packages (from botocore<1.35.0,>=1.34.19->boto3) (2.8.2)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.10/site-packages (from importlib-metadata<7.0,>=1.4.0->sagemaker) (3.8.0)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.10/site-packages (from packaging>=20.0->sagemaker) (3.0.9)\n",
      "Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.10/site-packages (from docker->sagemaker) (0.58.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests->sagemaker) (2.0.4)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests->sagemaker) (3.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests->sagemaker) (2023.11.17)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.10/site-packages (from google-pasta->sagemaker) (1.16.0)\n",
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /opt/conda/lib/python3.10/site-packages (from jsonschema->sagemaker) (2023.11.2)\n",
      "Requirement already satisfied: referencing>=0.28.4 in /opt/conda/lib/python3.10/site-packages (from jsonschema->sagemaker) (0.31.1)\n",
      "Requirement already satisfied: rpds-py>=0.7.1 in /opt/conda/lib/python3.10/site-packages (from jsonschema->sagemaker) (0.13.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas->sagemaker) (2022.1)\n",
      "Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas->sagemaker) (2023.3)\n",
      "Requirement already satisfied: ppft>=1.7.6.7 in /opt/conda/lib/python3.10/site-packages (from pathos->sagemaker) (1.7.6.7)\n",
      "Requirement already satisfied: dill>=0.3.7 in /opt/conda/lib/python3.10/site-packages (from pathos->sagemaker) (0.3.7)\n",
      "Requirement already satisfied: pox>=0.3.3 in /opt/conda/lib/python3.10/site-packages (from pathos->sagemaker) (0.3.3)\n",
      "Requirement already satisfied: multiprocess>=0.70.15 in /opt/conda/lib/python3.10/site-packages (from pathos->sagemaker) (0.70.15)\n",
      "Requirement already satisfied: contextlib2>=0.5.5 in /opt/conda/lib/python3.10/site-packages (from schema->sagemaker) (21.6.0)\n",
      "Requirement already satisfied: typing-extensions>=4.2.0 in /opt/conda/lib/python3.10/site-packages (from pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2->fastapi==0.95.2->sagemaker) (4.3.0)\n",
      "Requirement already satisfied: anyio<5,>=3.4.0 in /opt/conda/lib/python3.10/site-packages (from starlette<0.28.0,>=0.27.0->fastapi==0.95.2->sagemaker) (3.5.0)\n",
      "Requirement already satisfied: sniffio>=1.1 in /opt/conda/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi==0.95.2->sagemaker) (1.2.0)\n",
      "Downloading boto3-1.34.19-py3-none-any.whl (139 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m139.3/139.3 kB\u001b[0m \u001b[31m1.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading botocore-1.34.19-py3-none-any.whl (11.9 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.9/11.9 MB\u001b[0m \u001b[31m41.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m0:01\u001b[0m\n",
      "\u001b[?25hDownloading s3transfer-0.10.0-py3-none-any.whl (82 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m82.1/82.1 kB\u001b[0m \u001b[31m1.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m0:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: botocore, s3transfer, boto3\n",
      "  Attempting uninstall: botocore\n",
      "    Found existing installation: botocore 1.33.9\n",
      "    Uninstalling botocore-1.33.9:\n",
      "      Successfully uninstalled botocore-1.33.9\n",
      "  Attempting uninstall: s3transfer\n",
      "    Found existing installation: s3transfer 0.8.2\n",
      "    Uninstalling s3transfer-0.8.2:\n",
      "      Successfully uninstalled s3transfer-0.8.2\n",
      "  Attempting uninstall: boto3\n",
      "    Found existing installation: boto3 1.33.9\n",
      "    Uninstalling boto3-1.33.9:\n",
      "      Successfully uninstalled boto3-1.33.9\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "awscli 1.31.9 requires botocore==1.33.9, but you have botocore 1.34.19 which is incompatible.\n",
      "awscli 1.31.9 requires s3transfer<0.9.0,>=0.8.0, but you have s3transfer 0.10.0 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed boto3-1.34.19 botocore-1.34.19 s3transfer-0.10.0\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install sagemaker boto3 --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cf0c89f4-679c-4557-b95d-1d954c15a020",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /root/.config/sagemaker/config.yaml\n"
     ]
    }
   ],
   "source": [
    "import sagemaker\n",
    "import jinja2\n",
    "from sagemaker import image_uris\n",
    "import boto3\n",
    "import os\n",
    "import time\n",
    "import json\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "561b79ba-7354-4d40-87ce-dc5813092576",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "role = sagemaker.get_execution_role()  # execution role for the endpoint\n",
    "sess = sagemaker.session.Session()  # sagemaker session for interacting with different AWS APIs\n",
    "bucket = sess.default_bucket()  # bucket to house artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "f68b5181-d018-4564-9762-fa8770a9672f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_bucket = sess.default_bucket()  # bucket to house model artifacts\n",
    "s3_code_prefix = \"hf-large-model-djl/meta-llama/Llama-2-70b-fp16/code\"  # folder within bucket where code artifact will go\n",
    "\n",
    "s3_model_prefix = \"hf-large-model-djl/meta-llama/Llama-2-70b-fp16/model\"  # folder within bucket where model artifact will go\n",
    "region = sess._region_name\n",
    "account_id = sess.account_id()\n",
    "\n",
    "s3_client = boto3.client(\"s3\")\n",
    "sagemaker_client = boto3.client(\"sagemaker\")\n",
    "sagemaker_runtime_client = boto3.client(\"sagemaker-runtime\")\n",
    "\n",
    "jinja_env = jinja2.Environment()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d9697ac-e79b-4159-8c44-ffb6bcd8b7da",
   "metadata": {},
   "source": [
    "### Define a variable to contain the s3url of the location that has the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "76bc629a-c817-4bd8-be42-a4f30d0b075d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a variable to contain the s3url of the location that has the model. For demo purpose, we use Llama-2-70b-fp16 model artifacts from our S3 bucket\n",
    "pretrained_model_location = f\"s3://sagemaker-example-files-prod-{region}/models/llama-2/fp16/70B/\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9913de41-193a-4bfb-b42e-581c7677d0ed",
   "metadata": {},
   "source": [
    "## Create SageMaker compatible Model artifact,  upload Model to S3 and bring your own inference script.\n",
    "\n",
    "SageMaker Large Model Inference containers can be used to host models without providing your own inference code. This is extremely useful when there is no custom pre-processing of the input data or postprocessing of the model's predictions.\n",
    "\n",
    "SageMaker needs the model artifacts to be in a Tarball format. In this example, we provide the following files - serving.properties.\n",
    "\n",
    "The tarball is in the following format:\n",
    "\n",
    "```\n",
    "code\n",
    "├──── \n",
    "│   └── serving.properties\n",
    "```\n",
    "\n",
    "    serving.properties is the configuration file that can be used to configure the model server.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b50f5d44-6388-49fa-bc87-2e82bed226f3",
   "metadata": {},
   "source": [
    "#### Create serving.properties \n",
    "This is a configuration file to indicate to DJL Serving which model parallelization and inference optimization libraries you would like to use. Depending on your need, you can set the appropriate configuration.\n",
    "\n",
    "Here is a list of settings that we use in this configuration file -\n",
    "\n",
    "    engine: The engine for DJL to use. In this case, we have set it to MPI.\n",
    "    option.model_id: The model id of a pretrained model hosted inside a model repository on huggingface.co (https://huggingface.co/models) or S3 path to the model artifacts. \n",
    "    option.tensor_parallel_degree: Set to the number of GPU devices over which Accelerate needs to partition the model. This parameter also controls the no of workers per model which will be started up when DJL serving runs. As an example if we have a 4 GPU machine and we are creating 4 partitions then we will have 1 worker per model to serve the requests.\n",
    "\n",
    "For more details on the configuration options and an exhaustive list, you can refer the documentation - https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-large-model-configuration.html.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c2691f5b-4e98-4f7a-887c-49c05bbf7a8e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!rm -rf code_llama2_70b_fp16\n",
    "!mkdir -p code_llama2_70b_fp16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "42b040a8-2180-46cd-aa5c-b1f6d50c2dcf",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing code_llama2_70b_fp16/serving.properties\n"
     ]
    }
   ],
   "source": [
    "%%writefile code_llama2_70b_fp16/serving.properties\n",
    "engine = MPI\n",
    "option.tensor_parallel_degree = 8\n",
    "option.rolling_batch = auto\n",
    "option.max_rolling_batch_size = 4\n",
    "option.model_loading_timeout = 3600\n",
    "option.model_id = {{model_id}}\n",
    "option.paged_attention = true\n",
    "option.trust_remote_code = true\n",
    "option.dtype = fp16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "996ebeb2-cfee-4e7b-af8f-3ccc811fa1eb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     1\t\u001b[36mengine\u001b[39;49;00m\u001b[37m \u001b[39;49;00m=\u001b[37m \u001b[39;49;00m\u001b[33mMPI\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "     2\t\u001b[36moption.tensor_parallel_degree\u001b[39;49;00m\u001b[37m \u001b[39;49;00m=\u001b[37m \u001b[39;49;00m\u001b[33m8\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "     3\t\u001b[36moption.rolling_batch\u001b[39;49;00m\u001b[37m \u001b[39;49;00m=\u001b[37m \u001b[39;49;00m\u001b[33mauto\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "     4\t\u001b[36moption.max_rolling_batch_size\u001b[39;49;00m\u001b[37m \u001b[39;49;00m=\u001b[37m \u001b[39;49;00m\u001b[33m4\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "     5\t\u001b[36moption.model_loading_timeout\u001b[39;49;00m\u001b[37m \u001b[39;49;00m=\u001b[37m \u001b[39;49;00m\u001b[33m3600\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "     6\t\u001b[36moption.model_id\u001b[39;49;00m\u001b[37m \u001b[39;49;00m=\u001b[37m \u001b[39;49;00m\u001b[33ms3://sagemaker-example-files-prod-us-east-1/models/llama-2/fp16/70B/\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "     7\t\u001b[36moption.paged_attention\u001b[39;49;00m\u001b[37m \u001b[39;49;00m=\u001b[37m \u001b[39;49;00m\u001b[33mtrue\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "     8\t\u001b[36moption.trust_remote_code\u001b[39;49;00m\u001b[37m \u001b[39;49;00m=\u001b[37m \u001b[39;49;00m\u001b[33mtrue\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "     9\t\u001b[36moption.dtype\u001b[39;49;00m\u001b[37m \u001b[39;49;00m=\u001b[37m \u001b[39;49;00m\u001b[33mfp16\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n"
     ]
    }
   ],
   "source": [
    "# we plug in the appropriate model location into our `serving.properties`\n",
    "template = jinja_env.from_string(Path(\"code_llama2_70b_fp16/serving.properties\").open().read())\n",
    "Path(\"code_llama2_70b_fp16/serving.properties\").open(\"w\").write(\n",
    "    template.render(model_id=pretrained_model_location)\n",
    ")\n",
    "!pygmentize code_llama2_70b_fp16/serving.properties | cat -n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec9db9c4-5023-4125-a413-7c5afa135218",
   "metadata": {},
   "source": [
    "**Image URI for the DJL container is being used here**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8dfabe0a-04f8-486d-94ab-7d6066680954",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image going to be used is ---- > 763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:0.23.0-deepspeed0.9.5-cu118\n"
     ]
    }
   ],
   "source": [
    "inference_image_uri = image_uris.retrieve(\n",
    "    framework=\"djl-deepspeed\", region=region, version=\"0.23.0\"\n",
    ")\n",
    "print(f\"Image going to be used is ---- > {inference_image_uri}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "905903de-a4b9-4f41-8cc2-564416ae5d5f",
   "metadata": {},
   "source": [
    "**Create the Tarball and then upload to S3 location**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c005aa2e-ec1a-4ccf-8f39-67bcbabd0309",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: cannot remove 'model.tar.gz': No such file or directory\n",
      "code_llama2_70b_fp16/\n",
      "code_llama2_70b_fp16/serving.properties\n"
     ]
    }
   ],
   "source": [
    "!rm model.tar.gz\n",
    "!tar czvf model.tar.gz code_llama2_70b_fp16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "bb83ba3b-2ea5-4297-8e85-f16dd4c7c13a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3_code_artifact = sess.upload_data(\"model.tar.gz\", bucket, s3_code_prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efa56b75-356a-4ebf-bebd-7150521c95e9",
   "metadata": {},
   "source": [
    "### To create the end point the steps are:\n",
    "\n",
    "1. Create the Model using the Image container and the Model Tarball uploaded earlier\n",
    "2. Create the endpoint config using the following key parameters\n",
    "\n",
    "    a) Instance Type is ml.g5.48xlarge \n",
    "    \n",
    "    b) ContainerStartupHealthCheckTimeoutInSeconds is 3600 to ensure health check starts after the model is ready    \n",
    "3. Create the end point using the endpoint config created    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddee8273-8009-4739-af67-79525b6882cd",
   "metadata": {},
   "source": [
    "#### Create the Model\n",
    "Use the image URI for the DJL container and the s3 location to which the tarball was uploaded.\n",
    "\n",
    "The container downloads the model into the `/tmp` space on the instance because SageMaker maps the `/tmp` to the Amazon Elastic Block Store (Amazon EBS) volume that is mounted when we specify the endpoint creation parameter VolumeSizeInGB. \n",
    "It leverages `s5cmd`(https://github.com/peak/s5cmd) which offers a very fast download speed and hence extremely useful when downloading large models.\n",
    "\n",
    "For instances like p4dn, which come pre-built with the volume instance, we can continue to leverage the `/tmp` on the container. The size of this mount is large enough to hold the model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "eac40dc4-c9c3-4af1-8d54-7b6742259266",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391\n",
      "Created Model: arn:aws:sagemaker:us-east-1:079002598131:model/llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "model_name = name_from_base(f\"Llama-2-70b-fp16-mpi\")\n",
    "print(model_name)\n",
    "\n",
    "create_model_response = sagemaker_client.create_model(\n",
    "    ModelName=model_name,\n",
    "    ExecutionRoleArn=role,\n",
    "    PrimaryContainer={\n",
    "        \"Image\": inference_image_uri,\n",
    "        \"ModelDataUrl\": s3_code_artifact,\n",
    "        \"Environment\": {\"MODEL_LOADING_TIMEOUT\": \"3600\"},\n",
    "    },\n",
    ")\n",
    "model_arn = create_model_response[\"ModelArn\"]\n",
    "\n",
    "print(f\"Created Model: {model_arn}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "332e4a25-233f-4e75-aabb-0986c6c48f77",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'EndpointConfigArn': 'arn:aws:sagemaker:us-east-1:079002598131:endpoint-config/llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391-config',\n",
       " 'ResponseMetadata': {'RequestId': '020920e1-7acb-4249-a892-b10ed85ad12b',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '020920e1-7acb-4249-a892-b10ed85ad12b',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '132',\n",
       "   'date': 'Tue, 16 Jan 2024 04:32:59 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "endpoint_config_name = f\"{model_name}-config\"\n",
    "endpoint_name = f\"{model_name}-endpoint\"\n",
    "\n",
    "endpoint_config_response = sagemaker_client.create_endpoint_config(\n",
    "    EndpointConfigName=endpoint_config_name,\n",
    "    ProductionVariants=[\n",
    "        {\n",
    "            \"VariantName\": \"variant1\",\n",
    "            \"ModelName\": model_name,\n",
    "            \"InstanceType\": \"ml.g5.48xlarge\",\n",
    "            \"InitialInstanceCount\": 1,\n",
    "            \"ModelDataDownloadTimeoutInSeconds\": 3600,\n",
    "            \"ContainerStartupHealthCheckTimeoutInSeconds\": 3600,\n",
    "        },\n",
    "    ],\n",
    ")\n",
    "endpoint_config_response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4b6a0ca5-3ee6-4793-83f4-d9565a7d06e5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created Endpoint: arn:aws:sagemaker:us-east-1:079002598131:endpoint/llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391-endpoint\n"
     ]
    }
   ],
   "source": [
    "create_endpoint_response = sagemaker_client.create_endpoint(\n",
    "    EndpointName=f\"{endpoint_name}\", EndpointConfigName=endpoint_config_name\n",
    ")\n",
    "print(f\"Created Endpoint: {create_endpoint_response['EndpointArn']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9bf39db4-1317-48c2-937e-8e2199cc7be4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_1713/4280955313.py:1: DeprecationWarning: Importing display from IPython.core.display is deprecated since IPython 7.14, please import from IPython display\n",
      "  from IPython.core.display import display, HTML\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/endpoints/Llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391-endpoint\">SageMaker REST Endpoint</a></b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(\n",
    "    HTML(\n",
    "        '<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/endpoints/{}\">SageMaker REST Endpoint</a></b>'.format(\n",
    "            region, endpoint_name\n",
    "        )\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58385f65-b699-47d1-8449-268152c5b4d6",
   "metadata": {},
   "source": [
    "### This step can take ~ 20 min or longer so please be patient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f893ecb0-e0fe-450a-8bb1-5709d07b6e9e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: InService\n",
      "Arn: arn:aws:sagemaker:us-east-1:079002598131:endpoint/llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391-endpoint\n",
      "Status: InService\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "resp = sagemaker_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "status = resp[\"EndpointStatus\"]\n",
    "print(\"Status: \" + status)\n",
    "\n",
    "while status == \"Creating\":\n",
    "    time.sleep(60)\n",
    "    resp = sagemaker_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "    status = resp[\"EndpointStatus\"]\n",
    "    print(\"Status: \" + status)\n",
    "\n",
    "print(\"Arn: \" + resp[\"EndpointArn\"])\n",
    "print(\"Status: \" + status)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d0ec2d7-6ef8-4383-ac50-6a4984af5c03",
   "metadata": {},
   "source": [
    "#### While you wait for the endpoint to be created, you can read more about:\n",
    "- [Deep Learning containers for large model inference](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-large-model-dlc.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68aed031-b68c-4e23-8fa7-24483ec189d1",
   "metadata": {},
   "source": [
    "#### Leverage the Boto3 to invoke the endpoint. \n",
    "\n",
    "This is a generative model so we pass in a Text as a prompt and Model will complete the sentence and return the results.\n",
    "\n",
    "You can pass a prompt as input to the model. This done by setting inputs to a prompt. The model then returns a result for each prompt. The text generation can be configured using appropriate parameters.\n",
    "These parameters need to be passed to the endpoint as a dictionary of kwargs. Refer this documentation - https://huggingface.co/docs/transformers/main/en/main_classes/text_generation#transformers.GenerationConfig for more details.\n",
    "\n",
    "The below code sample illustrates the invocation of the endpoint using a text prompt and also sets some parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb965d9a-1c1e-4b03-90a1-d5cf31fade99",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sagemaker_runtime_client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    Body=json.dumps(\n",
    "        {\n",
    "            \"inputs\": \"The diamondback terrapin was the first reptile to do what?\",\n",
    "            \"parameters\": {\n",
    "                \"do_sample\": True,\n",
    "                \"max_new_tokens\": 100,\n",
    "                \"min_new_tokens\": 100,\n",
    "                \"temperature\": 0.3,\n",
    "                \"watermark\": True,\n",
    "            },\n",
    "        }\n",
    "    ),\n",
    "    ContentType=\"application/json\",\n",
    ")[\"Body\"].read().decode(\"utf8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c40a75c-57cb-4c20-af3f-f701eb6094f5",
   "metadata": {},
   "source": [
    "# Autoscaling a SageMaker Endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "1b611c12-12b4-4c48-b657-834001b049c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "autoscale = boto3.Session().client(service_name=\"application-autoscaling\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "8d4a513a-040a-4fe4-91a5-087c9302d680",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ScalableTargetARN': 'arn:aws:application-autoscaling:us-east-1:079002598131:scalable-target/056ma51e31786bfc49aca03c66fe0feb28f9',\n",
       " 'ResponseMetadata': {'RequestId': '8aecb2e7-b22f-4ea0-8289-9f55d53b4856',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '8aecb2e7-b22f-4ea0-8289-9f55d53b4856',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '131',\n",
       "   'date': 'Tue, 16 Jan 2024 04:47:09 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autoscale.register_scalable_target(\n",
    "    ServiceNamespace=\"sagemaker\",\n",
    "    ResourceId=\"endpoint/\" + endpoint_name + \"/variant/variant1\",\n",
    "    ScalableDimension=\"sagemaker:variant:DesiredInstanceCount\",\n",
    "    MinCapacity=1,\n",
    "    MaxCapacity=2,\n",
    "    RoleARN=role,\n",
    "    SuspendedState={\n",
    "        \"DynamicScalingInSuspended\": False,\n",
    "        \"DynamicScalingOutSuspended\": False,\n",
    "        \"ScheduledScalingSuspended\": False,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "cf4efa49-2b26-4449-b0da-eed2f1ec3c52",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ScalableTargets': [{'ServiceNamespace': 'sagemaker',\n",
       "   'ResourceId': 'endpoint/Llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391-endpoint/variant/variant1',\n",
       "   'ScalableDimension': 'sagemaker:variant:DesiredInstanceCount',\n",
       "   'MinCapacity': 1,\n",
       "   'MaxCapacity': 2,\n",
       "   'RoleARN': 'arn:aws:iam::079002598131:role/aws-service-role/sagemaker.application-autoscaling.amazonaws.com/AWSServiceRoleForApplicationAutoScaling_SageMakerEndpoint',\n",
       "   'CreationTime': datetime.datetime(2024, 1, 16, 4, 47, 9, 459000, tzinfo=tzlocal()),\n",
       "   'SuspendedState': {'DynamicScalingInSuspended': False,\n",
       "    'DynamicScalingOutSuspended': False,\n",
       "    'ScheduledScalingSuspended': False},\n",
       "   'ScalableTargetARN': 'arn:aws:application-autoscaling:us-east-1:079002598131:scalable-target/056ma51e31786bfc49aca03c66fe0feb28f9'}],\n",
       " 'ResponseMetadata': {'RequestId': '53321fdb-1cb2-4876-aad9-bfa67eaf3753',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '53321fdb-1cb2-4876-aad9-bfa67eaf3753',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '692',\n",
       "   'date': 'Tue, 16 Jan 2024 04:47:11 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the target is available\n",
    "autoscale.describe_scalable_targets(\n",
    "    ServiceNamespace=\"sagemaker\",\n",
    "    MaxResults=100,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "dd41b726-51e1-4b11-9ece-2ecd22aae470",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'PolicyARN': 'arn:aws:autoscaling:us-east-1:079002598131:scalingPolicy:a51e3178-6bfc-49ac-a03c-66fe0feb28f9:resource/sagemaker/endpoint/Llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391-endpoint/variant/variant1:policyName/autoscale-policy-gpu-400-llama2-70b',\n",
       " 'Alarms': [{'AlarmName': 'TargetTracking-endpoint/Llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391-endpoint/variant/variant1-AlarmHigh-b34fa515-574f-4f65-84f3-627a58040aa9',\n",
       "   'AlarmARN': 'arn:aws:cloudwatch:us-east-1:079002598131:alarm:TargetTracking-endpoint/Llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391-endpoint/variant/variant1-AlarmHigh-b34fa515-574f-4f65-84f3-627a58040aa9'},\n",
       "  {'AlarmName': 'TargetTracking-endpoint/Llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391-endpoint/variant/variant1-AlarmLow-cb39c7ac-448f-4a05-bb7b-3333c57acc02',\n",
       "   'AlarmARN': 'arn:aws:cloudwatch:us-east-1:079002598131:alarm:TargetTracking-endpoint/Llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391-endpoint/variant/variant1-AlarmLow-cb39c7ac-448f-4a05-bb7b-3333c57acc02'}],\n",
       " 'ResponseMetadata': {'RequestId': '33a3d584-fb05-41a0-918c-ae0c0f56a620',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '33a3d584-fb05-41a0-918c-ae0c0f56a620',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '986',\n",
       "   'date': 'Tue, 16 Jan 2024 04:47:21 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autoscale.put_scaling_policy(\n",
    "    PolicyName=\"autoscale-policy-gpu-400-llama2-70b\",\n",
    "    ServiceNamespace=\"sagemaker\",\n",
    "    ResourceId=\"endpoint/\" + endpoint_name + \"/variant/variant1\",\n",
    "    ScalableDimension=\"sagemaker:variant:DesiredInstanceCount\",\n",
    "    PolicyType=\"TargetTrackingScaling\",\n",
    "    TargetTrackingScalingPolicyConfiguration={\n",
    "        \"TargetValue\": 400, # 400% of 800% total GPU utilization (8 GPUs)\n",
    "        \"CustomizedMetricSpecification\":\n",
    "        {\n",
    "            \"MetricName\": \"GPUUtilization\",\n",
    "            \"Namespace\": \"/aws/sagemaker/Endpoints\",\n",
    "            \"Dimensions\": [\n",
    "                {\"Name\": \"EndpointName\", \"Value\": endpoint_name},\n",
    "                {\"Name\": \"VariantName\", \"Value\": \"variant1\"}\n",
    "            ],\n",
    "            \"Statistic\": \"Average\",\n",
    "            \"Unit\": \"Percent\"\n",
    "        },\n",
    "        \"ScaleOutCooldown\": 60,\n",
    "        \"ScaleInCooldown\": 300,\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df7cdc49-c737-4c7f-ab32-4f2c5accdda8",
   "metadata": {},
   "source": [
    "# Trigger the Autoscaling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8f574b0-565d-4513-97ee-cf68948c6252",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for i in range(0, 100):\n",
    "    res = sagemaker_runtime_client.invoke_endpoint(\n",
    "        EndpointName=endpoint_name,\n",
    "        Body=json.dumps(\n",
    "            {\n",
    "                \"inputs\": \"The diamondback terrapin was the first reptile to do what?\",\n",
    "                \"parameters\": {\n",
    "                    \"do_sample\": True,\n",
    "                    \"max_new_tokens\": 100,\n",
    "                    \"min_new_tokens\": 100,\n",
    "                    \"temperature\": 0.3,\n",
    "                    \"watermark\": True,\n",
    "                },\n",
    "            }\n",
    "        ),\n",
    "        ContentType=\"application/json\",\n",
    "    )\n",
    "    print(f'{i}: {res[\"Body\"].read().decode(\"utf8\")}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "2d076097-f685-4590-bcff-b3741c049aec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ScalingActivities': [{'ActivityId': 'f623aaf1-a0c8-4662-a513-e7860a90153a',\n",
       "   'ServiceNamespace': 'sagemaker',\n",
       "   'ResourceId': 'endpoint/Llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391-endpoint/variant/variant1',\n",
       "   'ScalableDimension': 'sagemaker:variant:DesiredInstanceCount',\n",
       "   'Description': 'Setting desired instance count to 2.',\n",
       "   'Cause': 'monitor alarm TargetTracking-endpoint/Llama-2-70b-fp16-mpi-2024-01-16-04-32-58-391-endpoint/variant/variant1-AlarmHigh-b34fa515-574f-4f65-84f3-627a58040aa9 in state ALARM triggered policy autoscale-policy-gpu-400-llama2-70b',\n",
       "   'StartTime': datetime.datetime(2024, 1, 16, 4, 51, 28, 298000, tzinfo=tzlocal()),\n",
       "   'StatusCode': 'InProgress',\n",
       "   'StatusMessage': 'Successfully set desired instance count to 2. Waiting for change to be fulfilled by sagemaker.'}],\n",
       " 'ResponseMetadata': {'RequestId': 'ccba093d-51d6-4527-9f31-f53fe75411ee',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': 'ccba093d-51d6-4527-9f31-f53fe75411ee',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '719',\n",
       "   'date': 'Tue, 16 Jan 2024 04:52:39 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autoscale.describe_scaling_activities(\n",
    "    ServiceNamespace=\"sagemaker\",\n",
    "    ResourceId=\"endpoint/\" + endpoint_name + \"/variant/variant1\",\n",
    "    ScalableDimension=\"sagemaker:variant:DesiredInstanceCount\",\n",
    "    MaxResults=100\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38886de1-a30b-41e3-b99d-b4454e37760b",
   "metadata": {},
   "source": [
    "## Clean Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bb72ed7-ce42-480a-998f-809157fadb4e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # - Delete the end point\n",
    "# sagemaker_client.delete_endpoint(EndpointName=endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82bca632-7305-4dcb-bf96-ae76337ab53a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # - In case the end point failed we still want to delete the model\n",
    "# sagemaker_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)\n",
    "# sagemaker_client.delete_model(ModelName=model_name)"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}