src/toxic_comment_classification.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Toxic Comment Classification\n",
    "\n",
    "![SageMaker](https://img.shields.io/badge/SageMaker-%23FF9900.svg?style=for-the-badge&logo=amazon-aws&logoColor=white)\n",
    "\n",
    "This notebook is a part of [ToChiquinho](https://dougtrajano.github.io/ToChiquinho/) project, which trains a model to classify comments as toxic or not toxic. We used the [OLID-BR](https://dougtrajano.github.io/olid-br/) dataset as the training data.\n",
    "\n",
    "The model is trained using [Amazon SageMaker](https://aws.amazon.com/sagemaker/).\n",
    "\n",
    "- [Setup](#Setup)\n",
    "- [Prepare the data](#Prepare-the-data)\n",
    "  - [Uploading the data to S3](#Uploading-the-data-to-S3)\n",
    "- [Training process](#Training-process)\n",
    "  - [Define the estimator](#Define-the-estimator)\n",
    "  - [Hyperparameter tuning](#Hyperparameter-tuning)\n",
    "  - [Training best model](#Training-best-model)\n",
    "- [Documentation](#Documentation)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "In this section, we will import the necessary libraries and set up the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"../.env\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NotebookArguments(num_train_epochs=30, early_stopping_patience=5, batch_size=8, validation_split=0.2, seed=1993, mlflow_experiment_name='toxic-comment-classification', mlflow_tags='{\"project\": \"ToChiquinho\", \"dataset\": \"OLID-BR\", \"model_type\": \"bert\", \"problem_type\": \"binary_classification\"}', sagemaker_image_uri='215993976552.dkr.ecr.us-east-1.amazonaws.com/sagemaker-transformers:1.12.0-gpu-py38')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "from ml.arguments import NotebookArguments\n",
    "from ml.utils import remove_checkpoints\n",
    "\n",
    "params = NotebookArguments(\n",
    "    num_train_epochs=30,\n",
    "    early_stopping_patience=5,\n",
    "    mlflow_experiment_name=\"toxic-comment-classification\",\n",
    "    mlflow_tags={\n",
    "        \"project\": \"ToChiquinho\",\n",
    "        \"dataset\": \"OLID-BR\",\n",
    "        \"model_type\": \"bert\",\n",
    "        \"problem_type\": \"binary_classification\"\n",
    "    },\n",
    "    sagemaker_execution_role_arn=os.environ.get(\"SAGEMAKER_EXECUTION_ROLE_ARN\"),\n",
    "    sagemaker_tuning_job_name = \"pytorch-training-230202-1408\",\n",
    "    aws_profile_name=os.environ.get(\"AWS_PROFILE\")\n",
    ")\n",
    "\n",
    "params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "\n",
    "sagemaker_session = sagemaker.Session(\n",
    "    boto_session=boto3.Session(profile_name=params.aws_profile_name)\n",
    ")\n",
    "\n",
    "bucket_name = sagemaker_session.default_bucket()\n",
    "prefix = f\"ToChiquinho/{params.mlflow_experiment_name}\"\n",
    "\n",
    "if params.sagemaker_execution_role_arn is None:\n",
    "    params.sagemaker_execution_role_arn = sagemaker.get_execution_role(sagemaker_session)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the data\n",
    "\n",
    "In this section, we will prepare the data to be used in the training process.\n",
    "\n",
    "For this task, we will use OLID-BR dataset from [HuggingFace Datasets](https://huggingface.co/datasets/olidbr) and non-offensive comments from other Brazilian Portuguese datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration dougtrajano--olid-br-f83aad8215e23434\n",
      "Found cached dataset parquet (C:/Users/trajano/.cache/huggingface/datasets/dougtrajano___parquet/dougtrajano--olid-br-f83aad8215e23434/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "672416b591564f5fa67c57322a350d27",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    test: Dataset({\n",
       "        features: ['id', 'text', 'is_offensive', 'is_targeted', 'targeted_type', 'toxic_spans', 'health', 'ideology', 'insult', 'lgbtqphobia', 'other_lifestyle', 'physical_aspects', 'profanity_obscene', 'racism', 'religious_intolerance', 'sexism', 'xenophobia'],\n",
       "        num_rows: 1589\n",
       "    })\n",
       "    train: Dataset({\n",
       "        features: ['id', 'text', 'is_offensive', 'is_targeted', 'targeted_type', 'toxic_spans', 'health', 'ideology', 'insult', 'lgbtqphobia', 'other_lifestyle', 'physical_aspects', 'profanity_obscene', 'racism', 'religious_intolerance', 'sexism', 'xenophobia'],\n",
       "        num_rows: 4765\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "dataset = load_dataset(\"dougtrajano/olid-br\")\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import boto3\n",
    "\n",
    "session = boto3.Session(profile_name=params.aws_profile_name)\n",
    "s3 = session.resource(\"s3\")\n",
    "bucket = s3.Bucket(\"toxicity-detection-dev-bucket\")\n",
    "obj = bucket.Object(\"processed/related_datasets/is_offensive.json\")\n",
    "related_datasets = json.loads(obj.get()[\"Body\"].read().decode(\"utf-8\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size: 6493\n",
      "Test size: 1624\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "non_offensive_comments = [\n",
    "    i for i in related_datasets if i[\"is_offensive\"] == \"NOT\"\n",
    "]\n",
    "\n",
    "X = [i[\"text\"] for i in non_offensive_comments]\n",
    "y = [i[\"is_offensive\"] for i in non_offensive_comments]\n",
    "\n",
    "# Reduce dataset size\n",
    "X, _, y, _ = train_test_split(\n",
    "    X, y,\n",
    "    test_size=0.6,\n",
    "    random_state=params.seed,\n",
    "    stratify=y\n",
    ")\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y,\n",
    "    test_size=0.2,\n",
    "    random_state=params.seed,\n",
    "    stratify=y\n",
    ")\n",
    "\n",
    "print(f\"Train size: {len(X_train)}\")\n",
    "print(f\"Test size: {len(X_test)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size: 11258\n",
      "Test size: 3213\n"
     ]
    }
   ],
   "source": [
    "import datasets\n",
    "\n",
    "X_train = X_train + dataset[\"train\"][\"text\"]\n",
    "y_train = y_train + dataset[\"train\"][\"is_offensive\"]\n",
    "X_test = X_test + dataset[\"test\"][\"text\"]\n",
    "y_test = y_test + dataset[\"test\"][\"is_offensive\"]\n",
    "\n",
    "print(f\"Train size: {len(X_train)}\")\n",
    "print(f\"Test size: {len(X_test)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['text', 'is_offensive'],\n",
       "        num_rows: 11258\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['text', 'is_offensive'],\n",
       "        num_rows: 3213\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = datasets.DatasetDict(\n",
    "    {\n",
    "        \"train\": datasets.Dataset.from_dict(\n",
    "            {\n",
    "                \"text\": X_train,\n",
    "                \"is_offensive\": y_train\n",
    "            }\n",
    "        ),\n",
    "        \"test\": datasets.Dataset.from_dict(\n",
    "            {\n",
    "                \"text\": X_test,\n",
    "                \"is_offensive\": y_test\n",
    "            }\n",
    "        )\n",
    "    }\n",
    ")\n",
    "\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f95da4959084129a4a9ce2b4ee798eb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/11258 [00:00<?, ?ex/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ea18418fe1d74859b3060a8025c03182",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3213 [00:00<?, ?ex/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 9006\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 3213\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 2252\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import ast\n",
    "import datasets\n",
    "from typing import Union\n",
    "\n",
    "def prepare_dataset(\n",
    "    dataset: Union[datasets.Dataset, datasets.DatasetDict],\n",
    "    test_size: float = 0.2,\n",
    "    seed: int = 42\n",
    ") -> Union[datasets.Dataset, datasets.DatasetDict]:    \n",
    "    # Filter only \"text\" and \"is_offensive\" columns\n",
    "    dataset = dataset.remove_columns(\n",
    "        [\n",
    "            col for col in dataset[\"train\"].column_names if col not in [\n",
    "                \"text\",\n",
    "                \"is_offensive\"\n",
    "            ]\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    dataset = dataset.rename_column(\"is_offensive\", \"label\")\n",
    "\n",
    "    # Replace \"OFF\": 1 and \"NOT\": 0\n",
    "    def replace_labels(example):\n",
    "        if example[\"label\"] == \"OFF\":\n",
    "            example[\"label\"] = 1\n",
    "        elif example[\"label\"] == \"NOT\":\n",
    "            example[\"label\"] = 0\n",
    "        else:\n",
    "            raise ValueError(f\"Invalid label: {example['label']}\")\n",
    "        return example\n",
    "\n",
    "    dataset = dataset.map(replace_labels)\n",
    "\n",
    "    train_dataset = dataset[\"train\"].train_test_split(\n",
    "        test_size=test_size,\n",
    "        shuffle=True,\n",
    "        seed=seed\n",
    "    )\n",
    "\n",
    "    dataset[\"train\"] = train_dataset[\"train\"]\n",
    "    dataset[\"validation\"] = train_dataset[\"test\"]\n",
    "\n",
    "    return dataset\n",
    "\n",
    "\n",
    "dataset = prepare_dataset(\n",
    "    dataset,\n",
    "    test_size=params.validation_split,\n",
    "    seed=params.seed\n",
    ")\n",
    "\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3UAAAE/CAYAAADhUuoDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAu40lEQVR4nO3de1hVdaLG8XfDRkvxEh02Ot66aPb4eKsobwWnwQQVvKDNQTiSo3axEdMSBxXj4CXNSIoUm8zuWDqdYhfhNrWTXVBDcjTPaFOmKGpc0lRQLm7X+aPTPsNRgwz23ku+n+eZ53H/1lp7/37xPO/wstZey2IYhiEAAAAAgCn5eHoCAAAAAIDLR6kDAAAAABOj1AEAAACAiVHqAAAAAMDEKHUAAAAAYGKUOgAAAAAwMUod3GLixIk6fvx4vff/6quvNG3atEacEQD8n1+bUf8sOTlZe/bsaeAZAWiKkpKStHr1aknSyJEjderUqQv2Wb16tZKSkup8r3/Oprlz5yovL69hJwuvQqmDW3z++ee/av9evXopIyOjkWYDALX92oz6Z3l5eeKRrwAamt1uV+vWrS/7+H/OpkWLFmngwIENNTV4IaunJ4Ar3+zZsyVJ9913n7799luFh4fr66+/1qOPPiqr1aq//OUvqq6u1vHjxzVq1ChNnz5d27dv14IFC5STk6OkpCT5+/vr66+/1vfff68bbrhBy5YtU8uWLT28MgBXgn/OqOeff15PPPGEjh07ppqaGg0fPlwPPfSQzp07pwULFujLL7+Un5+fOnbsqMWLF+uFF15QSUmJZs6cqaVLl6pPnz4eXg0Ab/LYY4+pR48emjRpkiTpzTff1NatW2Wz2bRr1y5VVFTIMAwtXLhQt912W61ju3fvrq1bt6pVq1ZauHCh8vLydO211+raa69Vq1atJEl/+9vf9NRTT6m6ulqlpaUaOHCgnnjiCaWnp9fKprS0NMXFxSkiIkKbNm3S8uXL5XQ65e/vr9mzZ6t379567rnndOTIEZWWlurIkSMKCAhQenq6goKC3P7fDb8eZ+rQ6BYvXixJevXVV9W+fXt169ZN69ev1+DBg/XSSy9pyZIleuedd7R27Vq98MILF70Eas+ePVq9erVyc3NVUlIih8Ph7mUAuEL9c0bNnj1bY8aM0TvvvKO3335beXl5ys3N1d/+9jd98cUXeu+99/TOO++oU6dO+vrrrzVjxgzZbDalpaVR6ABc4N5771V2drbr9TvvvKPu3burpKREa9euVW5urkaPHq1Vq1Zd8j3WrFmjgwcP6oMPPtBLL72kY8eOuba99tprmjZtmv7617/qgw8+0EcffaQ9e/ZcMpv279+vlJQUPffcc3r//fc1bdo0PfzwwyovL5ck7dixQ88++6wcDodat26ttWvXNvx/FDQKztTB7YKDgyVJFotFzz//vD7++GPl5ORo//79MgxDZ8+eveCYu+66S82aNZMk3XTTTTp58qRb5wzgynf27Fnl5+fr5MmTevbZZyVJZ86c0b59+3TnnXfK19dX9957r+68806Fh4erd+/eHp4xAG/Xr18/VVVV6auvvtLVV1+t48eP6+GHH9aBAwf01ltv6fDhw9q+ffsvXn20detWRUZGqlmzZmrWrJmioqL09ddfS5KWLFmiTz75RM8//7y+++47VVZW6syZM5d8r23btql///7q1KmTJGnAgAEKCAhwfffujjvukL+/vySpR48e/L5lIpQ6uF2LFi0k/fTL0ujRozV48GAFBwdrzJgx2rRp00W/m3LVVVe5/m2xWPj+CoAG93O2vPXWW7r66qslScePH1fz5s3VsmVL2e12ffnll9q2bZumT5+u+Ph4TZgwwbOTBuDVLBaLxo4dK7vdLj8/P40dO1ZbtmzRokWL9Mc//lFhYWG64YYb9N5779X7PX19fV3/jouL080336y77rpLQ4cO1a5du37xd6SLbTMMQ+fOnZPE71tmxuWXcAtfX19XYPyssLBQ5eXlmj59un7/+9/riy++UHV1tc6fP++hWQJoqnx9fWW1WtW3b1+9/PLLkqRTp05p3Lhx2rx5s/7rv/5LEyZM0C233KKEhASNGjVK+/btcx37//MNAH42evRoffTRR9qwYYOio6P1+eef6+6771ZsbKx69eqlTZs2yel0XvL4u+66S9nZ2aqqqlJVVZVyc3MlSSdPntSePXs0c+ZMDRkyRMXFxTp06JDr96iLZVP//v31+eef6/Dhw5J+Ogt47NgxLh+/AnCmDm5xzz33KDY2VhUVFa6x7t2761//9V81dOhQtW7dWp07d1bXrl1VWFjoutQSANzh54zKzMzU008/raioKFVXVysyMlIjRoyQ0+nUJ598osjISLVo0UJt2rTRggULJEmDBw/WjBkztHDhQt15550eXgkAbxMYGKgePXro3LlzCgoKUkxMjGbOnKmoqCj5+voqODhYH3744SX/qB0TE6NDhw4pMjJSbdu2VZcuXSRJbdq00QMPPKDRo0erbdu2uuaaa3TrrbeqsLBQAwYMqJVNP+vatatSUlI0depUOZ1OXXXVVXr++eddN16BeVkMzqsCAAAAgGlx+SUAAAAAmFi9St1HH32k6OhoRUREuE7h5uXlKSoqSkOGDFF6erpr371792rMmDEKDw/X3LlzXdfyHj161PV8jClTptS6DA8ALgfZBMBbkU8A3Mqow6FDh4w777zTOHbsmFFdXW2MGzfO+Pjjj43Q0FDj0KFDRk1NjTFx4kTj448/NgzDMIYPH27s3LnTMAzDmD17tpGVlWUYhmE88MADRk5OjmEYhrF8+XJj6dKldX00AFwS2QTAW5FPANytzjN1Gzdu1LBhw9SuXTv5+fkpPT1dV199tbp06aJOnTrJarUqKipKDodDR44cUWVlpfr27StJio6OlsPhUE1NjfLz8xUeHl5rHAAuF9kEwFuRTwDcrc67XxYWFsrPz0+TJk1SaWmp7r77bnXr1k2BgYGufWw2m4qLi1VSUlJrPDAwUMXFxTpx4oT8/f1ltVprjdfHuXPn9P3336tdu3au4wHA09kkkU8ALs7T+UQ2AU1PnWfqnE6ntm7dqqeeekrr1q3TV199paKiogv2u9QDCn9pvD6+//57hYWF6fvvv6/X/gCaBk9nk0Q+Abg4T+cT2QQ0PXX++eZf/uVfNGDAAAUEBEiSwsLC5HA4aj3NvqSkRDabTUFBQSorK3ONl5aWymazKSAgQOXl5XI6nfL19XWNA8DlIpsAeCvyCYC71Xmm7u6779Znn32mU6dOyel06tNPP1VERIQOHDigwsJCOZ1O5eTkKCQkRB06dFDz5s1VUFAgScrOzlZISIj8/PwUHBys3NzcWuMAcLnIJgDeinwC4G51nqnr06ePJk+erNjYWNXU1GjQoEEaN26cbrjhBiUkJKiqqkqhoaGKiIiQJKWlpSk5OVkVFRXq0aOH4uPjJUkpKSlKSkrSypUr1b59ey1btqxxVwbgikY2AfBW5BMAd7MYF7to24sUFRUpLCxMmzdvVseOHT09HQBwIZ8AeCOyCWh6uCWSGy1ZskQOh0Nt2rSRJF1//fV65plnFB0drcrKSvn5+UmSoqKiNHnyZMXExOjs2bOu4w8cOKA//OEPSk5O1qeffqr09HQ5nU75+Pjo0Ucf1V133eWRdQEAAADwHEqdG+3cuVPLli3Trbfe6ho7c+aMDh06pK1bt7pK3c/eeust1783b96sp59+Wo888ohOnz6tmTNn6o033lC3bt20b98+/fu//7s+/vhj+fv7u209AAAAADyvzhuloGFUV1fr73//u1566SWNGDFCCQkJOnr0qHbv3q0WLVrowQcfVFRUlJ544glVVlbWOvbHH39USkqKnnzySbVq1Uo1NTVKSUlRt27dJEldu3aVYRg6ceKEJ5YGAAAAwIModW5SXFys/v3769FHH5XdblefPn308MMPq7y8XP369VNGRobefvttHTt2TE8//XStY1etWqXQ0FD16tVLkhQQEKBhw4a5tmdkZOi6665Tp06d3LomAAAAAJ5HqXOTTp06adWqVbrhhhtksVg0adIkHTp0SN27d9dTTz0lf39/NW/eXA8++KA2bdrkOq6qqkrr1q3TQw89dMF7njt3TgsXLpTD4dBzzz3nzuUAAAAA8BKUOjfZt2+fsrOza40ZhqGCggLl5+fXGrNa/++rjp988oluvvnmC87CnTx5UpMmTdI//vEPrV27Vr/73e8adf4AAAAAvBOlzk18fHy0aNEiHT58WJK0Zs0ade/eXWfOnNGTTz6pyspKOZ1OvfLKK7Uurfziiy80YMCAWu9VXV2tiRMnqmPHjnrppZd0zTXXuHUtAAAAALwHd790k5tuuknJycmaMmWKnE6n2rVrp2XLlqldu3Y6fPiwRo8eLafTqX79+ulPf/qT67jCwkL17Nmz1ns5HA7t2bNH1dXVGjNmjGt86dKl6t69u9vWBAAAAMDzePg4AFwm8gmANyKbgKaHyy8BAAAAwMQodQAAAABgYpQ6AAAAADAxSh0AAAAAmBilDgAAAABM7IotddU1Tk9PAeLnAAAAADS2K/Y5dc38fBU7K8vT02jy1iyN8/QUAAAAgCvaFXumDgAAAACaAkodAAAAAJgYpQ4AAAAATIxSBwAAAAAmRqkDAAAAABOj1AEAAACAiVHqAAAAAMDEKHUAAAAAYGKUOgAAAAAwMUodAAAAAJgYpQ4AAAAATIxSBwAAAAAmRqkDAAAAABOj1AEAAACAiVHqAAAAAMDEKHUAAAAAYGKUOgAAAAAwMUodAAAAAJgYpQ4AAAAATIxSBwAAAAAmZq3PTvHx8frhhx9ktf60+/z583Xo0CGtXLlSNTU1mjBhguLi4iRJeXl5Wrx4saqqqjR06FDNmDFDkrR3714lJyervLxcwcHBSk1Ndb0fAFwOsgmAtyKfALhTnWfqDMPQd999J7vd7vpfu3btlJ6erjVr1shut2vt2rX69ttvVVlZqTlz5igzM1O5ubnas2ePtmzZIklKTEzUvHnztGHDBhmGoXXr1jX64gBcucgmAN6KfALgbnWWuu+++04Wi0X333+/RowYoTfeeEN5eXnq37+/2rZtqxYtWig8PFwOh0O7d+9Wly5d1KlTJ1mtVkVFRcnhcOjIkSOqrKxU3759JUnR0dFyOByNvTYAVzCyCYC3Ip8AuFudpe7UqVMaMGCAVqxYoVdeeUVvvfWWjh49qsDAQNc+NptNxcXFKikpqdd4YGCgiouLG3gpAJoSsgmAtyKfALhbnaXulltu0dKlS9WiRQsFBARo7NixysjIuGA/i8UiwzB+1TgAXC6yCYC3Ip8AuFudpW7Hjh3aunWr67VhGOrQoYPKyspcYyUlJbLZbAoKCqrXeGlpqWw2W0OtAUATRDYB8FbkEwB3q7PUnT59WkuXLlVVVZXKy8v17rvv6qmnntLWrVt1/PhxnT17Vh9++KFCQkLUp08fHThwQIWFhXI6ncrJyVFISIg6dOig5s2bq6CgQJKUnZ2tkJCQRl8cgCsX2QTAW5FPANytzvvi3n333dq1a5dGjRql8+fPKzY2VrfddptmzJih+Ph41dTUaOzYserdu7ckacmSJUpISFBVVZVCQ0MVEREhSUpLS1NycrIqKirUo0cPxcfHN+7KAFzRyCYA3op8AuBuFuNiF217kaKiIoWFhWnz5s3q2LHjrzo2dlZWI80K9bVmaZynpwA0mt+STwDQWMgmoOmp8/JLAAAAAID3otQBAAAAgIlR6gAAAADAxCh1AAAAAGBilDoAAAAAMDFKHQAAAACYGKUOAAAAAEyMUgcAAAAAJkapAwAAAAATo9QBAAAAgIlR6gAAAADAxCh1AAAAAGBilDoAAAAAMDFKHQAAAACYGKUOAAAAAEyMUgcAAAAAJkapAwAAAAATo9QBAAAAgIlR6gAAAADAxCh1AAAAAGBilDoAAAAAMDFKHQAAAACYGKUOAAAAAEyMUgcAAAAAJkapAwAAAAATo9QBAAAAgIlR6gAAAADAxCh1AAAAAGBilDoAAAAAMDFKHQAAAACYGKUOAAAAAEyMUgcAAAAAJkapAwAAAAATo9QBAAAAgIlR6gAAAADAxCh1AAAAAGBilDoAAAAAMLF6l7onn3xSSUlJkqS9e/dqzJgxCg8P19y5c3Xu3DlJ0tGjRxUXF6eIiAhNmTJFFRUVkqRTp07pgQce0NChQxUXF6fS0tJGWAqApop8AuCNyCYA7lKvUrd161a9++67rteJiYmaN2+eNmzYIMMwtG7dOklSamqqYmNj5XA41LNnT2VmZkqSnnnmGQUHB2v9+vW69957tWjRokZYCoCmiHwC4I3IJgDuVGep+/HHH5Wenq6HHnpIknTkyBFVVlaqb9++kqTo6Gg5HA7V1NQoPz9f4eHhtcYl6eOPP1ZUVJQkKTIyUp988olqamoaYz0AmhDyCYA3IpsAuFudpe7xxx/XjBkz1Lp1a0lSSUmJAgMDXdsDAwNVXFysEydOyN/fX1artdb4/z/GarXK399fx48fb/DFAGhayCcA3ohsAuBuv1jq/vrXv6p9+/YaMGCAa8wwjAv2s1gslxy/5Af7cI8WAJePfALgjcgmAJ5g/aWNubm5Ki0t1ciRI3Xy5EmdOXNGFotFZWVlrn1KS0tls9kUEBCg8vJyOZ1O+fr6usYlyWazqaysTO3atdO5c+dUXl6utm3bNurCAFzZyCcA3ohsAuAJv/gnn5dfflk5OTmy2+2aNm2afv/732vx4sVq3ry5CgoKJEnZ2dkKCQmRn5+fgoODlZubW2tckkJDQ5WdnS3pp7ALDg6Wn59fIy4LwJWOfALgjcgmAJ7wi2fqLiUtLU3JycmqqKhQjx49FB8fL0lKSUlRUlKSVq5cqfbt22vZsmWSpEceeURJSUkaPny4WrVqpbS0tIZbAQD8E/IJgDcimwA0JotxsQu6vUhRUZHCwsK0efNmdezY8VcdGzsrq5FmhfpaszTO01MAGs1vyScAaCxkE9D08I1bAAAAADAxSh0AAAAAmBilDgAAAABMjFIHAAAAACZGqQMAAAAAE6PUAY3gjTfe0PDhwxUZGakpU6bohx9+qLV96tSpmj9/vuv1vn37FBMTo5EjR2rUqFHasmXLBe+5adMm3XrrrY0+dwAAAJgLpQ5oYHv27NFLL72kt956Szk5Obruuuv07LPPuravWrVKO3bsqHVMYmKiJk+eLLvdrqVLl2r69Omqrq52bT948KCefPJJefkTSAAAAOABlDqggfXs2VMbNmxQq1atVFVVpeLiYrVt21aStG3bNn366aeKiYmpdcy7776rsLAwSdKhQ4fUunVr+fr6SpLOnj2rxMREJSUluXUdAAAAMAdKHdAI/Pz8tGnTJoWEhCg/P1/R0dEqLi7WokWLlJaW5ipsP7NarZKkwYMHKyEhQffff79rn8cff1z/9m//pu7du7t9HQAAAPB+Vk9PALhSDR48WIMHD9a6des0ceJEtW/fXnPmzJHNZrvo/haLRZs2bdLhw4cVFxenG2+8Ud99952sVqvGjh2roqIiN68AAAAAZkCpAxpYYWGhSktLFRwcLEkaM2aMUlJSdOLECS1ZskSSVFZWJqfTqaqqKqWkpGjjxo0aOnSofHx81KlTJw0cOFB79+5Vbm6uKisrNXLkSNXU1Lj+/cILLygoKMiTywQAAICXoNQBDay0tFSPPvqosrOzFRAQoPfff1/dunXTe++959rnueee04kTJ/T4449Lkp555hmdP39eUVFRKi4u1vbt2xUXF6eJEye6jikqKlJUVJTsdrvb1wQAAADvRakDGlhwcLAeeughxcfHy9fXVzabTStWrPjFY5YvX6758+frxRdflI+PjxITE9WrVy83zRgAAABmRqkDGkFsbKxiY2MvuT0hIaHW6+7duysrK+sX37Njx47auXNng8wPAAAAVw7ufgkAAAAAJkapAwAAAAATo9QBAAAAgIlR6gAAAADAxCh1AAAAAGBilDoAAAAAMDFKHUzv/LkaT08B4ucAAADgKTynDqbnY/VTwdLJnp5Gk3fbrBc9PQUAAIAmiTN1AAAAAGBilDoAAAAAMDFKHQAAAACYGKUOAAAAAEyMUgcAAAAAJkapAwAAAAATo9QBAAAAgIlR6gAAAADAxCh1AAAAAGBilDoAAAAAMDFKHQAAAACYGKUOAAAAAEyMUgcAAAAAJkapAwAAAAATq1epe/bZZzVs2DANHz5cL7/8siQpLy9PUVFRGjJkiNLT01377t27V2PGjFF4eLjmzp2rc+fOSZKOHj2quLg4RUREaMqUKaqoqGiE5QBoSsgmAN6KfALgTnWWui+++ELbtm3Te++9p//8z//U66+/rn379mnOnDnKzMxUbm6u9uzZoy1btkiSEhMTNW/ePG3YsEGGYWjdunWSpNTUVMXGxsrhcKhnz57KzMxs3JUBuKKRTQC8FfkEwN3qLHV33HGHXnvtNVmtVv3www9yOp06deqUunTpok6dOslqtSoqKkoOh0NHjhxRZWWl+vbtK0mKjo6Ww+FQTU2N8vPzFR4eXmscAC4X2QTAW5FPANytXpdf+vn5KSMjQ8OHD9eAAQNUUlKiwMBA13abzabi4uILxgMDA1VcXKwTJ07I399fVqu11jgA/BZkEwBvRT4BcKd63yhl2rRp2rp1q44dO6aDBw9esN1iscgwjF81DgC/FdkEwFuRTwDcpc5St3//fu3du1eSdPXVV2vIkCHavn27ysrKXPuUlJTIZrMpKCio1nhpaalsNpsCAgJUXl4up9NZaxwALhfZBMBbkU8A3K3OUldUVKTk5GRVV1erurpamzdvVkxMjA4cOKDCwkI5nU7l5OQoJCREHTp0UPPmzVVQUCBJys7OVkhIiPz8/BQcHKzc3Nxa4wBwucgmAN6KfALgbta6dggNDdWuXbs0atQo+fr6asiQIRo+fLgCAgKUkJCgqqoqhYaGKiIiQpKUlpam5ORkVVRUqEePHoqPj5ckpaSkKCkpSStXrlT79u21bNmyxl0ZgCsa2QTAW5FPANzNYlzsom0vUlRUpLCwMG3evFkdO3b8VcfGzspqpFmhvtYsjXPL5xQsneyWz8Gl3TbrRU9Pwe1+Sz4BQGMhm4Cmp943SgEAAAAAeB9KHQAAAACYGKUOAAAAAEyMUgcAAAAAJkapAwAAAAATo9QBAAAAgIlR6gAAAADAxCh1AAAAAGBilDoAAAAAMDFKHQAAAACYGKUOAAAAAEyMUgcAAAAAJkapAwAAAAATo9QBAAAAgIlR6gAAAADAxCh1AAAAAGBilDoAAAAAMDFKHQAAAACYGKUOAAAAAEyMUgcAAAAAJkapAwAAAAATo9QBAAAAgIlR6gAAAADAxCh1AAAAAGBilDoAAAAAMDFKHQAAAACYGKUOAAAAAEyMUgcAAAAAJkapAwAAAAATo9QBAAAAgIlR6gAAAADAxCh1AAAAAGBilDoAAAAAMDFKHQAAAACYGKUOAAAAAEyMUgcAAAAAJmatz07Lly/X+vXrJUmhoaGaNWuW8vLytHjxYlVVVWno0KGaMWOGJGnv3r1KTk5WeXm5goODlZqaKqvVqqNHjyoxMVE//PCDrr/+eqWlpally5aNtzIAVzyyqeEYhqHZs2erW7dumjRpkqZNm6bCwkLX9qKiIt1+++2aOXOmHnvsMdf4+fPn9Y9//EPPPfechgwZooSEBO3bt08tWrSQJPXr109z5sxx+3oATyKbALhbnWfq8vLy9Nlnn+ndd99Vdna2/vu//1s5OTmaM2eOMjMzlZubqz179mjLli2SpMTERM2bN08bNmyQYRhat26dJCk1NVWxsbFyOBzq2bOnMjMzG3dlAK5oZFPD2b9/v+677z7XL6GSlJGRIbvdLrvdrgULFqh169ZKSUlR165dXeN2u12DBg1SZGSkhgwZIknauXOn3njjDdd2Ch2aGrIJgCfUWeoCAwOVlJSkZs2ayc/PTzfeeKMOHjyoLl26qFOnTrJarYqKipLD4dCRI0dUWVmpvn37SpKio6PlcDhUU1Oj/Px8hYeH1xoHgMtFNjWcrKwsRUdHa+jQoRdsq66uVlJSkubMmaP27dvX2rZjxw5t2LBBqampkqTDhw+roqJCKSkpioqK0uzZs/Xjjz+6YwmA1yCbAHhCnaWuW7durrA5ePCgcnNzZbFYFBgY6NrHZrOpuLhYJSUltcYDAwNVXFysEydOyN/fX1artdY4AFwusqnhPP744xo1atRFt7399tuy2Wy65557Ltj25JNPavr06fL395ckHT9+XAMHDtT8+fOVnZ2tFi1acKYOTQ7ZBMAT6n2jlG+++UYTJ07Un//8Z3Xu3PmC7RaLRYZh/KpxAPityKbG9eqrr2rKlCkXjH/55Zc6ceKEoqKiXGN9+vTRihUrZLPZ5Ovrq6lTp2rLli2qrq5255QBr0A2AXCnepW6goICTZgwQY899phGjx6toKAglZWVubaXlJTIZrNdMF5aWiqbzaaAgACVl5fL6XTWGgeA34Jsalx///vfde7cOd1xxx0XbMvNzdWoUaPk4/N//zeyY8cObd682fXaMAxZLBb5+vq6Zb6AtyCbALhbnaXu2LFj+tOf/qS0tDQNHz5c0k9/jT1w4IAKCwvldDqVk5OjkJAQdejQQc2bN1dBQYEkKTs7WyEhIfLz81NwcLByc3NrjQPA5SKbGt8XX3yh/v37X/QMQX5+vvr3719rrKKiQgsXLnR9j2716tUKDw+n1KFJIZsAeEKdjzRYvXq1qqqqtGTJEtdYTEyMlixZooSEBFVVVSk0NFQRERGSpLS0NCUnJ6uiokI9evRQfHy8JCklJUVJSUlauXKl2rdvr2XLljXSkgA0BWRT4yssLFSHDh0uua1jx461xkJDQzV+/HiNGzdO58+fV/fu3bVgwQJ3TBXwGmQTAE+wGBe7cNuLFBUVKSwsTJs3b77gF4i6xM7KaqRZob7WLI1zy+cULJ3sls/Bpd0260VPT8Htfks+AUBjIZuApqfeN0oBAAAAAHgfSh0AAAAAmBilDgAAAABMjFIHAAAAACZGqQMAAAAAE6PUAYAbVdc4PT0F/C9+FgCAK0Wdz6kDADScZn6+PG7FS7jrkSvexjAMzZ49W926ddOkSZNqbZs6dapsNpsef/xxSdK+ffv0H//xHzp79qwsFotmzJih0NBQSdLGjRuVkZEhHx8ftW7dWosWLVLnzp3dvh4AAGfqAABoMvbv36/77rtP69evv2DbqlWrtGPHjlpjiYmJmjx5sux2u5YuXarp06erurpalZWVSkxM1PLly2W32xUWFqaFCxe6axkAgP+HUgcAQBORlZWl6OhoDR06tNb4tm3b9OmnnyomJqbW+LvvvquwsDBJ0qFDh9S6dWv5+vrK6XTKMAydPn1aklRRUaHmzZu7ZxFocgzDUFJSklavXi1JOn36tKZNm6bIyEgNGzZML7zwgmvfbdu2afTo0YqKitL48eO1b9++C97v1VdfVWRkpNvmD7gDl18CANBE/HxZ5bZt21xjxcXFWrRokVavXq21a9fW2t9qtcowDA0ePFhHjhzR3Llz5evrq5YtWyo1NVUxMTFq27atzp8/rzfffNOta0HTsH//fqWmpmrXrl3q1q2bJOnZZ59VUFCQMjIydObMGUVGRur2229X165dlZCQoIyMDA0YMED79+/Xww8/rPfff1/NmjWTJBUUFGjVqlVq27atB1cFNDxKHQAATVRNTY0effRRzZkzRzab7aL7WCwWbdq0SYcPH1ZcXJxuvPFGBQQEaMWKFcrNzVXnzp312muvKSEhQXa7XRaLxc2rwJXs57PLv/vd71xjc+fOldP5042OSktLVV1drVatWungwYNq1aqVBgwYIEm68cYb5e/vr507d6pfv34qKyvT/PnzNWvWrFpn94ArAaUOAIAmas+ePSoqKtKSJUskSWVlZXI6naqqqlJKSoo2btyooUOHysfHR506ddLAgQO1d+9eGYahW2+91XVjlLi4OC1evFgnTpxQQECAJ5eEK8zFzi5bLBZZrVbNnDlTGzZs0D333KPrr79eZ8+eVUVFhT777DPdeeed2r17t7799luVlpbK6XTqscce06xZs2S18usvrjx8pw4AgCbqlltu0ZYtW2S322W32xUTE6Nhw4Zp0aJFatasmZ555hl98MEHkn66THP79u26/fbb1aNHD+Xn56usrEyStGnTJnXs2JFCB7dKS0vTtm3bdPLkSa1YsUL+/v7KzMzUX/7yF40YMUJ2u139+/eXn5+fnn76ad1+++0aNGiQp6cNNAr+VAEAAC5q+fLlmj9/vl588UX5+PgoMTFRvXr1kiRNmjRJ48ePl5+fn9q0aaPMzEwPzxZNxaeffqqbbrpJQUFBatmypYYPH64PP/xQ58+fV8uWLfX666+79h06dKi6dOmiBQsWKCAgQBs3btSZM2dUXFyskSNHym63e3AlQMOh1AEA0MT8fLnl/5eQkFDrdffu3ZWVdfHnKsbFxSkurmk+6w+etX79em3cuFGpqamqqanR+vXrNWjQIFksFt1///3KzMxUr169tH79elmtVnXv3l2fffaZ6/jt27drwYIFFDpcUSh1AAAAMI2kpCSlpKQoKipKFotFYWFhio+Pl8Vi0dNPP6158+appqZGgYGByszM5OY9aBIodQAAAPBq/3x2uXXr1kpPT7/ofnfccYeys7N/8b369eunnJychpwe4HHcKAUAAAAATIxSBwAAAAAmRqkDAAAAABOj1AEA0AjOn6vx9BTwv/hZALjScaMUAAAagY/VTwVLJ3t6GpB026wXPT0FAGhUnKkDAABo4qprnJ6eAv4XPwtcDs7UAQAANHHN/HwVO+viD5qHe61ZGufpKcCEOFMHAAAAACZGqQMAAAAAE6PUAQAAAICJUeoAAAAAwMQodQAAAABgYpQ6AAAAADAxSh0AAAAAmBilDgAAAABMjIePAwAAAPAaGzduVEZGhnx8fNS6dWstWrRInTt3Vv/+/RUUFOTab9KkSRoxYoSKi4s1Z84clZaWyjAMTZ48WSNHjvTgCtyPUgcAAADAK1RWVioxMVF2u11dunTRK6+8ooULFyopKUlt2rSR3W6/4Jj09HT17t1bjzzyiIqLixUREaGBAwcqMDDQAyvwDC6/BAAAAOAVnE6nDMPQ6dOnJUkVFRVq3ry5du7cKR8fH40fP15RUVFavny5nE6n65jTp0/LMAydPXtWVqtVPj5Nq+Zwpg4AAACAV2jZsqVSU1MVExOjtm3b6vz583rzzTe1fft2DRo0SLNmzVJlZaUeeOAB+fv7a8KECXrssccUGxsrh8OhEydO6M9//rOuvfZaTy/FrSh1AAAAALzC119/rRUrVig3N1edO3fWa6+9poSEBNntdlksFklSs2bN9Mc//lGvv/66JkyYoJkzZ2ry5MmKjY3VwYMHNX78ePXt21e9e/f28Grcp97nJcvLyxUZGamioiJJUl5enqKiojRkyBClp6e79tu7d6/GjBmj8PBwzZ07V+fOnZMkHT16VHFxcYqIiNCUKVNUUVHRwEsB0BSRTQC8EdkEXJ7PPvtMt956qzp37ixJiouL0zfffCO73a59+/a59jMMQ1arVcePH1dBQYH+8Ic/SJKuu+46DRo0SPn5+R6Zv6fUq9Tt2rVL48aN08GDByX99AXGOXPmKDMzU7m5udqzZ4+2bNkiSUpMTNS8efO0YcMGGYahdevWSZJSU1Ndp0V79uypzMzMxlkRgCaDbALgjcgm4PL16NFD+fn5KisrkyRt2rRJHTt21DfffKOMjAw5nU5VVlYqKytLw4YN0zXXXKN27dppw4YNkqTjx48rPz9fffr08eQy3K5epW7dunVKSUmRzWaTJO3evVtdunRRp06dZLVaFRUVJYfDoSNHjqiyslJ9+/aVJEVHR8vhcKimpkb5+fkKDw+vNQ4AvwXZBMAbkU3A5RswYIAmTZqk8ePHa8SIEXrjjTeUmZmpqVOnqk2bNoqKitKIESN0yy236N5775XFYtHKlSu1Zs0aDR8+XPfdd58efPBBBQcHe3opblWv79QtWrSo1uuSkpJatwi12WwqLi6+YDwwMFDFxcU6ceKE/P39ZbVaa40DwG9BNgHwRmQT8NvExcUpLi7ugvHFixdfdP+bb75ZWVlZjT0tr3ZZ9/o0DOOCMYvF8qvHAaAhkU0AvBHZBKCxXVapCwoKcl3nKv30FyibzXbBeGlpqWw2mwICAlReXu56lsTP4wDQkMgmAN6IbALQ2C6r1PXp00cHDhxQYWGhnE6ncnJyFBISog4dOqh58+YqKCiQJGVnZyskJER+fn4KDg5Wbm5urXEAaEhkEwBvRDYBaGyX9Zy65s2ba8mSJUpISFBVVZVCQ0MVEREhSUpLS1NycrIqKirUo0cPxcfHS5JSUlKUlJSklStXqn379lq2bFnDrQIARDYB8E5kE4DG9qtK3UcffeT694ABA/Tee+9dsM/NN9+st99++4LxDh066PXXX7+MKQLALyObAHgjsgmAu1zW5ZcAAAAAzOn8uRpPTwFq2J/DZV1+CQAAAMCcfKx+Klg62dPTaPJum/Vig70XZ+oAAAAAwMQodQAAAABgYpQ6AAAAADAxSh0AAAAAmBilDgAAAABMjFIHAAAAACZGqQMAAAAAE6PUAQAAAICJUeoAAAAAwMQodQAAAABgYpQ6AAAAADAxSh0AAAAAmBilDgAAAABMjFIHAAAAACZGqQMAAAAAE6PUAQAAAICJUeoAAAAAwMQodQAAAABgYpQ6AAAAADAxSh0AAAAAmBilDgAAAABMjFIHAAAAACZGqQMAAAAAE6PUAQAAAICJUeoAAAAAwMQodQAAAABgYpQ6AAAAADAxSh0AAAAAmBilDgAAAABMjFIHAAAAACZGqQMAAAAAE6PUAQAAAICJUeoAAAAAwMQodQAAAABgYm4tde+//76GDRume+65R1lZWe78aAD4ReQTAG9ENgGoD6u7Pqi4uFjp6el655131KxZM8XExKhfv37q2rWru6YAABdFPgHwRmQTgPpyW6nLy8tT//791bZtW0lSeHi4HA6Hpk6d+ovHOZ1OSdL333//qz+z6syPv/oYNKyioiK3fE7p6Uq3fA4u7XJ/1u3atZPV6rYouih35xPZ5B3ckU9kk3e4nJ812QRP4XenpqMhs8ltaVVSUqLAwEDXa5vNpt27d9d5XGlpqSQpLi6u0eaGxhO2McPTU4C7rAu7rMM2b96sjh07NvBkfh3yqWkin5qQy8gnsgmeQjY1IQ2YTW4rdYZhXDBmsVjqPK5nz57KyspSYGCgfH19G2NqADyoXbt2np4C+QTgAmQTAG90qWxyW6kLCgrSjh07XK9LSkpks9nqPO6qq65ScHBwY04NQBNHPgHwRmQTgPpy290vBw4cqK1bt+r48eM6e/asPvzwQ4WEhLjr4wHgksgnAN6IbAJQX249UzdjxgzFx8erpqZGY8eOVe/evd318QBwSeQTAG9ENgGoL4txsQu2AQAAAACm4NaHjwMAAAAAGhalDgAAAABMjFIHAAAAACZGqQMAAAAAE6PUeaH3339fw4YN0z333KOsrCxPTweNrLy8XJGRkSoqKvL0VIA6kU9NB9kEMyGbmhby6UKUOi9TXFys9PR0rVmzRna7XWvXrtW3337r6WmhkezatUvjxo3TwYMHPT0VoE7kU9NBNsFMyKamhXy6OEqdl8nLy1P//v3Vtm1btWjRQuHh4XI4HJ6eFhrJunXrlJKSIpvN5umpAHUin5oOsglmQjY1LeTTxbnt4eOon5KSEgUGBrpe22w27d6924MzQmNatGiRp6cA1Bv51HSQTTATsqlpIZ8ujjN1XuZiz4K3WCwemAkA1EY+AfBGZBNAqfM6QUFBKisrc70uKSnh9DIAr0A+AfBGZBNAqfM6AwcO1NatW3X8+HGdPXtWH374oUJCQjw9LQAgnwB4JbIJ4Dt1XicoKEgzZsxQfHy8ampqNHbsWPXu3dvT0wIA8gmAVyKbAMliXOxCZAAAAACAKXD5JQAAAACYGKUOAAAAAEyMUgcAAAAAJkapAwAAAAATo9QBAAAAgIlR6gAAAADAxCh1AAAAAGBilDoAAAAAMLH/AXWkwXyAQlJiAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1080x360 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the distribution of the labels in each dataset split\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "sns.set_theme(\n",
    "    style=\"white\",\n",
    "    rc={\n",
    "        \"axes.spines.right\": False,\n",
    "        \"axes.spines.top\": False\n",
    "    }\n",
    ")\n",
    "\n",
    "def plot_label_distribution(dataset: datasets.DatasetDict):\n",
    "    fig, ax = plt.subplots(1, 3, figsize=(15, 5))\n",
    "\n",
    "    for i, split in enumerate(dataset):\n",
    "        sns.countplot(\n",
    "            x=\"label\",\n",
    "            data=dataset[split].to_pandas(),\n",
    "            ax=ax[i]\n",
    "        )\n",
    "\n",
    "        ax[i].set_ylim(0, 6000)\n",
    "\n",
    "        ax[i].set_title(split)\n",
    "        ax[i].set_ylabel(\"\")\n",
    "        ax[i].set_xlabel(\"\")\n",
    "\n",
    "        # Add annotations\n",
    "        for p in ax[i].patches:\n",
    "            ax[i].annotate(\n",
    "                format(p.get_height(), \".0f\"),\n",
    "                (p.get_x() + p.get_width() / 2., p.get_height()),\n",
    "                ha=\"center\",\n",
    "                va=\"center\",\n",
    "                xytext=(0, 10),\n",
    "                textcoords=\"offset points\"\n",
    "            )\n",
    "\n",
    "    plt.show()\n",
    "\n",
    "plot_label_distribution(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0acbf44c9b2147b1b3cdf7a01f192e70",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Flattening the indices:   0%|          | 0/10 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "70f211da27aa43eb9b6b3f7fe2ccc109",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Saving the dataset (0/1 shards):   0%|          | 0/9006 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9def059871ff4ec0919e01305a2efe3e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Saving the dataset (0/1 shards):   0%|          | 0/3213 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b054284ded749dea1e5dc487a7bb287",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Flattening the indices:   0%|          | 0/3 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "38ad73c3e8d34932a82e385901939bd1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Saving the dataset (0/1 shards):   0%|          | 0/2252 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dataset.save_to_disk(\"data\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Uploading the data to S3\n",
    "\n",
    "We are going to use the `sagemaker.Session.upload_data` function to upload our datasets to an S3 location.\n",
    "\n",
    "The return value inputs identifies the location -- we will use later when we start the training job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input spec (in this case, just an S3 path): s3://sagemaker-us-east-1-215993976552/ToChiquinho/toxic-comment-classification/data\n"
     ]
    }
   ],
   "source": [
    "# inputs = sagemaker_session.upload_data(\n",
    "#     path=\"data\",\n",
    "#     bucket=bucket_name,\n",
    "#     key_prefix=f\"{prefix}/data\"\n",
    "# )\n",
    "\n",
    "inputs = \"s3://sagemaker-us-east-1-215993976552/ToChiquinho/toxic-comment-classification/data\"\n",
    "\n",
    "print(\"input spec (in this case, just an S3 path): {}\".format(inputs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "shutil.rmtree(\"data\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training session\n",
    "\n",
    "In this section, we will run the training process.\n",
    "\n",
    "To use Amazon SageMaker to run Docker containers, we need to provide a Python script for the container to run. In our case, all the code is in the `ml` folder, including the `train.py` script.\n",
    "\n",
    "We will start doing a hyperparameter tuning process to find the best hyperparameters for our model.\n",
    "\n",
    "Then, we will train the model using the best hyperparameters found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MLFlow run ID: 227c8f6d9b144862a866d6de38d206dc\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import mlflow\n",
    "\n",
    "mlflow.start_run()\n",
    "\n",
    "print(f\"MLFlow run ID: {mlflow.active_run().info.run_id}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the estimator\n",
    "\n",
    "We will use the `sagemaker.huggingface.HuggingFace` class to define our estimator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "from sagemaker.huggingface import HuggingFace\n",
    "\n",
    "checkpoint_s3_uri = f\"s3://{bucket_name}/{prefix}/checkpoints\"\n",
    "\n",
    "# instance_type = \"ml.g4dn.xlarge\" # 4 vCPUs, 16 GB RAM, 1 x NVIDIA T4 16GB GPU - $ 0.736 per hour\n",
    "# instance_type = \"ml.g4dn.2xlarge\" # 8 vCPUs, 32 GB RAM, 1 x NVIDIA T4 16GB GPU - $ 0.94 per hour\n",
    "instance_type = \"ml.g5.xlarge\" # 4 vCPUs, 16 GB RAM, 1 x NVIDIA A10G 24GB GPU - $ 1.408 per hour\n",
    "# instance_type = \"ml.g5.2xlarge\" # 8 vCPUs, 32 GB RAM, 1 x NVIDIA A10G 24GB GPU - $ 1.515 per hour\n",
    "# instance_type = \"ml.g5.4xlarge\" # 16 vCPUs, 64 GB RAM, 2 x NVIDIA A10G 24GB GPU - $ 2.03 per hour\n",
    "# instance_type = \"ml.g5.8xlarge\" # 32 vCPUs, 128 GB RAM, 4 x NVIDIA A10G 24GB GPU - $ 3.06 per hour\n",
    "\n",
    "estimator = HuggingFace(\n",
    "    entry_point=\"train.py\",\n",
    "    source_dir=\"ml\",\n",
    "    base_job_name=params.mlflow_experiment_name,\n",
    "    container_log_level=logging.DEBUG,\n",
    "    role=params.sagemaker_execution_role_arn,\n",
    "    sagemaker_session=sagemaker_session,\n",
    "    py_version=\"py38\",\n",
    "    pytorch_version=\"1.10.2\",\n",
    "    transformers_version=\"4.17.0\",\n",
    "    instance_count=1,\n",
    "    instance_type=instance_type,\n",
    "    use_spot_instances=True,\n",
    "    max_wait=10800,\n",
    "    max_run=10800,\n",
    "    checkpoint_s3_uri=checkpoint_s3_uri,\n",
    "    checkpoint_local_path=\"/opt/ml/checkpoints\",\n",
    "    environment={\n",
    "        \"MLFLOW_TRACKING_URI\": params.mlflow_tracking_uri,\n",
    "        \"MLFLOW_EXPERIMENT_NAME\": params.mlflow_experiment_name,\n",
    "        \"MLFLOW_TRACKING_USERNAME\": params.mlflow_tracking_username,\n",
    "        \"MLFLOW_TRACKING_PASSWORD\": params.mlflow_tracking_password,\n",
    "        \"MLFLOW_TAGS\": params.mlflow_tags,\n",
    "        \"MLFLOW_RUN_ID\": mlflow.active_run().info.run_id,\n",
    "        \"MLFLOW_FLATTEN_PARAMS\": \"True\"\n",
    "    },\n",
    "    hyperparameters={\n",
    "        ## If you want to test the code, uncomment the following lines to use smaller datasets\n",
    "        # \"max_train_samples\": 100,\n",
    "        # \"max_val_samples\": 100,\n",
    "        # \"max_test_samples\": 100,\n",
    "        \"num_train_epochs\": params.num_train_epochs,\n",
    "        \"early_stopping_patience\": params.early_stopping_patience,\n",
    "        \"eval_dataset\": \"validation\",\n",
    "        \"batch_size\": params.batch_size,\n",
    "        \"seed\": params.seed\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlflow.log_params(\n",
    "    {\n",
    "        \"instance_type\": estimator.instance_type,\n",
    "        \"instance_count\": estimator.instance_count,\n",
    "        \"early_stopping_patience\": params.early_stopping_patience\n",
    "    }\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To test our training job before hyperparameter tuning, we will run it with a small number of samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n",
      "INFO:sagemaker.estimator:SMDebug Does Not Currently Support                         Distributed Training Jobs With Checkpointing Enabled\n",
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n",
      "INFO:sagemaker:Creating training-job with name: pytorch-training-2023-02-02-16-12-51-878\n"
     ]
    }
   ],
   "source": [
    "estimator.fit(inputs, wait=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter Tuning\n",
    "\n",
    "We will use the `sagemaker.tuner.HyperparameterTuner` class to run a hyperparameter tuning process.\n",
    "\n",
    "We use MLflow to track the training process, so we can analyze the results through the MLflow UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import (\n",
    "    IntegerParameter,\n",
    "    CategoricalParameter,\n",
    "    ContinuousParameter,\n",
    "    HyperparameterTuner,\n",
    ")\n",
    "\n",
    "estimator._hyperparameters.pop(\"max_train_samples\", None)\n",
    "estimator._hyperparameters.pop(\"max_val_samples\", None)\n",
    "estimator._hyperparameters.pop(\"max_test_samples\", None)\n",
    "\n",
    "tuner = HyperparameterTuner(\n",
    "    estimator,\n",
    "    max_jobs=18,\n",
    "    max_parallel_jobs=3,\n",
    "    objective_type=\"Maximize\",\n",
    "    objective_metric_name=\"eval_f1\",\n",
    "    metric_definitions=[\n",
    "        {\n",
    "            \"Name\": \"eval_f1\",\n",
    "            \"Regex\": \"eval_f1_weighted: ([0-9\\\\.]+)\"\n",
    "        }\n",
    "    ],\n",
    "    hyperparameter_ranges={\n",
    "        \"learning_rate\": ContinuousParameter(1e-5, 1e-3),\n",
    "        \"weight_decay\": ContinuousParameter(0.0, 0.1),\n",
    "        \"adam_beta1\": ContinuousParameter(0.8, 0.999),\n",
    "        \"adam_beta2\": ContinuousParameter(0.8, 0.999),\n",
    "        \"adam_epsilon\": ContinuousParameter(1e-8, 1e-6),\n",
    "        \"label_smoothing_factor\": ContinuousParameter(0.0, 0.1),\n",
    "        \"optim\": CategoricalParameter(\n",
    "            [\n",
    "                \"adamw_hf\",\n",
    "                \"adamw_torch\",\n",
    "                \"adamw_apex_fused\",\n",
    "                \"adafactor\"\n",
    "            ]\n",
    "        )\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No finished training job found associated with this estimator. Please make sure this estimator is only used for building workflow config\n",
      "No finished training job found associated with this estimator. Please make sure this estimator is only used for building workflow config\n"
     ]
    }
   ],
   "source": [
    "tuner.fit(inputs, wait=False)\n",
    "\n",
    "params.sagemaker_tuning_job_name = tuner.latest_tuning_job.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SageMaker tuning job name: pytorch-training-230202-1408\n"
     ]
    }
   ],
   "source": [
    "print(f\"SageMaker tuning job name: {params.sagemaker_tuning_job_name}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TrainingJobName</th>\n",
       "      <th>FinalObjectiveValue</th>\n",
       "      <th>TrainingJobStatus</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>pytorch-training-230202-1408-001-3a7ba5a5</td>\n",
       "      <td>0.868192</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>pytorch-training-230202-1408-010-fa54f873</td>\n",
       "      <td>0.865947</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>pytorch-training-230202-1408-011-51c0a2c0</td>\n",
       "      <td>0.865674</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>pytorch-training-230202-1408-008-38cd8920</td>\n",
       "      <td>0.864526</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>pytorch-training-230202-1408-015-edf8d39c</td>\n",
       "      <td>0.863600</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>pytorch-training-230202-1408-018-9deed799</td>\n",
       "      <td>0.861810</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>pytorch-training-230202-1408-012-566ef2d0</td>\n",
       "      <td>0.860839</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>pytorch-training-230202-1408-014-ed39b6e7</td>\n",
       "      <td>0.860671</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>pytorch-training-230202-1408-009-140c0893</td>\n",
       "      <td>0.860427</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>pytorch-training-230202-1408-005-777c7510</td>\n",
       "      <td>0.859007</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>pytorch-training-230202-1408-017-da98cd69</td>\n",
       "      <td>0.857632</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pytorch-training-230202-1408-016-b60a8aa2</td>\n",
       "      <td>0.855531</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>pytorch-training-230202-1408-003-421033a5</td>\n",
       "      <td>0.722852</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>pytorch-training-230202-1408-013-6a315783</td>\n",
       "      <td>0.473337</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>pytorch-training-230202-1408-007-6ead9eae</td>\n",
       "      <td>0.473337</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>pytorch-training-230202-1408-006-fb80af01</td>\n",
       "      <td>0.473337</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>pytorch-training-230202-1408-004-9bf72426</td>\n",
       "      <td>0.473337</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>pytorch-training-230202-1408-002-5ee87efd</td>\n",
       "      <td>0.473337</td>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              TrainingJobName  FinalObjectiveValue  \\\n",
       "17  pytorch-training-230202-1408-001-3a7ba5a5             0.868192   \n",
       "8   pytorch-training-230202-1408-010-fa54f873             0.865947   \n",
       "7   pytorch-training-230202-1408-011-51c0a2c0             0.865674   \n",
       "10  pytorch-training-230202-1408-008-38cd8920             0.864526   \n",
       "3   pytorch-training-230202-1408-015-edf8d39c             0.863600   \n",
       "0   pytorch-training-230202-1408-018-9deed799             0.861810   \n",
       "6   pytorch-training-230202-1408-012-566ef2d0             0.860839   \n",
       "4   pytorch-training-230202-1408-014-ed39b6e7             0.860671   \n",
       "9   pytorch-training-230202-1408-009-140c0893             0.860427   \n",
       "13  pytorch-training-230202-1408-005-777c7510             0.859007   \n",
       "1   pytorch-training-230202-1408-017-da98cd69             0.857632   \n",
       "2   pytorch-training-230202-1408-016-b60a8aa2             0.855531   \n",
       "15  pytorch-training-230202-1408-003-421033a5             0.722852   \n",
       "5   pytorch-training-230202-1408-013-6a315783             0.473337   \n",
       "11  pytorch-training-230202-1408-007-6ead9eae             0.473337   \n",
       "12  pytorch-training-230202-1408-006-fb80af01             0.473337   \n",
       "14  pytorch-training-230202-1408-004-9bf72426             0.473337   \n",
       "16  pytorch-training-230202-1408-002-5ee87efd             0.473337   \n",
       "\n",
       "   TrainingJobStatus  \n",
       "17         Completed  \n",
       "8          Completed  \n",
       "7          Completed  \n",
       "10         Completed  \n",
       "3          Completed  \n",
       "0          Completed  \n",
       "6          Completed  \n",
       "4          Completed  \n",
       "9          Completed  \n",
       "13         Completed  \n",
       "1          Completed  \n",
       "2          Completed  \n",
       "15         Completed  \n",
       "5          Completed  \n",
       "11         Completed  \n",
       "12         Completed  \n",
       "14         Completed  \n",
       "16         Completed  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "tuner_metrics: pd.DataFrame = sagemaker.HyperparameterTuningJobAnalytics(\n",
    "    hyperparameter_tuning_job_name=params.sagemaker_tuning_job_name,\n",
    "    sagemaker_session=sagemaker_session\n",
    ").dataframe()\n",
    "\n",
    "tuner_metrics[\"optim\"] = tuner_metrics[\"optim\"].apply(lambda x: x.strip('\"'))\n",
    "\n",
    "tuner_metrics.sort_values(\"FinalObjectiveValue\", ascending=False, inplace=True)\n",
    "tuner_metrics[[\"TrainingJobName\", \"FinalObjectiveValue\", \"TrainingJobStatus\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'adam_beta1': 0.8445637934160373,\n",
       " 'adam_beta2': 0.8338816842140165,\n",
       " 'adam_epsilon': 2.527092625455385e-08,\n",
       " 'label_smoothing_factor': 0.07158711257743958,\n",
       " 'learning_rate': 3.255788747459486e-05,\n",
       " 'optim': 'adamw_hf',\n",
       " 'weight_decay': 0.031031065174245122,\n",
       " 'TrainingJobName': 'pytorch-training-230202-1408-001-3a7ba5a5',\n",
       " 'TrainingJobStatus': 'Completed',\n",
       " 'FinalObjectiveValue': 0.8681923151016235,\n",
       " 'TrainingStartTime': Timestamp('2023-02-02 14:09:25-0300', tz='tzlocal()'),\n",
       " 'TrainingEndTime': Timestamp('2023-02-02 14:37:19-0300', tz='tzlocal()'),\n",
       " 'TrainingElapsedTimeSeconds': 1674.0}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_job = tuner_metrics.iloc[0].copy()\n",
    "best_job.to_dict()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training best model\n",
    "\n",
    "We will train the model using the best hyperparameters found.\n",
    "\n",
    "We will concatenate the training and validation datasets to train the model with more data and evaluate it using the test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n",
      "INFO:sagemaker:Creating training-job with name: toxic-comment-classification-2023-02-26-22-07-06-968\n"
     ]
    }
   ],
   "source": [
    "estimator.environment = {\n",
    "    \"MLFLOW_TRACKING_URI\": params.mlflow_tracking_uri,\n",
    "    \"MLFLOW_EXPERIMENT_NAME\": params.mlflow_experiment_name,\n",
    "    \"MLFLOW_TRACKING_USERNAME\": params.mlflow_tracking_username,\n",
    "    \"MLFLOW_TRACKING_PASSWORD\": params.mlflow_tracking_password,\n",
    "    \"MLFLOW_TAGS\": params.mlflow_tags,\n",
    "    \"MLFLOW_RUN_ID\": mlflow.active_run().info.run_id,\n",
    "    \"MLFLOW_FLATTEN_PARAMS\": \"True\",\n",
    "    \"HF_MLFLOW_LOG_ARTIFACTS\": \"True\",\n",
    "    \"HUGGINGFACE_HUB_TOKEN\": params.huggingface_hub_token\n",
    "}\n",
    "\n",
    "estimator._hyperparameters = {\n",
    "    \"push_to_hub\": \"True\",\n",
    "    \"hub_model_id\": f\"dougtrajano/{params.mlflow_experiment_name}\",\n",
    "    \"num_train_epochs\": params.num_train_epochs,\n",
    "    \"early_stopping_patience\": params.early_stopping_patience,\n",
    "    \"batch_size\": params.batch_size,\n",
    "    \"seed\": params.seed,\n",
    "    \"concat_validation_set\": \"True\",\n",
    "    \"eval_dataset\": \"test\",\n",
    "    \"adam_beta1\": best_job[\"adam_beta1\"],\n",
    "    \"adam_beta2\": best_job[\"adam_beta2\"],\n",
    "    \"adam_epsilon\": best_job[\"adam_epsilon\"],\n",
    "    \"learning_rate\": best_job[\"learning_rate\"],\n",
    "    \"weight_decay\": best_job[\"weight_decay\"],\n",
    "    \"label_smoothing_factor\": best_job[\"label_smoothing_factor\"],\n",
    "    \"optim\": best_job[\"optim\"]\n",
    "}\n",
    "\n",
    "estimator.fit(inputs, wait=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleted 42 checkpoints from sagemaker-us-east-1-215993976552/ToChiquinho/toxic-comment-classification/checkpoints.\n"
     ]
    }
   ],
   "source": [
    "remove_checkpoints(\n",
    "    bucket_name=bucket_name,\n",
    "    checkpoint_prefix=f\"{prefix}/checkpoints\",\n",
    "    aws_profile_name=params.aws_profile_name\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlflow.end_run()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Documentation\n",
    "\n",
    "- [Estimators — sagemaker documentation](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html)\n",
    "- [HyperparameterTuner — sagemaker documentation](https://sagemaker.readthedocs.io/en/stable/api/training/tuner.html)\n",
    "- [Configure and Launch a Hyperparameter Tuning Job - Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-ex-tuning-job.html)\n",
    "- [Managed Spot Training in Amazon SageMaker - Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "369f2c481f4da34e4445cda3fffd2e751bd1c4d706f27375911949ba6bb62e1c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}