1-fine-tune-model.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Project setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0aff20fcb85e46f7a562d878f320343d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)lve/main/config.json:   0%|          | 0.00/665 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b454155a00044e0bbafce003885c68c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)olve/main/vocab.json:   0%|          | 0.00/1.04M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "604ffa51627448a78357cd776e6550ca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)olve/main/merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ac560b18e6de414ea9c114ae8ef605ce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)/main/tokenizer.json:   0%|          | 0.00/1.36M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2510e781b2ea424c8b802562cdf3b1ea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading pytorch_model.bin:   0%|          | 0.00/548M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec06bdf87b5f4e2494c61426ea1c691f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)neration_config.json:   0%|          | 0.00/124 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments, Trainer\n",
    "import torch\n",
    "from torch.utils.data import Dataset, random_split\n",
    "\n",
    "model_name = \"gpt2\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name)\n",
    "\n",
    "device = 'cpu'\n",
    "if torch.cuda.is_available():\n",
    "    device = 'cuda'\n",
    "\n",
    "print(device)\n",
    "model = model.to(torch.device(device))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test out GPT2 as-is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    }
   ],
   "source": [
    "text = \"Data scientists use VS Code as their tool\"\n",
    "\n",
    "if device == 'cuda':\n",
    "    input_ids = tokenizer.encode(text, return_tensors='pt').cuda()\n",
    "else:\n",
    "    input_ids = tokenizer.encode(text, return_tensors='pt')\n",
    "\n",
    "\n",
    "output = model.generate(input_ids, max_length=100, do_sample=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data scientists use VS Code as their tool for understanding a data set. It is a simple, clean, and easy-to-use way to get results that accurately reflect what you've seen, read or seen before.\n",
      "\n",
      "The VS Lab tool creates a dataset of all my data for a given period of time and makes them as user-friendly as possible. This is the ultimate way to understand any aspect of your data set. They all automatically update as you change formats and data sets. In\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.decode(output[0], skip_special_tokens=True))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fine-tune model using Medium posts with 'Technology' tag\n",
    "Data from [Kaggle](https://www.kaggle.com/datasets/fabiochiusano/medium-articles), filtered to posts with `Technology` tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>⭐A Target Package is short for Target Package ...</td>\n",
       "      <td>9176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>‘WATCH’ ~ New Series HDTV! ~ The Good Fight Se...</td>\n",
       "      <td>11175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>⭐A Target Package is short for Target Package ...</td>\n",
       "      <td>9175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>⭐A Target Package is short for Target Package ...</td>\n",
       "      <td>9195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>⭐A Target Package is short for Target Package ...</td>\n",
       "      <td>9212</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  text_len\n",
       "0  ⭐A Target Package is short for Target Package ...      9176\n",
       "1  ‘WATCH’ ~ New Series HDTV! ~ The Good Fight Se...     11175\n",
       "2  ⭐A Target Package is short for Target Package ...      9175\n",
       "3  ⭐A Target Package is short for Target Package ...      9195\n",
       "4  ⭐A Target Package is short for Target Package ...      9212"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "data = pd.read_csv('./resources/medium-articles-technology.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2048 > 1024). Running this sequence through the model will result in indexing errors\n"
     ]
    }
   ],
   "source": [
    "texts = pd.Series(data.text)\n",
    "\n",
    "max_length = min(max(len(tokenizer.encode(text)) for text in texts), 1024)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Embedding(50259, 768)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(model_name, bos_token='<|startoftext|>',\n",
    "    eos_token='<|endoftext|>', pad_token='<|pad|>')\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name)\n",
    "model.resize_token_embeddings(len(tokenizer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TextsDataset(Dataset):\n",
    "    def __init__(self, txt_list, tokenizer, max_length):\n",
    "        self.input_ids = []\n",
    "        self.attn_masks = []\n",
    "        self.labels = []\n",
    "        for txt in txt_list:\n",
    "            encodings_dict = tokenizer('<|startoftext|>' + txt + '<|endoftext|>', truncation=True,\n",
    "                max_length=max_length, padding=\"max_length\")\n",
    "            self.input_ids.append(torch.tensor(encodings_dict['input_ids']))\n",
    "            self.attn_masks.append(torch.tensor(encodings_dict['attention_mask']))\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.input_ids)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return self.input_ids[idx], self.attn_masks[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = TextsDataset(texts, tokenizer, max_length=max_length)\n",
    "train_size = int(0.9 * len(dataset))\n",
    "train_dataset, val_dataset = random_split(dataset, [train_size, len(dataset) - train_size])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_args = TrainingArguments(\n",
    "    output_dir='./results',\n",
    "    num_train_epochs=3,\n",
    "    logging_steps=100,\n",
    "    save_steps=5000,\n",
    "    per_device_train_batch_size=4,\n",
    "    per_device_eval_batch_size=4,\n",
    "    warmup_steps=100,\n",
    "    weight_decay=0.05,\n",
    "    logging_dir='./logs',\n",
    "    report_to='none',\n",
    "    gradient_accumulation_steps=4,  # Use gradient accumulation\n",
    "    fp16=True,                      # Enable mixed precision training\n",
    "    learning_rate=5e-5,             # Adjust the learning rate\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/vscode/.local/lib/python3.10/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='96' max='96' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [96/96 01:16, Epoch 2.92/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=96, training_loss=25.34460957845052, metrics={'train_runtime': 78.4963, 'train_samples_per_second': 19.874, 'train_steps_per_second': 1.223, 'total_flos': 802689122304000.0, 'train_loss': 25.34460957845052, 'epoch': 2.95})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Trainer(model=model, args=training_args, train_dataset=train_dataset, \n",
    "        eval_dataset=val_dataset, data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),\n",
    "                                                              'attention_mask': torch.stack([f[1] for f in data]),\n",
    "                                                              'labels': torch.stack([f[0] for f in data])}).train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    }
   ],
   "source": [
    "input_ids = tokenizer.encode(\"<|startoftext|> \" + text, return_tensors='pt').cuda()\n",
    "\n",
    "output = model.generate(input_ids, max_length=100, do_sample=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Data scientists use VS Code as their tool of choice.\n",
      "\n",
      "The code is easy to code, but it is so incredibly difficult to comprehend this concept and execute. But the more complex data scientists have, the better we would be able to better understand it, and learning it.\n",
      "So, I understand like this. As well, I have been curious, how many different ways to communicate with friends, how many different ways to communicate with some of them, and how many various methods\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.decode(output[0], skip_special_tokens=True))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare outputs\n",
    "Comparing generated texts from original GPT2 model & fine-tuned GPT2 model, with starter sentence \"Data scientists use VS Code as their tool\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Original GPT2:**<br>\n",
    "Data scientists use VS Code as their tool to produce scientific experiments that are able to be studied together by colleagues on the project's two continents. Unlike normal collaboration research, these experiments are divided according to the types of results they will show. \"We're going to be using data from all areas that have a large share of clinical data from this project, which isn't even the largest area for this project. The reason we're doing that is simple: One way to use this data as a\n",
    "\n",
    "**Fine-tuned GPT2 with Medium `Technology` posts:**<br>\n",
    "Data scientists use VS Code as their tool of choice code. The code is written using the Language of choice made by the Code. The code is written using the Language as its foundation in the collection of applications and the resulting Collection. The resulting collection of applications is an example of the Code in the code of application. The resulting Code consists of different elements of the Application and an object from the code. The object contains various aspects of the application. The object consists of the application. The object"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('./model/medium-tech/tokenizer_config.json',\n",
       " './model/medium-tech/special_tokens_map.json',\n",
       " './model/medium-tech/vocab.json',\n",
       " './model/medium-tech/merges.txt',\n",
       " './model/medium-tech/added_tokens.json',\n",
       " './model/medium-tech/tokenizer.json')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.save_pretrained(\"./model/medium-tech\")\n",
    "tokenizer.save_pretrained(\"./model/medium-tech\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "3ad933181bd8a04b432d3370b9dc3b0662ad032c4dfaa4e4f1596c548f763858"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}