peft-flan-t5-int8-summarization-cn.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用 LoRA 和 Hugging Face 高效训练大语言模型\n",
    "\n",
    "在本文中，我们将展示如何使用[大语言模型低秩适配（Low-Rank Adaptation of Large Language Models，LoRA）](https://arxiv.org/abs/2106.09685)技术在单 GPU 上微调 110 亿参数的 FLAN-T5 XXL 模型。在此过程中，我们会使用到 Hugging Face 的 [Transformers](https://huggingface.co/docs/transformers/index)、[Accelerate](https://huggingface.co/docs/accelerate/index) 和 [PEFT](https://github.com/huggingface/peft) 库。\n",
    "\n",
    "通过本文，你会学到：\n",
    "\n",
    "1. 如何搭建开发环境\n",
    "2. 如何加载并准备数据集\n",
    "3. 如何使用 LoRA 和 bnb（即bitsandbytes） int-8 微调 T5\n",
    "4. 如何评估 LoRA FLAN-T5 并将其用于推理\n",
    "5. 如何比较不同方案的性价比\n",
    "\n",
    "### 快速入门：轻量化微调（Parameter Efficient Fine-Tuning，PEFT）\n",
    "\n",
    "[PEFT](https://github.com/huggingface/peft) 是 Hugging Face 的一个新的开源库。使用 PEFT 库，无需微调模型的全部参数，即可高效地将预训练语言模型 (Pre-trained Language Model，PLM) 适配到各种下游应用。PEFT 目前支持以下几种方法：\n",
    "\n",
    "- LoRA：[LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/pdf/2106.09685.pdf)\n",
    "- Prefix Tuning：[P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://arxiv.org/pdf/2110.07602.pdf)\n",
    "- P-Tuning：[GPT Understands, Too](https://arxiv.org/pdf/2103.10385.pdf)\n",
    "- Prompt Tuning：[The Power of Scale for Parameter-Efficient Prompt Tuning](https://arxiv.org/pdf/2104.08691.pdf)\n",
    "\n",
    "*注意：本教程是在 g5.2xlarge AWS EC2 实例上创建和运行的，该实例包含 1 个 NVIDIA A10G。*"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 搭建开发环境\n",
    "\n",
    "在本例中，我们使用 AWS 预置的 [PyTorch 深度学习 AMI](https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-pytorch.html)，其已安装了正确的 CUDA 驱动程序和 PyTorch。在此基础上，我们还需要安装一些 Hugging Face 库，包括 transformers 和 datasets。运行下面的代码就可安装所有需要的包。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install Hugging Face Libraries\n",
    "!pip install  git+https://github.com/huggingface/peft.git\n",
    "!pip install \"transformers==4.27.1\" \"datasets==2.9.0\" \"accelerate==0.17.1\" \"evaluate==0.4.0\" \"bitsandbytes==0.37.1\" loralib --upgrade --quiet\n",
    "# install additional dependencies needed for training\n",
    "!pip install rouge-score tensorboard py7zr "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.加载并准备数据集\n",
    "\n",
    "这里，我们使用 [samsum](https://huggingface.co/datasets/samsum) 数据集，该数据集包含大约 16k 个含摘要的聊天类对话数据。这些对话由精通英语的语言学家制作。\n",
    "\n",
    "```python\n",
    "{\n",
    "  \"id\": \"13818513\",\n",
    "  \"summary\": \"Amanda baked cookies and will bring Jerry some tomorrow.\",\n",
    "  \"dialogue\": \"Amanda: I baked cookies. Do you want some?\\r\\nJerry: Sure!\\r\\nAmanda: I'll bring you tomorrow :-)\"\n",
    "}\n",
    "```\n",
    "\n",
    "我们使用 🤗 Datasets 库中的 *​`load_dataset()`* 方法来加载 `samsum` 数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "# Load dataset from the hub\n",
    "dataset = load_dataset(\"samsum\")\n",
    "\n",
    "print(f\"Train dataset size: {len(dataset['train'])}\")\n",
    "print(f\"Test dataset size: {len(dataset['test'])}\")\n",
    "\n",
    "# Train dataset size: 14732\n",
    "# Test dataset size: 819"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了训练模型，我们要用 🤗 Transformers Tokenizer 将输入文本转换为词元 ID。如果你需要了解这一方面的知识，请移步 Hugging Face 课程的 **[第 6 章](https://huggingface.co/course/chapter6/1?fw=tf)**。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n",
    "\n",
    "model_id=\"google/flan-t5-xxl\"\n",
    "\n",
    "# Load tokenizer of FLAN-t5-XL\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在开始训练之前，我们还需要对数据进行预处理。生成式文本摘要属于文本生成任务。我们将文本输入给模型，模型会输出摘要。我们需要了解输入和输出文本的长度信息，以利于我们高效地批量处理这些数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import concatenate_datasets\n",
    "import numpy as np\n",
    "# The maximum total input sequence length after tokenization. \n",
    "# Sequences longer than this will be truncated, sequences shorter will be padded.\n",
    "tokenized_inputs = concatenate_datasets([dataset[\"train\"], dataset[\"test\"]]).map(lambda x: tokenizer(x[\"dialogue\"], truncation=True), batched=True, remove_columns=[\"dialogue\", \"summary\"])\n",
    "input_lenghts = [len(x) for x in tokenized_inputs[\"input_ids\"]]\n",
    "# take 85 percentile of max length for better utilization\n",
    "max_source_length = int(np.percentile(input_lenghts, 85))\n",
    "print(f\"Max source length: {max_source_length}\")\n",
    "\n",
    "# The maximum total sequence length for target text after tokenization. \n",
    "# Sequences longer than this will be truncated, sequences shorter will be padded.\"\n",
    "tokenized_targets = concatenate_datasets([dataset[\"train\"], dataset[\"test\"]]).map(lambda x: tokenizer(x[\"summary\"], truncation=True), batched=True, remove_columns=[\"dialogue\", \"summary\"])\n",
    "target_lenghts = [len(x) for x in tokenized_targets[\"input_ids\"]]\n",
    "# take 90 percentile of max length for better utilization\n",
    "max_target_length = int(np.percentile(target_lenghts, 90))\n",
    "print(f\"Max target length: {max_target_length}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将在训练前统一对数据集进行预处理并将预处理后的数据集保存到磁盘。你可以在本地机器或 CPU 上运行此步骤并将其上传到 [Hugging Face Hub](https://huggingface.co/docs/hub/datasets-overview)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_function(sample,padding=\"max_length\"):\n",
    "    # add prefix to the input for t5\n",
    "    inputs = [\"summarize: \" + item for item in sample[\"dialogue\"]]\n",
    "\n",
    "    # tokenize inputs\n",
    "    model_inputs = tokenizer(inputs, max_length=max_source_length, padding=padding, truncation=True)\n",
    "\n",
    "    # Tokenize targets with the `text_target` keyword argument\n",
    "    labels = tokenizer(text_target=sample[\"summary\"], max_length=max_target_length, padding=padding, truncation=True)\n",
    "\n",
    "    # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore\n",
    "    # padding in the loss.\n",
    "    if padding == \"max_length\":\n",
    "        labels[\"input_ids\"] = [\n",
    "            [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels[\"input_ids\"]\n",
    "        ]\n",
    "\n",
    "    model_inputs[\"labels\"] = labels[\"input_ids\"]\n",
    "    return model_inputs\n",
    "\n",
    "tokenized_dataset = dataset.map(preprocess_function, batched=True, remove_columns=[\"dialogue\", \"summary\", \"id\"])\n",
    "print(f\"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}\")\n",
    "\n",
    "# save datasets to disk for later easy loading\n",
    "tokenized_dataset[\"train\"].save_to_disk(\"data/train\")\n",
    "tokenized_dataset[\"test\"].save_to_disk(\"data/eval\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 使用 LoRA 和 bnb int-8 微调 T5\n",
    "\n",
    "除了 LoRA 技术，我们还使用 [bitsanbytes LLM.int8()](https://huggingface.co/blog/hf-bitsandbytes-integration) 把冻结的 LLM 量化为 int8。这使我们能够将 FLAN-T5 XXL 所需的内存降低到约四分之一。\n",
    "\n",
    "训练的第一步是加载模型。我们使用 [philschmid/flan-t5-xxl-sharded-fp16](https://huggingface.co/philschmid/flan-t5-xxl-sharded-fp16) 模型，它是 [google/flan-t5-xxl](https://huggingface.co/google/flan-t5-xxl) 的分片版。分片可以让我们在加载模型时不耗尽内存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelForSeq2SeqLM\n",
    "\n",
    "# huggingface hub model id\n",
    "model_id = \"philschmid/flan-t5-xxl-sharded-fp16\"\n",
    "\n",
    "# load model from the hub\n",
    "model = AutoModelForSeq2SeqLM.from_pretrained(model_id, load_in_8bit=True, device_map=\"auto\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，我们可以使用 `peft` 为 LoRA int-8 训练作准备了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import LoraConfig, get_peft_model, prepare_model_for_int8_training, TaskType\n",
    "\n",
    "# Define LoRA Config \n",
    "lora_config = LoraConfig(\n",
    " r=16, \n",
    " lora_alpha=32,\n",
    " target_modules=[\"q\", \"v\"],\n",
    " lora_dropout=0.05,\n",
    " bias=\"none\",\n",
    " task_type=TaskType.SEQ_2_SEQ_LM\n",
    ")\n",
    "# prepare int-8 model for training\n",
    "model = prepare_model_for_int8_training(model)\n",
    "\n",
    "# add LoRA adaptor\n",
    "model = get_peft_model(model, lora_config)\n",
    "model.print_trainable_parameters()\n",
    "\n",
    "# trainable params: 18874368 || all params: 11154206720 || trainable%: 0.16921300163961817"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如你所见，这里我们只训练了模型参数的 0.16%！这个巨大的内存增益让我们安心地微调模型，而不用担心内存问题。\n",
    "\n",
    "接下来需要创建一个 `DataCollat​​or`，负责对输入和标签进行填充，我们使用 🤗 Transformers 库中的`DataCollat​​orForSeq2Seq` 来完成这一环节。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DataCollatorForSeq2Seq\n",
    "\n",
    "# we want to ignore tokenizer pad token in the loss\n",
    "label_pad_token_id = -100\n",
    "# Data collator\n",
    "data_collator = DataCollatorForSeq2Seq(\n",
    "    tokenizer,\n",
    "    model=model,\n",
    "    label_pad_token_id=label_pad_token_id,\n",
    "    pad_to_multiple_of=8\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后一步是定义训练超参 (`TrainingArguments`)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments\n",
    "\n",
    "output_dir=\"lora-flan-t5-xxl\"\n",
    "\n",
    "# Define training args\n",
    "training_args = Seq2SeqTrainingArguments(\n",
    "    output_dir=output_dir,\n",
    "\t\tauto_find_batch_size=True,\n",
    "    learning_rate=1e-3, # higher learning rate\n",
    "    num_train_epochs=5,\n",
    "    logging_dir=f\"{output_dir}/logs\",\n",
    "    logging_strategy=\"steps\",\n",
    "    logging_steps=500,\n",
    "    save_strategy=\"no\",\n",
    "    report_to=\"tensorboard\",\n",
    ")\n",
    "\n",
    "# Create Trainer instance\n",
    "trainer = Seq2SeqTrainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    data_collator=data_collator,\n",
    "    train_dataset=tokenized_dataset[\"train\"],\n",
    ")\n",
    "model.config.use_cache = False  # silence the warnings. Please re-enable for inference!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "运行下面的代码，开始训练模型。请注意，对于 T5，出于收敛稳定性考量，某些层我们仍保持 `float32` 精度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train model\n",
    "trainer.train()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练耗时约 10 小时 36 分钟，训练 10 小时的成本约为 `13.22 美元`。相比之下，如果[在 FLAN-T5-XXL 上进行全模型微调](https://www.philschmid.de/fine-tune-flan-t5-deepspeed#3-results--experiments) 10 个小时，我们需要 8 个 A100 40GB，成本约为 322 美元。\n",
    "\n",
    "我们可以将模型保存下来以用于后面的推理和评估。我们暂时将其保存到磁盘，但你也可以使用 `model.push_to_hub` 方法将其上传到 [Hugging Face Hub](https://huggingface.co/docs/hub/main)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save our LoRA model & tokenizer results\n",
    "peft_model_id=\"results\"\n",
    "trainer.model.save_pretrained(peft_model_id)\n",
    "tokenizer.save_pretrained(peft_model_id)\n",
    "# if you want to save the base model to call\n",
    "# trainer.model.base_model.save_pretrained(peft_model_id)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后生成的 LoRA checkpoint 文件很小，仅需 84MB 就包含了从 `samsum` 数据集上学到的所有知识。\n",
    "\n",
    "## 4. 使用 LoRA FLAN-T5 进行评估和推理\n",
    "\n",
    "我们将使用 `evaluate` 库来评估 `rogue` 分数。我们可以使用 `PEFT` 和 `transformers` 来对 FLAN-T5 XXL 模型进行推理。对 FLAN-T5 XXL 模型，我们至少需要 18GB 的​​ GPU 显存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from peft import PeftModel, PeftConfig\n",
    "from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n",
    "\n",
    "# Load peft config for pre-trained checkpoint etc. \n",
    "peft_model_id = \"results\"\n",
    "config = PeftConfig.from_pretrained(peft_model_id)\n",
    "\n",
    "# load base LLM model and tokenizer\n",
    "model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path,  load_in_8bit=True,  device_map={\"\":0})\n",
    "tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)\n",
    "\n",
    "# Load the Lora model\n",
    "model = PeftModel.from_pretrained(model, peft_model_id, device_map={\"\":0})\n",
    "model.eval()\n",
    "\n",
    "print(\"Peft model loaded\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们用测试数据集中的一个随机样本来试试摘要效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset \n",
    "from random import randrange\n",
    "\n",
    "\n",
    "# Load dataset from the hub and get a sample\n",
    "dataset = load_dataset(\"samsum\")\n",
    "sample = dataset['test'][randrange(len(dataset[\"test\"]))]\n",
    "\n",
    "input_ids = tokenizer(sample[\"dialogue\"], return_tensors=\"pt\", truncation=True).input_ids.cuda()\n",
    "# with torch.inference_mode():\n",
    "outputs = model.generate(input_ids=input_ids, max_new_tokens=10, do_sample=True, top_p=0.9)\n",
    "print(f\"input sentence: {sample['dialogue']}\\n{'---'* 20}\")\n",
    "\n",
    "print(f\"summary:\\n{tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0]}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不错！我们的模型有效！现在，让我们仔细看看，并使用 `test` 集中的全部数据对其进行评估。为此，我们需要实现一些工具函数来帮助生成摘要并将其与相应的参考摘要组合到一起。评估摘要任务最常用的指标是 [rogue_score](https://en.wikipedia.org/wiki/ROUGE_(metric))，它的全称是 Recall-Oriented Understudy for Gisting Evaluation。与常用的准确率指标不同，它将生成的摘要与一组参考摘要进行比较。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import evaluate\n",
    "import numpy as np\n",
    "from datasets import load_from_disk\n",
    "from tqdm import tqdm\n",
    "\n",
    "# Metric\n",
    "metric = evaluate.load(\"rouge\")\n",
    "\n",
    "def evaluate_peft_model(sample,max_target_length=50):\n",
    "    # generate summary\n",
    "    outputs = model.generate(input_ids=sample[\"input_ids\"].unsqueeze(0).cuda(), do_sample=True, top_p=0.9, max_new_tokens=max_target_length)    \n",
    "    prediction = tokenizer.decode(outputs[0].detach().cpu().numpy(), skip_special_tokens=True)\n",
    "    # decode eval sample\n",
    "    # Replace -100 in the labels as we can't decode them.\n",
    "    labels = np.where(sample['labels'] != -100, sample['labels'], tokenizer.pad_token_id)\n",
    "    labels = tokenizer.decode(labels, skip_special_tokens=True)\n",
    "\n",
    "    # Some simple post-processing\n",
    "    return prediction, labels\n",
    "\n",
    "# load test dataset from distk\n",
    "test_dataset = load_from_disk(\"data/eval/\").with_format(\"torch\")\n",
    "\n",
    "# run predictions\n",
    "# this can take ~45 minutes\n",
    "predictions, references = [] , []\n",
    "for sample in tqdm(test_dataset):\n",
    "    p,l = evaluate_peft_model(sample)\n",
    "    predictions.append(p)\n",
    "    references.append(l)\n",
    "\n",
    "# compute metric \n",
    "rogue = metric.compute(predictions=predictions, references=references, use_stemmer=True)\n",
    "\n",
    "# print results \n",
    "print(f\"Rogue1: {rogue['rouge1']* 100:2f}%\")\n",
    "print(f\"rouge2: {rogue['rouge2']* 100:2f}%\")\n",
    "print(f\"rougeL: {rogue['rougeL']* 100:2f}%\")\n",
    "print(f\"rougeLsum: {rogue['rougeLsum']* 100:2f}%\")\n",
    "\n",
    "# Rogue1: 50.386161%\n",
    "# rouge2: 24.842412%\n",
    "# rougeL: 41.370130%\n",
    "# rougeLsum: 41.394230%"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们 PEFT 微调后的 ​​FLAN-T5-XXL 在测试集上取得了 `50.38%` 的 rogue1 分数。相比之下，[flan-t5-base 的全模型微调获得了 47.23 的 rouge1 分数](https://www.philschmid.de/fine-tune-flan-t5)。rouge1 分数提高了 `3%` 。\n",
    "\n",
    "令人难以置信的是，我们的 LoRA checkpoint 只有 84MB，而且性能比对更小的模型进行全模型微调后的 checkpoint 更好。"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 英文原文: <url> https://www.philschmid.de/fine-tune-flan-t5-peft </url>\n",
    "\n",
    "> 原文作者：Philipp Schmid\n",
    "\n",
    "> 译者: Matrix Yao (姚伟峰)，英特尔深度学习工程师，工作方向为 transformer-family 模型在各模态数据上的应用及大规模模型的训练推理。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "2d58e898dde0263bc564c6968b04150abacfd33eed9b19aaa8e45c040360e146"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}