ch07/04_preference-tuning-with-dpo/create-preference-data-ollama.zh.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "136a4efe-fb99-4311-8679-e0a5b6282755",
   "metadata": {},
   "source": [
    "<table style=\"width:100%\">\n",
    "<tr>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<font size=\"2\">\n",
    "Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "</font>\n",
    "</td>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
    "</td>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1910a06-e8a3-40ac-8201-ff70615b1ba4",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 使用 Llama 3.1 70B 和 Ollama 生成偏好数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a128651b-f326-4232-a994-42f38b7ed520",
   "metadata": {},
   "source": [
    "- 偏好微调是将指令微调的 LLM 与人类偏好对齐的过程\n",
    "- 有多种方法可以创建用于偏好微调 LLM 的数据集\n",
    "    1. 我们使用指令微调的 LLM 生成多个响应，并让人类根据他们的偏好和/或给定的偏好标准对它们进行排名\n",
    "    2. 我们使用指令微调的 LLM 生成多个响应，并让 LLM 根据给定的偏好标准对它们进行排名\n",
    "    3. 我们使用 LLM 在给定某些偏好标准的情况下生成首选和不首选的响应\n",
    "- 在此笔记本中，我们考虑方法 3\n",
    "- 此笔记本通过 ollama 使用 700 亿参数的 Llama 3.1-Instruct 模型为指令数据集生成偏好标签\n",
    "- 指令数据集的预期格式如下：\n",
    "\n",
    "\n",
    "### 输入\n",
    "\n",
    "```json\n",
    "[\n",
    "    {\n",
    "        \"instruction\": \"What is the state capital of California?\",\n",
    "        \"input\": \"\",\n",
    "        \"output\": \"The state capital of California is Sacramento.\",\n",
    "    },\n",
    "    {\n",
    "        \"instruction\": \"Provide a synonym for 'fast'.\",\n",
    "        \"input\": \"\",\n",
    "        \"output\": \"A synonym for 'fast' is 'quick'.\",\n",
    "    },\n",
    "    {\n",
    "        \"instruction\": \"What is the capital of Greece?\",\n",
    "        \"input\": \"\",\n",
    "        \"output\": \"The capital of Greece is Athens.\",\n",
    "\n",
    "    },\n",
    "...\n",
    "]\n",
    "```\n",
    "\n",
    "输出数据集将如下所示，其中更礼貌的回应更受青睐（`'chosen'`），而更不礼貌的回应则不受欢迎（`'rejected'`）：\n",
    "\n",
    "```json\n",
    "[\n",
    "    {\n",
    "        \"instruction\": \"What is the state capital of California?\",\n",
    "        \"input\": \"\",\n",
    "        \"output\": \"The state capital of California is Sacramento.\",\n",
    "        \"rejected\": \"Look, the state capital of California is obviously Sacramento.\",\n",
    "        \"chosen\": \"The state capital of California is Sacramento.\"\n",
    "    },\n",
    "    {\n",
    "        \"instruction\": \"Provide a synonym for 'fast'.\",\n",
    "        \"input\": \"\",\n",
    "        \"output\": \"A synonym for 'fast' is 'quick'.\",\n",
    "        \"chosen\": \"A suitable alternative to 'fast' would be 'quick'.\",\n",
    "        \"rejected\": \"A synonym for 'fast' is 'quick'.\"\n",
    "    },\n",
    "    {\n",
    "        \"instruction\": \"What is the capital of Greece?\",\n",
    "        \"input\": \"\",\n",
    "        \"output\": \"The capital of Greece is Athens.\",\n",
    "        \"chosen\": \"I'd be happy to help! The capital of Greece is indeed Athens.\",\n",
    "        \"rejected\": \"The capital of Greece is Athens.\"\n",
    "    },\n",
    "...\n",
    "]\n",
    "```\n",
    "\n",
    "### 输出\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "- 该代码不需要 GPU，只要有足够的 RAM 即可在笔记本电脑上运行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "63610acc-db94-437f-8d38-e99dca0299cb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tqdm version: 4.66.4\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "pkgs = [\"tqdm\",    # Progress bar\n",
    "        ]\n",
    "\n",
    "for p in pkgs:\n",
    "    print(f\"{p} version: {version(p)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bcdcb34-ac75-4f4f-9505-3ce0666c42d5",
   "metadata": {},
   "source": [
    "## 安装 Ollama 并下载 Llama 3.1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a092280-5462-4709-a3fe-8669a4a8a0a6",
   "metadata": {},
   "source": [
    "- Ollama 是一个高效运行 LLM 的应用程序\n",
    "- 它是 [llama.cpp](https://github.com/ggerganov/llama.cpp) 的包装器，它以纯 C/C++ 实现 LLM 以最大程度地提高效率\n",
    "- 请注意，它是一个使用 LLM 生成文本（推理）的工具，而不是训练或微调 LLM\n",
    "- 在运行以下代码之前，请访问 [https://ollama.com](https://ollama.com) 并按照说明安装 ollama（例如，单击“下载”按钮并下载适用于您的操作系统的 ollama 应用程序）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9558a522-650d-401a-84fc-9fd7b1f39da7",
   "metadata": {},
   "source": [
    "- 对于 macOS 和 Windows 用户，请单击您下载的 ollama 应用程序；如果它提示您安装命令行用法，请说“是”\n",
    "- Linux 用户可以使用 ollama 网站提供的安装命令\n",
    "\n",
    "- 通常，在我们可以从命令行使用 ollama 之前，我们必须启动 ollama 应用程序或在单独的终端中运行`ollama serve`\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/ollama-eval/ollama-serve.webp?1\">\n",
    "\n",
    "\n",
    "- 在 ollama 应用程序或 `ollama serve` 的运行下，在另一个终端的命令行上执行以下命令，试用 700 亿参数的 Llama 3.1 模型\n",
    "\n",
    "```bash\n",
    "# 70B model\n",
    "ollama run llama3.1:70b\n",
    "```\n",
    "\n",
    "\n",
    "The output looks like as follows:\n",
    "\n",
    "```\n",
    "$ ollama run llama3.1:70b\n",
    "pulling manifest\n",
    "pulling aa81b541aae6... 100% ▕████████████████▏ 39 GB\n",
    "pulling 8cf247399e57... 100% ▕████████████████▏ 1.7 KB\n",
    "pulling f1cd752815fc... 100% ▕████████████████▏ 12 KB\n",
    "pulling 56bb8bd477a5... 100% ▕████████████████▏ 96 B\n",
    "pulling 3c1c2d3df5b3... 100% ▕████████████████▏ 486 B\n",
    "verifying sha256 digest\n",
    "writing manifest\n",
    "removing any unused layers\n",
    "success\n",
    "```\n",
    "\n",
    "- 请注意， `llama3.1:70b` 指的是经过指令微调的 700 亿参数 Llama 3.1 模型\n",
    "\n",
    "- 或者，您也可以使用更小、更节省资源的 80 亿参数 Llama 3.1 模型，只需将 `llama3.1:70b` 替换为 `llama3.1` 即可\n",
    "\n",
    "- 下载完成后，您将看到一个命令行提示符，允许您与模型聊天\n",
    "\n",
    "- 尝试使用“骆驼吃什么？”之类的提示，它应该会返回类似于以下内容的输出：\n",
    "\n",
    "```\n",
    ">>> What do llamas eat?\n",
    "Llamas are ruminant animals, which means they have a four-chambered \n",
    "stomach and eat plants that are high in fiber. In the wild, llamas \n",
    "typically feed on:\n",
    "1. Grasses: They love to graze on various types of grasses, including tall \n",
    "grasses, wheat, oats, and barley.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b5addcb-fc7d-455d-bee9-6cc7a0d684c7",
   "metadata": {},
   "source": [
    "- 您可以使用输入“/bye”结束此会话"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dda155ee-cf36-44d3-b634-20ba8e1ca38a",
   "metadata": {},
   "source": [
    "## 使用 Ollama 的 REST API"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89343a84-0ddc-42fc-bf50-298a342b93c0",
   "metadata": {},
   "source": [
    "- 现在，与模型交互的另一种方法是通过 Python 中的 REST API，使用以下函数\n",
    "- 在运行此笔记本中的下一个单元之前，请确保 ollama 仍在运行，如上所述，通过\n",
    "  - 终端中的 `ollama serve` \n",
    "  - ollama 应用程序\n",
    "- 接下来，运行以下代码单元来查询模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16642a48-1cab-40d2-af08-ab8c2fbf5876",
   "metadata": {},
   "source": [
    "- 首先，让我们用一个简单的示例尝试该 API，以确保它能按预期工作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "65b0ba76-1fb1-4306-a7c2-8f3bb637ccdb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Llamas are herbivores, which means they primarily eat plants and plant-based foods. Their diet consists of:\n",
      "\n",
      "1. **Grasses**: Various types of grasses, including timothy grass, orchard grass, and brome grass.\n",
      "2. **Hay**: High-quality hay, such as alfalfa or clover hay, is a staple in a llama's diet.\n",
      "3. **Leaves**: Leaves from trees and shrubs, like willow, cottonwood, and mesquite, are also eaten.\n",
      "4. **Fruits and vegetables**: Llamas enjoy fruits like apples, carrots, and sweet potatoes, as well as leafy greens like kale and spinach.\n",
      "5. **Grains**: In moderation, llamas can eat grains like oats, barley, and corn.\n",
      "\n",
      "It's essential to note that llamas have a unique digestive system, with a three-part stomach and a large cecum (a specialized part of the large intestine). This allows them to break down and extract nutrients from plant material more efficiently than many other animals.\n",
      "\n",
      "A typical llama diet might consist of:\n",
      "\n",
      "* 1-2% of their body weight in hay per day\n",
      "* 0.5-1% of their body weight in grains per day (if fed)\n",
      "* Free-choice access to fresh water\n",
      "* Limited amounts of fruits and vegetables as treats\n",
      "\n",
      "It's also important to ensure that llamas have access to a mineral supplement, such as a salt lick or loose minerals, to help maintain optimal health.\n",
      "\n",
      "Remember, every llama is different, and their dietary needs may vary depending on factors like age, size, and activity level. Consult with a veterinarian or experienced llama breeder for specific guidance on feeding your llama.\n"
     ]
    }
   ],
   "source": [
    "import urllib.request\n",
    "import json\n",
    "\n",
    "\n",
    "def query_model(prompt, model=\"llama3.1:70b\", url=\"http://localhost:11434/api/chat\"):\n",
    "    # Create the data payload as a dictionary\n",
    "    data = {\n",
    "        \"model\": model,\n",
    "        \"messages\": [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": prompt\n",
    "            }\n",
    "        ],\n",
    "        \"options\": {\n",
    "            \"seed\": 123,\n",
    "            \"temperature\": 0,\n",
    "        }\n",
    "    }\n",
    "\n",
    "    # Convert the dictionary to a JSON formatted string and encode it to bytes\n",
    "    payload = json.dumps(data).encode(\"utf-8\")\n",
    "\n",
    "    # Create a request object, setting the method to POST and adding necessary headers\n",
    "    request = urllib.request.Request(url, data=payload, method=\"POST\")\n",
    "    request.add_header(\"Content-Type\", \"application/json\")\n",
    "\n",
    "    # Send the request and capture the response\n",
    "    response_data = \"\"\n",
    "    with urllib.request.urlopen(request) as response:\n",
    "        # Read and decode the response\n",
    "        while True:\n",
    "            line = response.readline().decode(\"utf-8\")\n",
    "            if not line:\n",
    "                break\n",
    "            response_json = json.loads(line)\n",
    "            response_data += response_json[\"message\"][\"content\"]\n",
    "\n",
    "    return response_data\n",
    "\n",
    "\n",
    "result = query_model(\"What do Llamas eat?\")\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "162a4739-6f03-4092-a5c2-f57a0b6a4c4d",
   "metadata": {},
   "source": [
    "## 加载 JSON 条目"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca011a8b-20c5-4101-979e-9b5fccf62f8a",
   "metadata": {},
   "source": [
    "- 现在，让我们进入数据生成部分\n",
    "- 在这里，作为一个实际示例，我们使用最初在第 7 章中用于指令微调模型的 `instruction-data.json` 文件："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8b2d393a-aa92-4190-9d44-44326a6f699b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 1100\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "json_file = Path(\"..\", \"01_main-chapter-code\", \"instruction-data.json\")\n",
    "\n",
    "with open(json_file, \"r\") as file:\n",
    "    json_data = json.load(file)\n",
    "\n",
    "print(\"Number of entries:\", len(json_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6c9751b-59b7-43fe-acc7-14e8daf2fa66",
   "metadata": {},
   "source": [
    "- 此文件的结构如下，其中我们在测试数据集（`'output'`）中给出了响应，我们根据 `'input'` 和 `'instruction'`通过指令微调训练模型来生成该响应"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7222fdc0-5684-4f2b-b741-3e341851359e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'instruction': 'Evaluate the following phrase by transforming it into the spelling given.',\n",
       " 'input': 'freind --> friend',\n",
       " 'output': 'The spelling of the given phrase \"freind\" is incorrect, the correct spelling is \"friend\".'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcf0331b-6024-4bba-89a9-a088b14a1046",
   "metadata": {},
   "source": [
    "- 下面是格式化指令和输入的小型实用函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "43263cd3-e5fb-4ab5-871e-3ad6e7d21a8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_input(entry):\n",
    "    instruction_text = (\n",
    "        f\"Below is an instruction that describes a task. Write a response that \"\n",
    "        f\"appropriately completes the request.\"\n",
    "        f\"\\n\\n### Instruction:\\n{entry['instruction']}\"\n",
    "    )\n",
    "\n",
    "    input_text = f\"\\n\\n### Input:\\n{entry['input']}\" if entry[\"input\"] else \"\"\n",
    "    instruction_text + input_text\n",
    "\n",
    "    return instruction_text + input_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39a55283-7d51-4136-ba60-f799d49f4098",
   "metadata": {},
   "source": [
    "- 现在，让我们尝试使用 ollama API 生成 `'chosen'` 和 `'rejected'` 响应，以便对模型进行偏好调整\n",
    "- 在这里，为了便于说明，我们创建了或多或少礼貌的答案\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "735cc089-d127-480a-b39d-0782581f0c41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Dataset response:\n",
      ">> The spelling of the given phrase \"freind\" is incorrect, the correct spelling is \"friend\".\n",
      "\n",
      "impolite response:\n",
      ">> The spelling of the given phrase \"freind\" is flat out wrong, get it together, the correct spelling is \"friend\".\n",
      "\n",
      "Dataset response:\n",
      ">> He goes to the park every day.\n",
      "\n",
      "polite response:\n",
      ">> He goes to the park daily, if I'm not mistaken.\n",
      "\n",
      "Dataset response:\n",
      ">> 45 kilometers is 45000 meters.\n",
      "\n",
      "polite response:\n",
      ">> 45 kilometers is equivalent to 45000 meters.\n",
      "\n",
      "Dataset response:\n",
      ">> Although it was raining, they went for a walk.\n",
      "\n",
      "polite response:\n",
      ">> Although it was raining outside, they still decided to go for a walk.\n",
      "\n",
      "Dataset response:\n",
      ">> 1, 4, 9, 16, 25, 36, 49, 64, 81, 100.\n",
      "\n",
      "impolite response:\n",
      ">> Here are your precious square numbers: 1, 4, 9, 16, 25, 36, 49, 64, 81, 100.\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "\n",
    "for entry in json_data[:5]:\n",
    "    \n",
    "    politeness = random.choice([\"polite\", \"impolite\"])    \n",
    "    prompt = (\n",
    "        f\"Given the input `{format_input(entry)}` \"\n",
    "        f\"and correct output `{entry['output']}`, \"\n",
    "        f\"slightly rewrite the output to be more {politeness}.\"\n",
    "        \"Keep the modification minimal.\"\n",
    "        \"Only return return the generated response and nothing else.\"\n",
    "    )\n",
    "    print(\"\\nDataset response:\")\n",
    "    print(\">>\", entry['output'])\n",
    "    print(f\"\\n{politeness} response:\")\n",
    "    print(\">>\", query_model(prompt))    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "142dfaa7-429f-4eb0-b74d-ff327f79547a",
   "metadata": {},
   "source": [
    "- 如果我们发现上面生成的响应看起来合理，我们可以进入下一步并将提示应用于整个数据集\n",
    "- 在这里，我们为首选响应添加一个 `'chosen'` 键，为不首选响应添加一个 `'rejected'` 响应"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3349dbbc-963f-4af3-9790-12dbfdca63c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "from tqdm import tqdm\n",
    "\n",
    "def generate_model_responses(json_data):\n",
    "\n",
    "    for i, entry in enumerate(tqdm(json_data, desc=\"Writing entries\")):\n",
    "        politeness = random.choice([\"polite\", \"impolite\"])    \n",
    "        prompt = (\n",
    "            f\"Given the input `{format_input(entry)}` \"\n",
    "            f\"and correct output `{entry['output']}`, \"\n",
    "            f\"slightly rewrite the output to be more {politeness}.\"\n",
    "            \"Keep the modification minimal.\"\n",
    "            \"Only return return the generated response and nothing else.\"\n",
    "        )\n",
    "        response = query_model(prompt)\n",
    "        \n",
    "        if politeness == \"polite\":\n",
    "            json_data[i][\"chosen\"] = response\n",
    "            json_data[i][\"rejected\"] = entry[\"output\"]\n",
    "        else:\n",
    "            json_data[i][\"rejected\"] = response\n",
    "            json_data[i][\"chosen\"] = entry[\"output\"]    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b071ce84-1866-427f-a272-b46700f364b2",
   "metadata": {},
   "source": [
    "- 现在让我们将此评估应用于整个数据集并计算每个模型的平均分数（在 M3 MacBook Air 笔记本电脑上，每个模型大约需要 1 分钟）\n",
    "- 请注意，ollama 在各个操作系统上并不完全确定（截至撰写本文时），因此您获得的数字可能与下面显示的数字略有不同"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4f700d4b-19e5-4404-afa7-b0f093024232",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Writing entries: 100%|██████████| 1100/1100 [17:20<00:00,  1.06it/s]\n"
     ]
    }
   ],
   "source": [
    "generate_model_responses(json_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "838d9747-0f7d-46fe-aab5-9ee6b765d021",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"instruction-data-with-preference.json\", \"w\") as file:\n",
    "    json.dump(json_data, file, indent=4)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}