examples/ECIR2023-Intents.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "mwbe7TYfjq_Z",
   "metadata": {
    "id": "mwbe7TYfjq_Z"
   },
   "source": [
    "# Crowdsourcing for Information Retrieval: Tutorial at ECIR '23\n",
    "\n",
    "![Crowd-Kit](https://repository-images.githubusercontent.com/343581364/50c743cc-9bc8-4bae-947c-7803158fb97e)\n",
    "\n",
    "In this part of our tutorial, we will use the [Crowd-Kit](https://github.com/Toloka/crowd-kit) library for Python to aggregate and learn from crowd annotations.\n",
    "\n",
    "We will be using the [CLINC150](https://paperswithcode.com/dataset/clinc150) dataset for evaluating the performance of *intent classification* systems. As this is a multi-class classification problem with 9 classes, we will use macro-averaged $F_1$ as the evaluation criterion.\n",
    "\n",
    "Note that we uploaded all the materials and labeled datasets to [Google Drive](https://drive.google.com/drive/folders/1jMNkCs1DzJESiL2-8Pr8NJhqyi7GZJp5?usp=sharing).\n",
    "\n",
    "**Outline:**\n",
    "\n",
    "1. Install dependencies, load annotated and ground truth datasets, `train` and `test`.\n",
    "2. Run traditional answer aggregation methods, such as [Majority Vote](https://toloka.ai/docs/crowd-kit/reference/crowdkit.aggregation.classification.majority_vote.MajorityVote) (MV), [Dawid-Skene](https://toloka.ai/docs/crowd-kit/reference/crowdkit.aggregation.classification.dawid_skene.DawidSkene) (DS), [Generative Models of Labels, Abilities, and Difficulties](https://toloka.ai/docs/crowd-kit/reference/crowdkit.aggregation.classification.glad.GLAD) (GLAD).\n",
    "3. Choose the aggregation method on `train` subset and evaluate it on `test`.\n",
    "4. Train a model on the aggregated dataset.\n",
    "5. Train a model on the raw (non-aggregated) dataset.\n",
    "6. Train a model on the raw dataset with [CrowdLayer](https://toloka.ai/docs/crowd-kit/reference/crowdkit.learning.crowd_layer.CrowdLayer) and [CoNAL](https://toloka.ai/docs/crowd-kit/reference/crowdkit.learning.conal.CoNAL)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "oinQl8p3YZX_",
   "metadata": {
    "editable": true,
    "id": "oinQl8p3YZX_",
    "tags": []
   },
   "source": [
    "# Answer Aggregation\n",
    "\n",
    "We will install and import all the necessary libraries, which are [Crowd-Kit](https://github.com/Toloka/crowd-kit), [PyTorch Lightning](https://lightning.ai/pytorch-lightning), and [Sentence Transformers](https://sbert.net/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "caadcc29-56fe-4c4c-a303-4038d335ee4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "%pip install -U crowd-kit lightning sentence-transformers gdown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3cb935d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "import lightning as L\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torchmetrics\n",
    "from sentence_transformers import SentenceTransformer  # noqa: F401\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.preprocessing import LabelBinarizer, LabelEncoder\n",
    "from torch import nn, utils\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "from crowdkit.aggregation import GLAD, DawidSkene, MajorityVote\n",
    "from crowdkit.learning import CoNAL, CrowdLayer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ff8fdba2-ba1d-427a-b09f-b4a5623a9be3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=15I1KGAF3DZMl_g26tEtJlZ6jhDAwde5y\n",
      "To: /content/labeled_train_data.csv\n",
      "100% 127k/127k [00:00<00:00, 83.9MB/s]\n",
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=1ekfvUlUTg1UWL25cZwMmZcOJh-HhrlNs\n",
      "To: /content/labeled_test_data.csv\n",
      "100% 76.9k/76.9k [00:00<00:00, 83.7MB/s]\n"
     ]
    }
   ],
   "source": [
    "!gdown --fuzzy 'https://drive.google.com/file/d/15I1KGAF3DZMl_g26tEtJlZ6jhDAwde5y/view?usp=share_link'\n",
    "!gdown --fuzzy 'https://drive.google.com/file/d/1ekfvUlUTg1UWL25cZwMmZcOJh-HhrlNs/view?usp=share_link'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "C3Mi5WiYaj3U",
   "metadata": {
    "id": "C3Mi5WiYaj3U"
   },
   "source": [
    "Let's open the crowdsourced datasets with labeled train subset of our data (`labeled_train_data.csv`) and test subset (`labeled_test_data.csv`). We will re-arrange our dataset so its column names match what Crowd-Kit expects: `task`, `worker`, and `label`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "pfBjOlUOh1r2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-17da3684-6e13-4ec1-95e1-43efc4def87d\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>task</th>\n",
       "      <th>label</th>\n",
       "      <th>worker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>please show me a recipe for chili</td>\n",
       "      <td>recipe</td>\n",
       "      <td>b5329f9413a8795bfc3d45ffbbb1d31a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>do i need to protect myself with some shots fo...</td>\n",
       "      <td>vaccines</td>\n",
       "      <td>b5329f9413a8795bfc3d45ffbbb1d31a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>i would love to know the exchange rate between...</td>\n",
       "      <td>exchange_rate</td>\n",
       "      <td>b5329f9413a8795bfc3d45ffbbb1d31a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>is it recommended to get any specific shots be...</td>\n",
       "      <td>vaccines</td>\n",
       "      <td>b5329f9413a8795bfc3d45ffbbb1d31a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>what are some fun things i can partake in in a...</td>\n",
       "      <td>travel_suggestion</td>\n",
       "      <td>b5329f9413a8795bfc3d45ffbbb1d31a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1345</th>\n",
       "      <td>what's a good place to travel to</td>\n",
       "      <td>travel_suggestion</td>\n",
       "      <td>193f02fc8190d97758d5634cfd5168dc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1346</th>\n",
       "      <td>where should i spend my time off</td>\n",
       "      <td>travel_suggestion</td>\n",
       "      <td>193f02fc8190d97758d5634cfd5168dc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1347</th>\n",
       "      <td>do i need shots before i get to africa</td>\n",
       "      <td>vaccines</td>\n",
       "      <td>193f02fc8190d97758d5634cfd5168dc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1348</th>\n",
       "      <td>i want to know the nutrition info for chicken ...</td>\n",
       "      <td>nutrition_info</td>\n",
       "      <td>193f02fc8190d97758d5634cfd5168dc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1349</th>\n",
       "      <td>book a reservation for 4 at outback steakhouse...</td>\n",
       "      <td>restaurant_reservation</td>\n",
       "      <td>193f02fc8190d97758d5634cfd5168dc</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1350 rows × 3 columns</p>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-17da3684-6e13-4ec1-95e1-43efc4def87d')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-17da3684-6e13-4ec1-95e1-43efc4def87d button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-17da3684-6e13-4ec1-95e1-43efc4def87d');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-206b2e8d-1f72-4352-a664-83fe3730d70b\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-206b2e8d-1f72-4352-a664-83fe3730d70b')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-206b2e8d-1f72-4352-a664-83fe3730d70b button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "                                                   task  \\\n",
       "0                     please show me a recipe for chili   \n",
       "1     do i need to protect myself with some shots fo...   \n",
       "2     i would love to know the exchange rate between...   \n",
       "3     is it recommended to get any specific shots be...   \n",
       "4     what are some fun things i can partake in in a...   \n",
       "...                                                 ...   \n",
       "1345                   what's a good place to travel to   \n",
       "1346                   where should i spend my time off   \n",
       "1347             do i need shots before i get to africa   \n",
       "1348  i want to know the nutrition info for chicken ...   \n",
       "1349  book a reservation for 4 at outback steakhouse...   \n",
       "\n",
       "                       label                            worker  \n",
       "0                     recipe  b5329f9413a8795bfc3d45ffbbb1d31a  \n",
       "1                   vaccines  b5329f9413a8795bfc3d45ffbbb1d31a  \n",
       "2              exchange_rate  b5329f9413a8795bfc3d45ffbbb1d31a  \n",
       "3                   vaccines  b5329f9413a8795bfc3d45ffbbb1d31a  \n",
       "4          travel_suggestion  b5329f9413a8795bfc3d45ffbbb1d31a  \n",
       "...                      ...                               ...  \n",
       "1345       travel_suggestion  193f02fc8190d97758d5634cfd5168dc  \n",
       "1346       travel_suggestion  193f02fc8190d97758d5634cfd5168dc  \n",
       "1347                vaccines  193f02fc8190d97758d5634cfd5168dc  \n",
       "1348          nutrition_info  193f02fc8190d97758d5634cfd5168dc  \n",
       "1349  restaurant_reservation  193f02fc8190d97758d5634cfd5168dc  \n",
       "\n",
       "[1350 rows x 3 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_crowd_train = pd.read_csv(\"labeled_train_data.csv\")\n",
    "df_crowd_test = pd.read_csv(\"labeled_test_data.csv\")\n",
    "\n",
    "df_crowd_train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "yxRDHpBra3p1",
   "metadata": {
    "id": "yxRDHpBra3p1"
   },
   "source": [
    "Now let's download and open the ground truth dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f6c7f4ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -sLO https://raw.githubusercontent.com/clinc/oos-eval/master/data/data_small.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dd7f6b96",
   "metadata": {},
   "outputs": [],
   "source": [
    "intents = {\n",
    "    \"restaurant_reviews\",\n",
    "    \"restaurant_reservation\",\n",
    "    \"nutrition_info\",\n",
    "    \"recipe\",\n",
    "    \"book_hotel\",\n",
    "    \"timezone\",\n",
    "    \"travel_suggestion\",\n",
    "    \"exchange_rate\",\n",
    "    \"vaccines\",\n",
    "}\n",
    "\n",
    "with open(\"data_small.json\") as f:\n",
    "    oos = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "m3PyE_Aoew62",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-cff8c570-71d3-46f4-8da4-deeb4ffd58a3\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>intent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>i need a blackberry pie recipe</td>\n",
       "      <td>recipe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>find a recipe for german chocolate cake</td>\n",
       "      <td>recipe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>please find me a recipe for spaghetti now</td>\n",
       "      <td>recipe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>can you find me a recipe for sugar cookies</td>\n",
       "      <td>recipe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>i need a recipe on how to make beef stew</td>\n",
       "      <td>recipe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>445</th>\n",
       "      <td>what timezone is los angeles in</td>\n",
       "      <td>timezone</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>446</th>\n",
       "      <td>what timezone is new york in</td>\n",
       "      <td>timezone</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>447</th>\n",
       "      <td>reno's timezone is what</td>\n",
       "      <td>timezone</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>448</th>\n",
       "      <td>what timezone is china in</td>\n",
       "      <td>timezone</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>449</th>\n",
       "      <td>what timezone is london is</td>\n",
       "      <td>timezone</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>450 rows × 2 columns</p>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cff8c570-71d3-46f4-8da4-deeb4ffd58a3')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-cff8c570-71d3-46f4-8da4-deeb4ffd58a3 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-cff8c570-71d3-46f4-8da4-deeb4ffd58a3');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-5b6de3f3-dd7b-4aae-ae08-7d5b0af9ddf6\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-5b6de3f3-dd7b-4aae-ae08-7d5b0af9ddf6')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-5b6de3f3-dd7b-4aae-ae08-7d5b0af9ddf6 button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "                                          query    intent\n",
       "0                i need a blackberry pie recipe    recipe\n",
       "1       find a recipe for german chocolate cake    recipe\n",
       "2     please find me a recipe for spaghetti now    recipe\n",
       "3    can you find me a recipe for sugar cookies    recipe\n",
       "4      i need a recipe on how to make beef stew    recipe\n",
       "..                                          ...       ...\n",
       "445             what timezone is los angeles in  timezone\n",
       "446                what timezone is new york in  timezone\n",
       "447                     reno's timezone is what  timezone\n",
       "448                   what timezone is china in  timezone\n",
       "449                  what timezone is london is  timezone\n",
       "\n",
       "[450 rows x 2 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train = pd.DataFrame(oos[\"train\"], columns=[\"query\", \"intent\"])\n",
    "\n",
    "df_train = df_train[df_train[\"intent\"].isin(intents)].reset_index(drop=True)\n",
    "\n",
    "df_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "pqV5InnrewtY",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-d0506086-8beb-449d-b9a3-2217f93e1fa6\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>intent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>i'd like to make a reservation at rooth chris,...</td>\n",
       "      <td>restaurant_reservation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>make a reservation for chik-fil-a at 3 o' cloc...</td>\n",
       "      <td>restaurant_reservation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>reserve a table for 3 at 7 for olive garden</td>\n",
       "      <td>restaurant_reservation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>will you reserve a table at olive garden for 3...</td>\n",
       "      <td>restaurant_reservation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>at 7, i need a table for 3 at olive garden</td>\n",
       "      <td>restaurant_reservation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>265</th>\n",
       "      <td>i need a pasta recipe</td>\n",
       "      <td>recipe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>266</th>\n",
       "      <td>i want a recipe for roasted veggies</td>\n",
       "      <td>recipe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>267</th>\n",
       "      <td>what is in a burrito recipe</td>\n",
       "      <td>recipe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>268</th>\n",
       "      <td>give me a tuna salad recipe</td>\n",
       "      <td>recipe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>269</th>\n",
       "      <td>find a recipe for baked ziti</td>\n",
       "      <td>recipe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>270 rows × 2 columns</p>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d0506086-8beb-449d-b9a3-2217f93e1fa6')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-d0506086-8beb-449d-b9a3-2217f93e1fa6 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-d0506086-8beb-449d-b9a3-2217f93e1fa6');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-4a4ec46c-e031-479a-becd-f20a0f04a5ae\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-4a4ec46c-e031-479a-becd-f20a0f04a5ae')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-4a4ec46c-e031-479a-becd-f20a0f04a5ae button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "                                                 query                  intent\n",
       "0    i'd like to make a reservation at rooth chris,...  restaurant_reservation\n",
       "1    make a reservation for chik-fil-a at 3 o' cloc...  restaurant_reservation\n",
       "2          reserve a table for 3 at 7 for olive garden  restaurant_reservation\n",
       "3    will you reserve a table at olive garden for 3...  restaurant_reservation\n",
       "4           at 7, i need a table for 3 at olive garden  restaurant_reservation\n",
       "..                                                 ...                     ...\n",
       "265                              i need a pasta recipe                  recipe\n",
       "266                i want a recipe for roasted veggies                  recipe\n",
       "267                        what is in a burrito recipe                  recipe\n",
       "268                        give me a tuna salad recipe                  recipe\n",
       "269                       find a recipe for baked ziti                  recipe\n",
       "\n",
       "[270 rows x 2 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test = pd.DataFrame(oos[\"test\"], columns=[\"query\", \"intent\"])\n",
    "\n",
    "df_test = df_test[df_test[\"intent\"].isin(intents)].reset_index(drop=True)\n",
    "\n",
    "df_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58b3ff86",
   "metadata": {
    "id": "58b3ff86"
   },
   "source": [
    "## Majority Vote\n",
    "\n",
    "As we have multiple labels per each task, we will *aggregate* the labels to get a single label per task. We will start with the simplest self-explanatory heuristic, which is called Majority Vote (MV)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9106e134",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9911108888666644"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agg_mv = MajorityVote().fit_predict(df_crowd_train)\n",
    "\n",
    "df_train_mv = pd.merge(df_train, agg_mv, left_on=\"query\", right_on=\"task\")\n",
    "\n",
    "f1_score(df_train_mv[\"intent\"], df_train_mv[\"agg_label\"], average=\"macro\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99635b5a",
   "metadata": {
    "id": "99635b5a"
   },
   "source": [
    "## Dawid-Skene\n",
    "\n",
    "Let's compare with the more specific probabilistic method called the Dawid-Skene model (DS)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d8a22c3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9866875576446533"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agg_ds = DawidSkene().fit_predict(df_crowd_train)\n",
    "\n",
    "df_train_ds = pd.merge(df_train, agg_ds, left_on=\"query\", right_on=\"task\")\n",
    "\n",
    "f1_score(df_train_ds[\"intent\"], df_train_ds[\"agg_label\"], average=\"macro\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05ec221a",
   "metadata": {
    "id": "05ec221a"
   },
   "source": [
    "## GLAD\n",
    "\n",
    "And finally, let's try one more probabilistic model for aggregation, called GLAD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "87c3645f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9933104421553267"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agg_glad_train = GLAD().fit_predict(df_crowd_train)\n",
    "\n",
    "df_train_glad = pd.merge(df_train, agg_glad_train, left_on=\"query\", right_on=\"task\")\n",
    "\n",
    "f1_score(df_train_glad[\"intent\"], df_train_glad[\"agg_label\"], average=\"macro\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tqednYA8jCAM",
   "metadata": {
    "id": "tqednYA8jCAM"
   },
   "source": [
    "As GLAD outperformed other models on `train`, we will use it to aggregate the answers. Let's first look how well it performs on the `test` subset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2CqJUsVOjIoN",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9850482025641298"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agg_glad_test = GLAD().fit_predict(df_crowd_test)\n",
    "\n",
    "df_test_glad = pd.merge(df_test, agg_glad_test, left_on=\"query\", right_on=\"task\")\n",
    "\n",
    "f1_score(df_test_glad[\"intent\"], df_test_glad[\"agg_label\"], average=\"macro\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "875e4233",
   "metadata": {
    "id": "875e4233"
   },
   "source": [
    "# Learning from Crowds\n",
    "\n",
    "Now let's train a machine learning model based on our crowdsourced data. We will need some boilerplate code for tranforming identifiers of labels and annotators into one-hot vectors as well as preparing data for PyTorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "GtBA-CmRfxl6",
   "metadata": {},
   "outputs": [],
   "source": [
    "ohe = LabelBinarizer().fit(list(intents))\n",
    "enc = LabelEncoder().fit(df_crowd_train[\"worker\"].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "QNHJ3IoTcXOz",
   "metadata": {
    "id": "QNHJ3IoTcXOz"
   },
   "source": [
    "This class, `CLINC150Dataset`, outputs pairs of $(\\text{query}, \\text{intent})$ from our dataset, where $\\text{intent}$ is the target variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "VR7b_vObxtIE",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CLINC150Dataset(Dataset):\n",
    "    def __init__(self, data, transform=None):\n",
    "        self.data = data\n",
    "\n",
    "        # We will use the transform() function to embed the query.\n",
    "        self.transform = transform\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        row = self.data.loc[idx]\n",
    "\n",
    "        x, y = row[\"query\"], row[\"intent\"]\n",
    "\n",
    "        if self.transform:\n",
    "            x = self.transform(x)\n",
    "\n",
    "        return torch.tensor(x), torch.tensor(ohe.transform([y])[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pY3yCtBKcjwz",
   "metadata": {
    "id": "pY3yCtBKcjwz"
   },
   "source": [
    "This class, `CLINC150CrowdDataset`, outputs triples of $(\\text{query}, \\text{worker}, \\text{intent})$ from our dataset, where $\\text{worker}$ is the annotator identifier and $\\text{intent}$ is the target variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "A4Xs6KDrPAKw",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CLINC150CrowdDataset(Dataset):\n",
    "    def __init__(self, data, transform=None):\n",
    "        self.data = data\n",
    "\n",
    "        # We will use the transform() function to embed the query.\n",
    "        self.transform = transform\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        row = self.data.loc[idx]\n",
    "\n",
    "        x, w, y = row[\"task\"], row[\"worker\"], row[\"label\"]\n",
    "\n",
    "        if self.transform:\n",
    "            x = self.transform(x)\n",
    "\n",
    "        return (\n",
    "            torch.tensor(x),\n",
    "            torch.tensor(enc.transform([w])[0]),\n",
    "            torch.tensor(ohe.transform([y])[0], dtype=torch.float32),\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "DUOvbFlbc5JE",
   "metadata": {
    "id": "DUOvbFlbc5JE"
   },
   "source": [
    "We will use the popular [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) sentence-transformers model:\n",
    "\n",
    "> It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "QepDXL4Ic3y1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n",
      "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
      "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
      "You will be able to reuse this secret in all of your notebooks.\n",
      "Please note that authentication is recommended but still optional to access public models or datasets.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "01b2615ebe1d4347960aaae2941f0901",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "modules.json:   0%|          | 0.00/349 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9804d335872b45938fdd3575753f7215",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config_sentence_transformers.json:   0%|          | 0.00/116 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e0990c3f69b44a5581e8c832781dc21f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "README.md:   0%|          | 0.00/10.6k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1324e91895e24d7f8e48b1413de3d606",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "sentence_bert_config.json:   0%|          | 0.00/53.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9802392bde2f46909a8fb7068539de5b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/612 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fad3cdb6b879437bb600e610d6c4d010",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "pytorch_model.bin:   0%|          | 0.00/90.9M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d1a1f86b14aa465885b5ba3f57c866da",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/350 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7385dfc58b0f4d1cb4fccee49ecaad1c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3167d9ee6bd944e9a3b12db8132743b5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "152db7d555b949e5946364c7d1f9e546",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "285defabeb404d1eb033830d78fd0c15",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "1_Pooling/config.json:   0%|          | 0.00/190 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "encoder = SentenceTransformer(\"all-MiniLM-L6-v2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "kDpdFE-XdO-O",
   "metadata": {
    "id": "kDpdFE-XdO-O"
   },
   "source": [
    "And load the data, configuring the loaders to use this pre-trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "Q-POuIVJwoMu",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = CLINC150Dataset(df_train, transform=encoder.encode)\n",
    "crowd_train_dataset = CLINC150CrowdDataset(df_crowd_train, transform=encoder.encode)\n",
    "test_dataset = CLINC150Dataset(df_test, transform=encoder.encode)\n",
    "aggregated_dataset = CLINC150Dataset(\n",
    "    df_train_glad[[\"query\", \"intent\"]], transform=encoder.encode\n",
    ")\n",
    "\n",
    "train_loader = utils.data.DataLoader(train_dataset, batch_size=64)\n",
    "crowd_train_loader = utils.data.DataLoader(crowd_train_dataset, batch_size=64)\n",
    "test_loader = utils.data.DataLoader(test_dataset, batch_size=64)\n",
    "aggregated_loader = utils.data.DataLoader(aggregated_dataset, batch_size=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Gc0RwjK3d7IG",
   "metadata": {
    "id": "Gc0RwjK3d7IG"
   },
   "source": [
    "We will use the following model architecture for classification:\n",
    "\n",
    "1. Input\n",
    "2. Query embedding using the pre-trained model (`all-MiniLM-L6-v2`, 384 dimensions)\n",
    "3. Fully-connected layer (384 dimensions → 9 dimensions)\n",
    "4. Optional during training: CrowdLayer or CoNAL (9 dimensions → 9 dimensions)\n",
    "\n",
    "We use the cross-entropy loss with five epochs in most cases. During training, CrowdLayer and CoNAL will need annotator identifiers, but these layers are omitted during inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "IlyvdQfmis-n",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CLINC150Classifier(L.LightningModule):\n",
    "    def __init__(self, n_intents, layer=None, n_workers=None):\n",
    "        super().__init__()\n",
    "\n",
    "        L.seed_everything(0)\n",
    "\n",
    "        self.W = nn.Linear(encoder.get_sentence_embedding_dimension(), n_intents)\n",
    "\n",
    "        self.f1 = torchmetrics.F1Score(task=\"multiclass\", num_classes=n_intents)\n",
    "\n",
    "        self.layer = layer\n",
    "\n",
    "        if layer == \"CrowdLayer\":\n",
    "            self.CL = CrowdLayer(n_intents, n_workers, conn_type=\"mw\")\n",
    "        elif layer == \"CoNAL\":\n",
    "            self.CL = CoNAL(n_intents, n_workers)\n",
    "        else:\n",
    "            self.CL = None\n",
    "\n",
    "    def forward(self, x, w=None):\n",
    "        emb = x.clone()\n",
    "\n",
    "        x = self.W(x)\n",
    "\n",
    "        if self.layer and w is not None:\n",
    "            if self.layer == \"CrowdLayer\":\n",
    "                x = self.CL(x, w)\n",
    "            elif self.layer == \"CoNAL\":\n",
    "                x = self.CL(emb, x, w)\n",
    "\n",
    "        x = x.to(torch.float)\n",
    "\n",
    "        return x\n",
    "\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        if len(batch) == 3:\n",
    "            x, w, y = batch\n",
    "            x = self.forward(x, w)\n",
    "        else:\n",
    "            x, y = batch\n",
    "            x = self.forward(x)\n",
    "\n",
    "        y = y.to(torch.float)\n",
    "\n",
    "        loss = torch.nn.functional.cross_entropy(x, y)\n",
    "\n",
    "        self.log(\"train_loss\", loss, on_step=True, on_epoch=True, prog_bar=True)\n",
    "\n",
    "        return loss\n",
    "\n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        x, y = batch\n",
    "\n",
    "        self.f1(torch.argmax(self.forward(x), dim=-1), torch.argmax(y, dim=-1))\n",
    "\n",
    "        self.log(\"val_f1\", self.f1, on_epoch=True, prog_bar=True)\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        return torch.optim.Adam(self.parameters(), lr=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lzsd4Bvtywi9",
   "metadata": {
    "id": "lzsd4Bvtywi9"
   },
   "source": [
    "## Training on Aggregated Data\n",
    "\n",
    "First, we will aggregate our annotated `train` subset using the GLAD model, and evaluate on the `test` ground truth dataset. We will do the same evaluation protocol for all models in our example. Since aggregation reduced our dataset times thrice, we'll use a proportionally higher number of epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "rAZkKx6Yyw6h",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: Seed set to 0\n",
      "INFO:lightning.fabric.utilities.seed:Seed set to 0\n",
      "INFO: GPU available: True (cuda), used: True\n",
      "INFO:lightning.pytorch.utilities.rank_zero:GPU available: True (cuda), used: True\n",
      "INFO: TPU available: False, using: 0 TPU cores\n",
      "INFO:lightning.pytorch.utilities.rank_zero:TPU available: False, using: 0 TPU cores\n",
      "INFO: IPU available: False, using: 0 IPUs\n",
      "INFO:lightning.pytorch.utilities.rank_zero:IPU available: False, using: 0 IPUs\n",
      "INFO: HPU available: False, using: 0 HPUs\n",
      "INFO:lightning.pytorch.utilities.rank_zero:HPU available: False, using: 0 HPUs\n",
      "WARNING: Missing logger folder: /content/lightning_logs\n",
      "WARNING:lightning.pytorch.loggers.tensorboard:Missing logger folder: /content/lightning_logs\n",
      "INFO: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "INFO:lightning.pytorch.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "INFO: \n",
      "  | Name | Type              | Params\n",
      "-------------------------------------------\n",
      "0 | W    | Linear            | 3.5 K \n",
      "1 | f1   | MulticlassF1Score | 0     \n",
      "-------------------------------------------\n",
      "3.5 K     Trainable params\n",
      "0         Non-trainable params\n",
      "3.5 K     Total params\n",
      "0.014     Total estimated model params size (MB)\n",
      "INFO:lightning.pytorch.callbacks.model_summary:\n",
      "  | Name | Type              | Params\n",
      "-------------------------------------------\n",
      "0 | W    | Linear            | 3.5 K \n",
      "1 | f1   | MulticlassF1Score | 0     \n",
      "-------------------------------------------\n",
      "3.5 K     Trainable params\n",
      "0         Non-trainable params\n",
      "3.5 K     Total params\n",
      "0.014     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "70775602efdb42dbb11c4de5f434ce2a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Sanity Checking: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b8cd64e33334fbb8a75844d271e09c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "029c55bfeefe416697a7ea6842aa0260",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8c125f8388ff4524a8497105d0c9bd9b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d3971caaa28c4e5784cb707a64fe4aa5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bd05bd34ab0545aca7d9436b374ba7ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a01f4786d8424642981b45c8eda053dc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a3d867d82261409c9a41eda496f51626",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4cd73be56a9e4ff58dcc43a3ea9fb157",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7cbca250e5a74031921d976c0b6aba81",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cb6880d353e6444ca8609bb5dce972c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c3e8d33adc7848939f6c5ca7a5851e36",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f685beee239c413e990d7226987cd803",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "87519f2378ed47b784d8d9bf7b6ef6bf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8307356e4a3a4a82a87d8fc6de864ccb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fc492bee5e5545948612947aec50e02c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5ca33cc523e84de1b51541e37af031cc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: `Trainer.fit` stopped: `max_epochs=15` reached.\n",
      "INFO:lightning.pytorch.utilities.rank_zero:`Trainer.fit` stopped: `max_epochs=15` reached.\n",
      "INFO: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "INFO:lightning.pytorch.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d343fdef67e64cec87f697c453b292ac",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">      Validate metric      </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">          val_f1           </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9777777791023254     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m     Validate metric     \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m         val_f1          \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9777777791023254    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'val_f1': 0.9777777791023254}]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = CLINC150Classifier(n_intents=len(intents))\n",
    "\n",
    "trainer = L.Trainer(max_epochs=5 * 3, log_every_n_steps=1)\n",
    "trainer.fit(\n",
    "    model=model, train_dataloaders=aggregated_loader, val_dataloaders=test_loader\n",
    ")\n",
    "\n",
    "trainer.validate(model, test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "nzDhAMrQyUxd",
   "metadata": {
    "id": "nzDhAMrQyUxd"
   },
   "source": [
    "## Training on Raw Data\n",
    "\n",
    "Instead, we can omit the aggregation step and train our model on the entire raw annotated `train` dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0R9zSVUao7_7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: Seed set to 0\n",
      "INFO:lightning.fabric.utilities.seed:Seed set to 0\n",
      "INFO: GPU available: True (cuda), used: True\n",
      "INFO:lightning.pytorch.utilities.rank_zero:GPU available: True (cuda), used: True\n",
      "INFO: TPU available: False, using: 0 TPU cores\n",
      "INFO:lightning.pytorch.utilities.rank_zero:TPU available: False, using: 0 TPU cores\n",
      "INFO: IPU available: False, using: 0 IPUs\n",
      "INFO:lightning.pytorch.utilities.rank_zero:IPU available: False, using: 0 IPUs\n",
      "INFO: HPU available: False, using: 0 HPUs\n",
      "INFO:lightning.pytorch.utilities.rank_zero:HPU available: False, using: 0 HPUs\n",
      "INFO: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "INFO:lightning.pytorch.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "INFO: \n",
      "  | Name | Type              | Params\n",
      "-------------------------------------------\n",
      "0 | W    | Linear            | 3.5 K \n",
      "1 | f1   | MulticlassF1Score | 0     \n",
      "-------------------------------------------\n",
      "3.5 K     Trainable params\n",
      "0         Non-trainable params\n",
      "3.5 K     Total params\n",
      "0.014     Total estimated model params size (MB)\n",
      "INFO:lightning.pytorch.callbacks.model_summary:\n",
      "  | Name | Type              | Params\n",
      "-------------------------------------------\n",
      "0 | W    | Linear            | 3.5 K \n",
      "1 | f1   | MulticlassF1Score | 0     \n",
      "-------------------------------------------\n",
      "3.5 K     Trainable params\n",
      "0         Non-trainable params\n",
      "3.5 K     Total params\n",
      "0.014     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d0f11ad4b4064c94a9caf20bebf0cd39",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Sanity Checking: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "317db930b10246bda53f5b7420bb5ddc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5fd31eb6977049be95c91da1d050fbe2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a2c35b0891cf47598f487187841480d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "495e36c884d346318eefa1272f6be36e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "42dd2d45bf8f4bf0928bb143a3eabc1c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d2ef40b868be4b55ab3e0d4347d9f1dd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: `Trainer.fit` stopped: `max_epochs=5` reached.\n",
      "INFO:lightning.pytorch.utilities.rank_zero:`Trainer.fit` stopped: `max_epochs=5` reached.\n",
      "INFO: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "INFO:lightning.pytorch.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7aa2e03c4985475aa61df238bae00288",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">      Validate metric      </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">          val_f1           </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9740740656852722     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m     Validate metric     \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m         val_f1          \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9740740656852722    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'val_f1': 0.9740740656852722}]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = CLINC150Classifier(n_intents=len(intents))\n",
    "\n",
    "trainer = L.Trainer(max_epochs=5, log_every_n_steps=1)\n",
    "trainer.fit(\n",
    "    model=model, train_dataloaders=crowd_train_loader, val_dataloaders=test_loader\n",
    ")\n",
    "\n",
    "trainer.validate(model, test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1V744e7KyknB",
   "metadata": {
    "id": "1V744e7KyknB"
   },
   "source": [
    "## Training with CrowdLayer\n",
    "\n",
    "CrowdLayer is a method that learns the confusion matrix $\\mathbf{A}_w$ of every annotator $w$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5O56M8xo02Sr",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: Seed set to 0\n",
      "INFO:lightning.fabric.utilities.seed:Seed set to 0\n",
      "INFO: GPU available: True (cuda), used: True\n",
      "INFO:lightning.pytorch.utilities.rank_zero:GPU available: True (cuda), used: True\n",
      "INFO: TPU available: False, using: 0 TPU cores\n",
      "INFO:lightning.pytorch.utilities.rank_zero:TPU available: False, using: 0 TPU cores\n",
      "INFO: IPU available: False, using: 0 IPUs\n",
      "INFO:lightning.pytorch.utilities.rank_zero:IPU available: False, using: 0 IPUs\n",
      "INFO: HPU available: False, using: 0 HPUs\n",
      "INFO:lightning.pytorch.utilities.rank_zero:HPU available: False, using: 0 HPUs\n",
      "INFO: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "INFO:lightning.pytorch.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "INFO: \n",
      "  | Name | Type              | Params\n",
      "-------------------------------------------\n",
      "0 | W    | Linear            | 3.5 K \n",
      "1 | f1   | MulticlassF1Score | 0     \n",
      "2 | CL   | CrowdLayer        | 4.5 K \n",
      "-------------------------------------------\n",
      "7.9 K     Trainable params\n",
      "0         Non-trainable params\n",
      "7.9 K     Total params\n",
      "0.032     Total estimated model params size (MB)\n",
      "INFO:lightning.pytorch.callbacks.model_summary:\n",
      "  | Name | Type              | Params\n",
      "-------------------------------------------\n",
      "0 | W    | Linear            | 3.5 K \n",
      "1 | f1   | MulticlassF1Score | 0     \n",
      "2 | CL   | CrowdLayer        | 4.5 K \n",
      "-------------------------------------------\n",
      "7.9 K     Trainable params\n",
      "0         Non-trainable params\n",
      "7.9 K     Total params\n",
      "0.032     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8347abcbf1434d2db9038253e2dd764f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Sanity Checking: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "44d35eda605344d79a5e625fb62b77d0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c5e275ad6eb24d1590da407413db61b5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f99e4d04ae94cb88b12270ac496ea89",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "04e08ffa97ed43ae9e6962602bc642a8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "66a0c3a4fbe04682a9c0412d1855ae86",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a6c9dd94d0624a008c3fb8bfbd0184b0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: `Trainer.fit` stopped: `max_epochs=5` reached.\n",
      "INFO:lightning.pytorch.utilities.rank_zero:`Trainer.fit` stopped: `max_epochs=5` reached.\n",
      "INFO: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "INFO:lightning.pytorch.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b9a3dc7bdb254e9b9aa7e01dba46c3ab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">      Validate metric      </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">          val_f1           </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9740740656852722     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m     Validate metric     \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m         val_f1          \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9740740656852722    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'val_f1': 0.9740740656852722}]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = CLINC150Classifier(\n",
    "    n_intents=len(intents),\n",
    "    layer=\"CrowdLayer\",\n",
    "    n_workers=df_crowd_train[\"worker\"].nunique(),\n",
    ")\n",
    "\n",
    "trainer = L.Trainer(max_epochs=5, log_every_n_steps=1)\n",
    "trainer.fit(\n",
    "    model=model, train_dataloaders=crowd_train_loader, val_dataloaders=test_loader\n",
    ")\n",
    "trainer.validate(model, test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2onYAVufGuyi",
   "metadata": {
    "id": "2onYAVufGuyi"
   },
   "source": [
    "## Training with CoNAL\n",
    "\n",
    "CoNAL is a method that learns annotator-specific confusion matrices $\\mathbf{A}_w$ and one common confusion matrix $\\mathbf{A}_g$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f9k4F40TGx2K",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: Seed set to 0\n",
      "INFO:lightning.fabric.utilities.seed:Seed set to 0\n",
      "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.\n",
      "  warnings.warn('Lazy modules are a new feature under heavy development '\n",
      "INFO: GPU available: True (cuda), used: True\n",
      "INFO:lightning.pytorch.utilities.rank_zero:GPU available: True (cuda), used: True\n",
      "INFO: TPU available: False, using: 0 TPU cores\n",
      "INFO:lightning.pytorch.utilities.rank_zero:TPU available: False, using: 0 TPU cores\n",
      "INFO: IPU available: False, using: 0 IPUs\n",
      "INFO:lightning.pytorch.utilities.rank_zero:IPU available: False, using: 0 IPUs\n",
      "INFO: HPU available: False, using: 0 HPUs\n",
      "INFO:lightning.pytorch.utilities.rank_zero:HPU available: False, using: 0 HPUs\n",
      "INFO: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "INFO:lightning.pytorch.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/utilities/model_summary/model_summary.py:452: A layer with UninitializedParameter was found. Thus, the total number of parameters detected may be inaccurate.\n",
      "INFO: \n",
      "  | Name | Type              | Params\n",
      "-------------------------------------------\n",
      "0 | W    | Linear            | 3.5 K \n",
      "1 | f1   | MulticlassF1Score | 0     \n",
      "2 | CL   | CoNAL             | 11.3 K\n",
      "-------------------------------------------\n",
      "11.7 K    Trainable params\n",
      "3.0 K     Non-trainable params\n",
      "14.7 K    Total params\n",
      "0.059     Total estimated model params size (MB)\n",
      "INFO:lightning.pytorch.callbacks.model_summary:\n",
      "  | Name | Type              | Params\n",
      "-------------------------------------------\n",
      "0 | W    | Linear            | 3.5 K \n",
      "1 | f1   | MulticlassF1Score | 0     \n",
      "2 | CL   | CoNAL             | 11.3 K\n",
      "-------------------------------------------\n",
      "11.7 K    Trainable params\n",
      "3.0 K     Non-trainable params\n",
      "14.7 K    Total params\n",
      "0.059     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ee52fff784b3403db4d3f2061a725556",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Sanity Checking: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f798b3c48e64ebaa9b7d22033f59413",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ad893670d3204387b2d679db98125bcb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "24da35d0819249798b05aa34d4160425",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ca1ccb91a8af4119b08cd215bd70767d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "69159d53df7045cfa09815ad1cd3bfe7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b202bc0caa7f4018a3de6f0176073ebf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: `Trainer.fit` stopped: `max_epochs=5` reached.\n",
      "INFO:lightning.pytorch.utilities.rank_zero:`Trainer.fit` stopped: `max_epochs=5` reached.\n",
      "INFO: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "INFO:lightning.pytorch.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eea83fe8329246c3b9b92f724594da54",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">      Validate metric      </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">          val_f1           </span>│<span style=\"color: #800080; text-decoration-color: #800080\">     0.970370352268219     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m     Validate metric     \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m         val_f1          \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m    0.970370352268219    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'val_f1': 0.970370352268219}]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = CLINC150Classifier(\n",
    "    n_intents=len(intents), layer=\"CoNAL\", n_workers=df_crowd_train[\"worker\"].nunique()\n",
    ")\n",
    "\n",
    "trainer = L.Trainer(max_epochs=5, log_every_n_steps=1)\n",
    "trainer.fit(\n",
    "    model=model, train_dataloaders=crowd_train_loader, val_dataloaders=test_loader\n",
    ")\n",
    "trainer.validate(model, test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "uZ4SE4JWhaZj",
   "metadata": {
    "id": "uZ4SE4JWhaZj"
   },
   "source": [
    "Thank you! This is how one can use [Crowd-Kit](https://github.com/Toloka/crowd-kit) to learn from crowdsourced and noisy-labeled data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}