dataset_curation/Lab - Dataset Curation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fec18a04",
   "metadata": {},
   "source": [
    "# Lab -- Dataset Curation with Multiple Annotators\n",
    "\n",
    "Intended to accompany the lecture on Dataset Creation and Curation, this notebook contains exercises to analyze an existing classification dataset labeled by multiple annotators (e.g. collected via crowdsourcing).\n",
    "\n",
    "You may find it helpful to first install the following dependencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b07588b1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: cleanlab in /Users/jonas/virtual/dcaiclass/lib/python3.10/site-packages (2.2.0)\n",
      "Requirement already satisfied: tqdm>=4.53.0 in /Users/jonas/virtual/dcaiclass/lib/python3.10/site-packages (from cleanlab) (4.64.1)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /Users/jonas/virtual/dcaiclass/lib/python3.10/site-packages (from cleanlab) (2.2.0)\n",
      "Requirement already satisfied: numpy>=1.11.3 in /Users/jonas/virtual/dcaiclass/lib/python3.10/site-packages (from cleanlab) (1.24.1)\n",
      "Requirement already satisfied: scikit-learn>=0.18 in /Users/jonas/virtual/dcaiclass/lib/python3.10/site-packages (from cleanlab) (1.2.0)\n",
      "Requirement already satisfied: pandas>=1.0.0 in /Users/jonas/virtual/dcaiclass/lib/python3.10/site-packages (from cleanlab) (1.5.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/jonas/virtual/dcaiclass/lib/python3.10/site-packages (from pandas>=1.0.0->cleanlab) (2022.7)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /Users/jonas/virtual/dcaiclass/lib/python3.10/site-packages (from pandas>=1.0.0->cleanlab) (2.8.2)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/jonas/virtual/dcaiclass/lib/python3.10/site-packages (from scikit-learn>=0.18->cleanlab) (3.1.0)\n",
      "Requirement already satisfied: joblib>=1.1.1 in /Users/jonas/virtual/dcaiclass/lib/python3.10/site-packages (from scikit-learn>=0.18->cleanlab) (1.2.0)\n",
      "Requirement already satisfied: scipy>=1.3.2 in /Users/jonas/virtual/dcaiclass/lib/python3.10/site-packages (from scikit-learn>=0.18->cleanlab) (1.10.0)\n",
      "Requirement already satisfied: six>=1.5 in /Users/jonas/virtual/dcaiclass/lib/python3.10/site-packages (from python-dateutil>=2.8.1->pandas>=1.0.0->cleanlab) (1.16.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install cleanlab\n",
    "# We originally used the version: cleanlab==2.2.0\n",
    "# This automatically installs other required packages like numpy, pandas, sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7c2885ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b04a118",
   "metadata": {},
   "source": [
    "## Analyzing dataset labeled by multiple annotators\n",
    "\n",
    "We simulate a small classification dataset (3 classes, 2-dimensional features) with ground truth labels that are then hidden from our analysis. The analysis is conducted on labels from noisy annotators whose labels are derived from the ground truth labels, but with some probability of error in each annotated label where the probability is determined by the underlying quality of the annotator. In subsequent exercises, you should assume the ground truth labels and the true annotator qualities are unknown to you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ece750ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "## You don't need to understand this cell, it's just used for generating the dataset\n",
    "\n",
    "SEED = 123  # for reproducibility\n",
    "np.random.seed(seed=SEED)\n",
    "\n",
    "def make_data(sample_size = 300):\n",
    "    \"\"\" Produce a 3-class classification dataset with 2-dimensional features and multiple noisy annotations per example. \"\"\"\n",
    "    num_annotators=50  # total number of data annotators\n",
    "    class_frequencies = [0.5, 0.25, 0.25]\n",
    "    sizes=[int(np.ceil(freq*sample_size)) for freq in class_frequencies]  # number of examples belonging to each class\n",
    "    good_annotator_quality = 0.6\n",
    "    bad_annotator_quality = 0.3\n",
    "    \n",
    "    # Underlying statistics of the datset (unknown to you)\n",
    "    means=[[3, 2], [7, 7], [0, 8]]\n",
    "    covs=[[[5, -1.5], [-1.5, 1]], [[1, 0.5], [0.5, 4]], [[5, 1], [1, 5]]]\n",
    "    \n",
    "    m = len(means)  # number of classes\n",
    "    n = sum(sizes)\n",
    "    local_data = []\n",
    "    labels = []\n",
    "\n",
    "    # Generate features and true labels\n",
    "    for idx in range(m):\n",
    "        local_data.append(\n",
    "            np.random.multivariate_normal(mean=means[idx], cov=covs[idx], size=sizes[idx])\n",
    "        )\n",
    "        labels.append(np.array([idx for i in range(sizes[idx])]))\n",
    "    X_train = np.vstack(local_data)\n",
    "    true_labels_train = np.hstack(labels)\n",
    "\n",
    "    # Generate noisy labels from each annotator\n",
    "    s = pd.DataFrame(\n",
    "        np.vstack(\n",
    "            [\n",
    "                generate_noisy_labels(true_labels_train, good_annotator_quality)\n",
    "                if i < num_annotators - 10  # last 10 annotators are worse\n",
    "                else generate_noisy_labels(true_labels_train, bad_annotator_quality)\n",
    "                for i in range(num_annotators)\n",
    "            ]\n",
    "        ).transpose()\n",
    "    )\n",
    "\n",
    "    # Each annotator only labels approximately 10% of the dataset (unlabeled points represented with NaN)\n",
    "    s = s.apply(lambda x: x.mask(np.random.random(n) < 0.9)).astype(\"Int64\")\n",
    "    s.dropna(axis=1, how=\"all\", inplace=True)\n",
    "    s.columns = [\"A\" + str(i).zfill(4) for i in range(1, num_annotators+1)]\n",
    "    # Drop rows not annotated by anybody\n",
    "    row_NA_check = pd.notna(s).any(axis=1)\n",
    "    X_train = X_train[row_NA_check]\n",
    "    true_labels_train = true_labels_train[row_NA_check]\n",
    "    multiannotator_labels = s[row_NA_check].reset_index(drop=True)\n",
    "    # Shuffle the rows of the dataset\n",
    "    shuffled_indices = np.random.permutation(len(X_train))\n",
    "    return {\n",
    "        \"X_train\": X_train[shuffled_indices],\n",
    "        \"true_labels_train\": true_labels_train[shuffled_indices],\n",
    "        \"multiannotator_labels\": multiannotator_labels.iloc[shuffled_indices],\n",
    "    }\n",
    "\n",
    "def generate_noisy_labels(true_labels, annotator_quality):\n",
    "    \"\"\" Randomly flips each true label to a different class with probability that depends on annotator_quality. \"\"\"\n",
    "    n = len(true_labels)\n",
    "    m = np.max(true_labels) + 1  # number of classes\n",
    "    annotated_labels = np.random.randint(low=0, high=3, size=n)\n",
    "    correctly_labeled_indices = np.random.random(n) < annotator_quality\n",
    "    annotated_labels[correctly_labeled_indices] = true_labels[correctly_labeled_indices]\n",
    "    return annotated_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9c9e83da",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dict = make_data(sample_size = 300)\n",
    "\n",
    "X = data_dict[\"X_train\"]\n",
    "multiannotator_labels = data_dict[\"multiannotator_labels\"]\n",
    "true_labels = data_dict[\"true_labels_train\"] # used for comparing the accuracy of consensus labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21412429",
   "metadata": {},
   "source": [
    "Let's view the first few rows of the data used for this exercise. Here are the labels selected by each annotator for the first few examples. Here each example is a row, and the annotators are columns. Not all annotators labeled each example; valid class annotations from those that did label the example are integers (either 0, 1, or 2 for our 3 classes), and otherwise the annotation is left as `NA` if a particular annotator did not label a particular example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "aa80889d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A0001</th>\n",
       "      <th>A0002</th>\n",
       "      <th>A0003</th>\n",
       "      <th>A0004</th>\n",
       "      <th>A0005</th>\n",
       "      <th>A0006</th>\n",
       "      <th>A0007</th>\n",
       "      <th>A0008</th>\n",
       "      <th>A0009</th>\n",
       "      <th>A0010</th>\n",
       "      <th>...</th>\n",
       "      <th>A0041</th>\n",
       "      <th>A0042</th>\n",
       "      <th>A0043</th>\n",
       "      <th>A0044</th>\n",
       "      <th>A0045</th>\n",
       "      <th>A0046</th>\n",
       "      <th>A0047</th>\n",
       "      <th>A0048</th>\n",
       "      <th>A0049</th>\n",
       "      <th>A0050</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>247</th>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>2</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>...</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290</th>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>...</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>262</th>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>2</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>...</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>...</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>...</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 50 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     A0001  A0002  A0003  A0004  A0005  A0006  A0007  A0008  A0009  A0010  \\\n",
       "247   <NA>      2   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   \n",
       "290   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   \n",
       "262   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>      2   <NA>   <NA>   <NA>   \n",
       "182   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   \n",
       "143   <NA>   <NA>      0   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   \n",
       "\n",
       "     ...  A0041  A0042  A0043  A0044  A0045  A0046  A0047  A0048  A0049  A0050  \n",
       "247  ...   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>  \n",
       "290  ...   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>  \n",
       "262  ...   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>  \n",
       "182  ...   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>  \n",
       "143  ...   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>  \n",
       "\n",
       "[5 rows x 50 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multiannotator_labels.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2058d6fb",
   "metadata": {},
   "source": [
    "Here are the corresponding 2D data features for these examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f5a59e23",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.01592896, 10.62213634],\n",
       "       [-1.91393643,  6.53944268],\n",
       "       [ 0.55962291,  5.35885902],\n",
       "       [ 6.73677377,  5.02311322],\n",
       "       [ 6.95949986,  1.61434817]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f0f836e",
   "metadata": {},
   "source": [
    "### Train model with cross-validation\n",
    "\n",
    "In this exercise, we consider the simple K Nearest Neighbors classification model, which produces predicted class probabilities for a particular example via a (weighted) average of the labels of the K closest examples. We will train this model via cross-validation and use it to produce held-out predictions for each example in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "77527d5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "\n",
    "def train_model(labels_to_fit):\n",
    "    \"\"\" Trains a simple feedforward neural network model on the data features X with y = labels_to_fit, via cross-validation.\n",
    "        Returns out-of-sample predicted class probabilities for each example in the dataset\n",
    "        (from a copy of model that was never trained on this example).\n",
    "        Also evaluates the held-out class predictions against ground truth labels.\n",
    "    \"\"\"\n",
    "    num_crossval_folds = 5  # number of folds of cross-validation\n",
    "    # model = MLPClassifier(max_iter=1000, random_state=SEED)\n",
    "    model = KNeighborsClassifier(weights=\"distance\")\n",
    "    pred_probs = cross_val_predict(\n",
    "        estimator=model, X=X, y=labels_to_fit, cv=num_crossval_folds, method=\"predict_proba\"\n",
    "    )\n",
    "    class_predictions = np.argmax(pred_probs, axis=1)\n",
    "    held_out_accuracy = np.mean(class_predictions == true_labels)\n",
    "    print(f\"Accuracy of held-out model predictions against ground truth labels: {held_out_accuracy}\")\n",
    "    return pred_probs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4630fa59",
   "metadata": {},
   "source": [
    "Here we demonstrate how to train and evaluate this model. Note that the evaluation is against ground truth labels, which you wouldn't have in real applications, so this evaluation is just for demonstration purposes. We'll first fit this model using labels comprised of one randomly selected annotation for each example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0d6cde68",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of random annotators' labels against ground truth labels: 0.6822742474916388\n",
      "Accuracy of held-out model predictions against ground truth labels: 0.8093645484949833\n"
     ]
    }
   ],
   "source": [
    "labels_from_random_annotators = true_labels.copy()\n",
    "for i in range(len(multiannotator_labels)):\n",
    "    annotations_for_example_i = multiannotator_labels.iloc[i][pd.notna(multiannotator_labels.iloc[i])]\n",
    "    labels_from_random_annotators[i] = np.random.choice(annotations_for_example_i.values)\n",
    "\n",
    "print(f\"Accuracy of random annotators' labels against ground truth labels: {np.mean(labels_from_random_annotators == true_labels)}\")\n",
    "pred_probs_from_model_fit_to_random_annotators = train_model(labels_to_fit = labels_from_random_annotators)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a60403d0",
   "metadata": {},
   "source": [
    "We can also fit this model using the ground truth labels (which you would not be able to in practice), just to see how good it could be:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ee1a3d99",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of held-out model predictions against ground truth labels: 0.9765886287625418\n"
     ]
    }
   ],
   "source": [
    "pred_probs_from_unrealistic_model_fit_to_true_labels = train_model(labels_to_fit = true_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "262b36c4",
   "metadata": {},
   "source": [
    "## Exercise 1\n",
    "\n",
    "Compute majority-vote consensus labels for each example from the data in `multiannotator_labels`. Think about how to best break ties!\n",
    "\n",
    "- Evaluate the accuracy of these majority-vote consensus labels against the ground truth labels.\n",
    "- Also set these as `labels_to_fit` in `train_model()` to see the resulting model's accuracy when trained with majority vote consensus labels.\n",
    "- Estimate the quality of annotator (how accurate their labels tend to be overall) using only these majority-vote consensus labels (assume the ground truth labels are unavailable as they would be in practice). Who do you guess are the worst 10 annotators?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ee57297",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Code your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a38fb1a9",
   "metadata": {},
   "source": [
    "## Exercise 2\n",
    "\n",
    "Estimate consensus labels for each example from the data in `multiannotator_labels`, this time using the CROWDLAB algorithm. You may find it helpful to reference: https://docs.cleanlab.ai/stable/tutorials/multiannotator.html\n",
    "Recall that CROWDLAB requires out of sample predicted class probabilities from a trained classifier. You may use the `pred_probs` from your model trained on majority-vote consensus labels or our randomly-selected annotator labels. Which do you think would be better to use?\n",
    "\n",
    "- Evaluate the accuracy of these CROWDLAB consensus labels against the ground truth labels.\n",
    "- Also set these as `labels_to_fit` in `train_model()` to see the resulting model's accuracy when trained with CROWDLAB consensus labels.\n",
    "- Estimate the quality of annotator (how accurate their labels tend to be overall) using CROWDLAB (assume the ground truth labels are unavailable as they would be in practice). Who do you guess are the worst 10 annotators based on this method?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ae8771c",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Code your solution here"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dcaiclass",
   "language": "python",
   "name": "dcaiclass"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}