clean/07_sizing_and_tta.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "! [ -e /content ] && pip install -Uqq fastbook\n",
    "import fastbook\n",
    "fastbook.setup_book()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from fastbook import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a State-of-the-Art Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imagenette"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.vision.all import *\n",
    "path = untar_data(URLs.IMAGENETTE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dblock = DataBlock(blocks=(ImageBlock(), CategoryBlock()),\n",
    "                   get_items=get_image_files,\n",
    "                   get_y=parent_label,\n",
    "                   item_tfms=Resize(460),\n",
    "                   batch_tfms=aug_transforms(size=224, min_scale=0.75))\n",
    "dls = dblock.dataloaders(path, bs=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = xresnet50(n_out=dls.c)\n",
    "learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), metrics=accuracy)\n",
    "learn.fit_one_cycle(5, 3e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x,y = dls.one_batch()\n",
    "x.mean(dim=[0,2,3]),x.std(dim=[0,2,3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dls(bs, size):\n",
    "    dblock = DataBlock(blocks=(ImageBlock, CategoryBlock),\n",
    "                   get_items=get_image_files,\n",
    "                   get_y=parent_label,\n",
    "                   item_tfms=Resize(460),\n",
    "                   batch_tfms=[*aug_transforms(size=size, min_scale=0.75),\n",
    "                               Normalize.from_stats(*imagenet_stats)])\n",
    "    return dblock.dataloaders(path, bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls = get_dls(64, 224)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x,y = dls.one_batch()\n",
    "x.mean(dim=[0,2,3]),x.std(dim=[0,2,3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = xresnet50(n_out=dls.c)\n",
    "learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), metrics=accuracy)\n",
    "learn.fit_one_cycle(5, 3e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Progressive Resizing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls = get_dls(128, 128)\n",
    "learn = Learner(dls, xresnet50(n_out=dls.c), loss_func=CrossEntropyLossFlat(), \n",
    "                metrics=accuracy)\n",
    "learn.fit_one_cycle(4, 3e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.dls = get_dls(64, 224)\n",
    "learn.fine_tune(5, 1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Time Augmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds,targs = learn.tta()\n",
    "accuracy(preds, targs).item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mixup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sidebar: Papers and Math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### End sidebar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "church = PILImage.create(get_image_files_sorted(path/'train'/'n03028079')[0])\n",
    "gas = PILImage.create(get_image_files_sorted(path/'train'/'n03425413')[0])\n",
    "church = church.resize((256,256))\n",
    "gas = gas.resize((256,256))\n",
    "tchurch = tensor(church).float() / 255.\n",
    "tgas = tensor(gas).float() / 255.\n",
    "\n",
    "_,axs = plt.subplots(1, 3, figsize=(12,4))\n",
    "show_image(tchurch, ax=axs[0]);\n",
    "show_image(tgas, ax=axs[1]);\n",
    "show_image((0.3*tchurch + 0.7*tgas), ax=axs[2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Label Smoothing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sidebar: Label Smoothing, the Paper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### End sidebar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questionnaire"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. What is the difference between ImageNet and Imagenette? When is it better to experiment on one versus the other?\n",
    "1. What is normalization?\n",
    "1. Why didn't we have to care about normalization when using a pretrained model?\n",
    "1. What is progressive resizing?\n",
    "1. Implement progressive resizing in your own project. Did it help?\n",
    "1. What is test time augmentation? How do you use it in fastai?\n",
    "1. Is using TTA at inference slower or faster than regular inference? Why?\n",
    "1. What is Mixup? How do you use it in fastai?\n",
    "1. Why does Mixup prevent the model from being too confident?\n",
    "1. Why does training with Mixup for five epochs end up worse than training without Mixup?\n",
    "1. What is the idea behind label smoothing?\n",
    "1. What problems in your data can label smoothing help with?\n",
    "1. When using label smoothing with five categories, what is the target associated with the index 1?\n",
    "1. What is the first step to take when you want to prototype quick experiments on a new dataset?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further Research\n",
    "\n",
    "1. Use the fastai documentation to build a function that crops an image to a square in each of the four corners, then implement a TTA method that averages the predictions on a center crop and those four crops. Did it help? Is it better than the TTA method of fastai?\n",
    "1. Find the Mixup paper on arXiv and read it. Pick one or two more recent articles introducing variants of Mixup and read them, then try to implement them on your problem.\n",
    "1. Find the script training Imagenette using Mixup and use it as an example to build a script for a long training on your own project. Execute it and see if it helps.\n",
    "1. Read the sidebar \"Label Smoothing, the Paper\", look at the relevant section of the original paper and see if you can follow it. Don't be afraid to ask for help!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "split_at_heading": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}