demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Video reading benchmark demo\n",
    "\n",
    "First let's start with a single video reads; \n",
    "here we demonstrate the speeds of various video readers. \n",
    "There are many cells with little text, but it should be self evident which is which."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_video = \"./videos/original/WUzgd7C1pWA.mp4\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import av\n",
    "images_av = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "292 ms ± 9.52 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "\n",
    "container = av.open(path_to_video)\n",
    "for frame in container.decode(video=0):\n",
    "    images_av.append(frame.to_image())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2616\n"
     ]
    }
   ],
   "source": [
    "print(len(images_av))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "images_cv2 = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "96.6 ms ± 2.01 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "\n",
    "cap = cv2.VideoCapture(path_to_video)\n",
    "while(cap.isOpened()):\n",
    "    ret, frame = cap.read()\n",
    "    if ret is True:\n",
    "        rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n",
    "        images_cv2.append(frame)\n",
    "    else:\n",
    "        break\n",
    "cap.release()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision\n",
    "torchvision.set_video_backend(\"video_reader\")\n",
    "tv_frames = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bjuncek/.conda/envs/vb/lib/python3.7/site-packages/torchvision-0.12.0a0+4bf6086-py3.7-linux-x86_64.egg/torchvision/io/_video_opt.py:455: UserWarning: The pts_unit 'pts' gives wrong results and will be removed in a follow-up version. Please use pts_unit 'sec'.\n",
      "  \"The pts_unit 'pts' gives wrong results and will be removed in a \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "213 ms ± 12.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "vframes, _, _ = torchvision.io.read_video(path_to_video)\n",
    "tv_frames.append(vframes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision.io import VideoReader\n",
    "tv_frames = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "238 ms ± 17.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "video = VideoReader(path_to_video)\n",
    "for frame in video:\n",
    "    tv_frames.append(frame['data'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'video': {'fps': [29.97002997002997], 'duration': [10.9109]},\n",
       " 'audio': {'framerate': [48000.0], 'duration': [10.9]}}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "video = VideoReader(path_to_video)\n",
    "video.get_metadata()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from decord import VideoReader, cpu\n",
    "images_dcrd = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "79.1 ms ± 1.15 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "#in memory\n",
    "with open(path_to_video, 'rb') as f:\n",
    "  vr = VideoReader(f, ctx=cpu(0))\n",
    "for i in range(len(vr)):\n",
    "    # the video reader will handle seeking and skipping in the most efficient manner\n",
    "    images_dcrd.append(vr[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_dcrd = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit\n",
    "# \"normal\"\n",
    "vr = VideoReader(path_to_video, ctx=cpu(0))\n",
    "for i in range(len(vr)):\n",
    "    # the video reader will handle seeking and skipping in the most efficient manner\n",
    "    images_dcrd.append(vr[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmark results \n",
    "\n",
    "|    Reader   | avg time (ms) | stddev |      notes      |\n",
    "|:-----------:|:-------------:|:------:|:---------------:|\n",
    "| torchvision |      228      |  8.07  |    read_video   |\n",
    "| torchvision |      242      |  29.3  | VideoReader API |\n",
    "|     pyav    |      292      |  4.96  |                 |\n",
    "|     cv2     |      96.6     |  1.47  |                 |\n",
    "|    decord   |      79.1     |   1.2  |    in-memory    |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GPU Contexts\n",
    "At the moment, only decord supports direct GPU encoding (with not insignificant build hassle).\n",
    "\n",
    "Having said that, installing decord from source against FFMPEG from conda_forge actually yielded performance benefits in the grand scheme of things (79 vs 101ms on average) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from decord import VideoReader, cpu, gpu\n",
    "images_dcrd = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "77.6 ms ± 6.09 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "# \"normal\"\n",
    "vr = VideoReader(path_to_video, ctx=cpu(0))\n",
    "for i in range(len(vr)):\n",
    "    # the video reader will handle seeking and skipping in the most efficient manner\n",
    "    images_dcrd.append(vr[i])\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_dcrd = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "79.3 ms ± 2.18 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "# \"normal\"\n",
    "vr = VideoReader(path_to_video, ctx=gpu(0))\n",
    "for i in range(len(vr)):\n",
    "    # the video reader will handle seeking and skipping in the most efficient manner\n",
    "    images_dcrd.append(vr[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Base results\n",
    "\n",
    "Interestingly, the GPU context didn't significantly improve the performance over single video; something documented in https://github.com/dmlc/decord/issues/106\n",
    "The authors argue that the benefit comes for a) different video encodings and b) with various bridges (e.g. copying the memory to GPU); I'm not sure if there is a neat way of platform-independent way of measuring this. \n",
    "\n",
    "\n",
    "Interesting observation was made by Mike (from pyAV):\n",
    "\n",
    "> There are some features we may elect to not implement because we don’t believe they fit the PyAV ethos. The only one that we’ve encountered so far is hardware decoding. The FFmpeg man page discusses the drawback of -hwaccel:\n",
    ">> Note that most acceleration methods are intended for playback and will not be faster than software decoding on modern CPUs. Additionally, ffmpeg will usually need to copy the decoded frames from the GPU memory into the system memory, resulting in further performance loss.\n",
    "\n",
    "Which means that, if we find a way to keep the frames in GPU memory, this would be beneficial, but otherwise it is rather useless. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}