Write-Up Part II.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automatic Reporting in Python\n",
    "## Part 2 - From Hello World to Real Insights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As outlined in my previous post, the goal of this project is to make an automatic reporting tool.\n",
    "\n",
    "In this guide, the outcome I'm shooting for is a single HTML page that allows me to interrogate and compare the outputs of machine learnings models.\n",
    "\n",
    "<img src=\"Static/2_summary.png\">\n",
    "\n",
    "At the [conclusion of the previous tutorial](https://dev.to/goyder/automatic-reporting-in-python---part-1-from-planning-to-hello-world-32n1), we'd gotten up to the \"Hello, World!\" stage. Some very basic content could be fed into our very basic structure - but now it's time to add some real features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A note on tutorials and design choices\n",
    "\n",
    "Rather than just providing a dump of code that works, I'd like to incrementally work through the development, step-by-step. \n",
    "\n",
    "Personally, I've always found this kind of tutorial more effective as:\n",
    "\n",
    "* It gives an insight of how someone else approaches a problem\n",
    "* It actually reflects how I code - feature by feature, commit by commit.\n",
    "\n",
    "When I was totally fresh to programming, I worked from a large number of tutorials. The structure of most tutorials gives you the impression that there is one and only one way to accomplish the goal that the tutorial sets out to do - and, admittedly, sometimes there is, depending on how niche and technical the tutorial is. \n",
    "\n",
    "But more often I've found that any programming job, even the very small, requires choices and trade-offs to be made in the implementation of the solution from the get-go. Should this be a function or a class? Do I import a library or roll my own? Should I be using a dictionary or a list? What kind of flow-on effects are these choices going to have?\n",
    "\n",
    "<img src=\"Static/2_qmarks.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One approach to simplify things: mark your assumptions\n",
    "\n",
    "Now, there's obviously no simple solution to any of these challenge. However, one approach to making this process easier is being clear on the assumptions underpinning your project. \n",
    "\n",
    "For instance, in this project:\n",
    "* I am the only consumer of these reports, and therefore;\n",
    "* My users (me!) will not be hostile\n",
    "* A minimum of error-checking is required, as I can interrogate and troubleshoot myself\n",
    "* An internet connection is available\n",
    "* Portability is not required\n",
    "\n",
    "Most of these are simplifying assumptions that serve to make clear exactly how thorough or involved my solution needs to be. OF course, as soon as you start looking at providing your code to others, or in an enterprise situation, or if a higher degree of automation is required, the game can change significantly!\n",
    "\n",
    "It makes the ecosystem in which this tool will exist a bit clearer.  When I get stuck and I'm not sure where to go, I'll refer back to these assumptions and see if I've not answered my own question somewhere."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Step Three - Generate some data to work with\n",
    "\n",
    "As mentioned in [Part 1 of this Post](https://dev.to/goyder/automatic-reporting-in-python---part-1-from-planning-to-hello-world-32n1), there are a host of use-cases for this tool that come from the machine learning space. \n",
    "\n",
    "In this case, we'll consider the following scenario:\n",
    "* We have an existing, baseline model that determines whether an image contains a **cat**.\n",
    "* We regularly develop new models that carry out the same task. These models may or may not be an improvement on your baseline model.\n",
    "* Every time we develop a new model, we run it over an existing set of data and capture the outputs. In this case, the outputs are what the model believes to be in each image.\n",
    "\n",
    "It would therefore be convenient to be able to:\n",
    "\n",
    "2. Automatically review these outputs and get a status report on how the new model did.\n",
    "3. Automatically compare the performance of these models against each other. Where did one outperform the other? Are there images that both struggle with?\n",
    "\n",
    "For the purposes of this study, we'll assess the performance of two models in the well respected field of detecting cats in images. [**VGG19**](https://arxiv.org/abs/1409.1556), the first model, is a well respected and important image classifier. [**MobileNet**](https://arxiv.org/abs/1704.04861) is a significantly more lightweight architecture (running approximately six times faster than VGG19, for example) but with reduced accuracy. \n",
    "\n",
    "The use of these models to generate the data can be found under a separate repo, [available here](https://github.com/goyder/dataset_generation). While the process of generating this data is a bit beyond the scope of this discussion, the end-result is we have two datasets to play with - `VGG19_results.csv` and `MobileNet_results.csv` - each stored as [CSV files](https://en.wikipedia.org/wiki/Comma-separated_values). \n",
    "\n",
    "<img src=\"Static/2_model_output.png\">\n",
    "\n",
    "For each image in the dataset, we have a column for the predicted ImageNet category ID (`imagenet_index`), the text description of the category (`imagenet_category`), and whether this corresponds to a housecat (`correct`). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "\n",
    "## Step Four - Set up a structure to put things in \n",
    "\n",
    "Well - after our digressions - back to the reporter. When we left off, we had the reporter in a functional but featureless state:\n",
    "\n",
    "<img src=\"Static/ItWorks.png\">\n",
    "\n",
    "Let's make the next move by adding a basic stucture to the report that we can populate with more sections of content. First, we'll update `report.html`. We'll make some minor tweaks, like specifying an actual `title` point to be entered and adding a brief preamble, but the real interesting part is below that:\n",
    "\n",
    "**`report.html`**\n",
    "\n",
    "```html\n",
    "<!DOCTYPE html>\n",
    "<html lang=\"en\">\n",
    "<head>\n",
    "    <meta charset=\"UTF-8\">\n",
    "    <title>{{ title }}</title>\n",
    "</head>\n",
    "<body>\n",
    "<h1>{{ title }}</h1>\n",
    "<p>This report was automatically generated.</p>\n",
    "{% for section in sections %}\n",
    "    {{ section }}\n",
    "{% endfor %}\n",
    "</body>\n",
    "</html>\n",
    "```\n",
    "\n",
    "We're limited to inserting blocks of text into our template - Jinja allows control flow structures! In this case, we'll feed in some iterable variable `sections` - note the plural - and then we'll publish each `section`  object within.\n",
    "\n",
    "But what are we to include in these sections? As a first run, we can generate a section for each model that displays a brief bit of information about the model output, and then displays the results. Let's create a new template, `table_section.html`, \n",
    "that we'll use to render this information.\n",
    "\n",
    "**`table_section.html`**\n",
    "\n",
    "```html\n",
    "<h2>{{ model }} - Model Results</h2>\n",
    "<p>Results for each image as predicted by model <i>'{{ model }}'</i>, as captured in file <i>'{{ dataset }}'</i>.</p>\n",
    "{{ table }}\n",
    "```\n",
    "\n",
    "Nice and simple - this gives us all the structure we need.\n",
    "\n",
    "Before we get to inserting the tables into the document, let's see what this gives us if we populate it with some dummy data. We'll update `autoreporting.py` to a) refer to the new updated terms in the updated `report.html` template, and b) render and input some `sections` so we can check out the new template.\n",
    "\n",
    "**`autoreporting.py`**\n",
    "```python\n",
    "from jinja2 import FileSystemLoader, Environment\n",
    "\n",
    "# Configure Jinja and ready the loader\n",
    "env = Environment(\n",
    "    loader=FileSystemLoader(searchpath=\"templates\")\n",
    ")\n",
    "\n",
    "# Assemble the templates we'll use\n",
    "base_template = env.get_template(\"report.html\")\n",
    "table_section_template = env.get_template(\"table_section.html\")\n",
    "\n",
    "# Content to be published\n",
    "title = \"Model Report\"\n",
    "sections = list()\n",
    "sections.append(table_section_template.render(\n",
    "    model=\"VGG19\",\n",
    "    dataset=\"VGG19_results.csv\",\n",
    "    table=\"Table goes here.\"\n",
    "))\n",
    "sections.append(table_section_template.render(\n",
    "    model=\"MobileNet\",\n",
    "    dataset=\"MobileNet_results.csv\",\n",
    "    table=\"Table goes here.\"\n",
    "))\n",
    "\n",
    "\n",
    "def main():\n",
    "    \"\"\"\n",
    "    Entry point for the script.\n",
    "    Render a template and write it to file.\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    with open(\"outputs/report.html\", \"w\") as f:\n",
    "        f.write(base_template.render(\n",
    "            title=title,\n",
    "            sections=sections\n",
    "        ))\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n",
    "```\n",
    "\n",
    "When we run the script, the two sections we rendered with the `table_section.html` template will be included in our report, one after the other.\n",
    "\n",
    "<img src=\"Static/OutputWithoutTables.png\">\n",
    "\n",
    "Here, we only have two sections, and they're of the same kind - but we can imagine it'd be no concern to string together more models, or generate blocks that compare performance between models, to suggest just some examples.\n",
    "\n",
    "### GitHub status\n",
    "\n",
    "At this point, your project will look something like [this](https://github.com/goyder/autoreporting/tree/9ded9bf1867ba6e8149591d34a1525589f0d785e)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Step Five - Insert the tables into the document\n",
    "\n",
    "It's time to extract and insert our tables into the report. \n",
    "\n",
    "The basic approach I've suggested is to use [*pandas*](https://pandas.pydata.org/) to read in the `.csv` file as a DataFrame object (the workhorse class of *pandas*). From this DataFrame object, we can readily export it as HTML using the `to_html` method. Our modified version of `autoreporting.py` therefore looks as follows:\n",
    "\n",
    "**`autoreporting.py`**\n",
    "```python\n",
    "import pandas as pd\n",
    "from jinja2 import FileSystemLoader, Environment\n",
    "\n",
    "# Allow for very wide columns - otherwise columns are spaced and ellipse'd\n",
    "pd.set_option(\"display.max_colwidth\", 200)\n",
    "\n",
    "\n",
    "def csv_to_html(filepath):\n",
    "    \"\"\"\n",
    "    Open a .csv file and return it in HTML format.\n",
    "    :param filepath: Filepath to a .csv file to be read.\n",
    "    :return: String of HTML to be published.\n",
    "    \"\"\"\n",
    "    df = pd.read_csv(filepath, index_col=0)\n",
    "    html = df.to_html()\n",
    "    return html\n",
    "\n",
    "\n",
    "# Configure Jinja and ready the loader\n",
    "env = Environment(\n",
    "    loader=FileSystemLoader(searchpath=\"templates\")\n",
    ")\n",
    "\n",
    "# Assemble the templates we'll use\n",
    "base_template = env.get_template(\"report.html\")\n",
    "table_section_template = env.get_template(\"table_section.html\")\n",
    "\n",
    "\n",
    "def main():\n",
    "    \"\"\"\n",
    "    Entry point for the script.\n",
    "    Render a template and write it to file.\n",
    "    :return:\n",
    "    \"\"\"\n",
    "\n",
    "    # Content to be published\n",
    "    title = \"Model Report\"\n",
    "    sections = list()\n",
    "    sections.append(table_section_template.render(\n",
    "        model=\"VGG19\",\n",
    "        dataset=\"VGG19_results.csv\",\n",
    "        table=csv_to_html(\"datasets/VGG19_results.csv\")\n",
    "    ))\n",
    "    sections.append(table_section_template.render(\n",
    "        model=\"MobileNet\",\n",
    "        dataset=\"MobileNet_results.csv\",\n",
    "        table=csv_to_html(\"datasets/MobileNet_results.csv\")\n",
    "    ))\n",
    "\n",
    "    with open(\"outputs/report.html\", \"w\") as f:\n",
    "            f.write(base_template.render(\n",
    "                title=title,\n",
    "                sections=sections\n",
    "            ))\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n",
    "```\n",
    "\n",
    "The main thing that jumps out at me here is that this is starting to get fairly involved for a script. To keep things on the straight and narrow, I'm applying the following rough flow for ordering:\n",
    "1. Imports to start\n",
    "2. Run any configuration code required\n",
    "3. Define any functions required\n",
    "4. Run any standalone code\n",
    "5. `main()` function and call to run it at the end of the script. \n",
    "\n",
    "This raises the question: what should be standalone and what should go in `main()`? As a rule of thumb, code that we only want executed when we explicitly run the script goes in `main()`; code (like classes, definitions, objects) that we might like to use elsewhere go in the body, available for import elsewhere. You can find a good many explanations of this functionality around, but I find [this](https://stackoverflow.com/a/22493194) to be a thorough discussion of the concept.\n",
    "\n",
    "Some other minor points on the above:\n",
    "* Note the call to widen the `max_colwidth` setting via `set_option` for `pandas` at the start of the script. Without this, `pandas` will silently apply some funny formatting if any column exceeds fifty characters. Increasing the `max_colwidth` is a trick that has only been learnt through painful experience.\n",
    "* There's repetition as we call `table_section_template.render()` - note that the filename of the same file is entered manually twice, violating the principle of [Don't Repeat Yourself](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself). Would it be worthwhile to tweak this so the filename is automatically split out from the filepath?\n",
    "\n",
    "With this implemented, we can re-run the script, and voila - tables!\n",
    "\n",
    "<img src=\"static/OutputWithTables.png\">\n",
    "\n",
    "### GitHub status\n",
    "\n",
    "If you're playing at home, your project will resemble [this](https://github.com/goyder/autoreporting/tree/f64e986b0775df694b3969a7271b9b1c9814b613)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Step Six - Include a summary and comparison\n",
    "\n",
    "Well, we have the raw data displayed, but we haven't really answered our key question: which model performs better? Using our 'section' based approach, let's develop a new 'summary' section. \n",
    "\n",
    "Two questions that often come to mind when comparing models is:\n",
    "* How accurate is each model? \n",
    "* Are there patterns in how the models are inaccurate - in this instance, are there images that *all* the models get wrong? \n",
    "\n",
    "This moves beyond simply *displaying* the raw data: instead, we're starting to draw inferences and make summaries, marking a significant increasing in the complexity of the code. I find this situation tends to induce a kind of [analysis paralysis](https://en.wikipedia.org/wiki/Analysis_paralysis) if I'm not careful: I can spend too long thinking about how best to approach the problem to ensure the solution is neat and elegant. Do I dive straight into the code? Do I write a new template? How do I know if the code as it exists now is under or overcooked?\n",
    "\n",
    "One way to focus this process is to very clearly articulate our end-goal, allowing us to focus on figuring out the most effective way there. So, before diving straight back into the Python script, I spent five minutes scribbling down how I'd like this section to read, and settled on some explicit phrases that I was happy with. After some scribbling, I ended up with this target:\n",
    "\n",
    "**`Summary block prototype`**\n",
    "> ## Quick Summary\n",
    ">\n",
    "> ### Accuracy\n",
    ">\n",
    "> **Model 1** analysed **100** images, achieving an accuracy of **97.2%**.\n",
    ">\n",
    "> **Model 2** analysed **100** images, achieving an accuracy of **94.2%**.\n",
    "> \n",
    "> ### Trouble spots\n",
    ">\n",
    "> **Model 1** misidentified **10 images**.\n",
    "> \n",
    "> **Model 2** misidentified **12 images**.\n",
    ">\n",
    "> **8 images** misidentified images were common to all.\n",
    "\n",
    "This seemed straightforward, and *implies* a neat translation into `Jinja` template structuring - it makes the next step feel obvious. So with this target clearly in mind, I could easily produce the template: \n",
    "\n",
    "**`summary_section.html`**\n",
    "```html\n",
    "<h2>Quick summary</h2>\n",
    "<h3>Accuracy</h3>\n",
    "{% for model_results in model_results_list %}\n",
    "<p><em>{{ model_results.model_name }}</em> analysed <em>{{ model_results.number_of_images }} image(s)</em>, achieving an accuracy of <em>{{ \"{:.2%}\".format(model_results.accuracy) }}.</em></p>\n",
    "{% endfor %}\n",
    "<h3>Trouble spots</h3>\n",
    "{% for model_results in model_results_list %}\n",
    "<p><em>{{ model_results.model_name }}</em> misidentified <em>{{ model_results.number_misidentified }} image(s)</em>.</p>\n",
    "{% endfor %}\n",
    "<em>{{ number_misidentified }}</em> misidentified image(s) were common to all models.\n",
    "```\n",
    "\n",
    "This template assumes some that I'll produce a *list* of some objects `model_results`, which in turn contains properties like `model_name`, `number_of_images`, and `accuracy`. I'll also need some value `number_misidentified`, referring to the number of images misidentifed across all models: this doesn't quite mesh into a theoretical, singular `model_results` object, so it stands alone. (For now.)\n",
    "\n",
    "This seems plausible enough, and implies some useful information I can take back into the code, giving me a good structure to start with. If, as I start to work back into the code, I find that these early and rough requirements don't quite translate well or make things too difficult, I won't hesitate to feed design decisions *forward* and tweak them. However, in this instance, this looks to be fairly straightforward and I can't imagine any significant issues.\n",
    "\n",
    "With this in mind, I worked back into `autoreporting.py` and started making some changes. By this point I feel `autoreporting.py` is getting a bit long to be posting every time, so I'll work by exception from here on out: you can find this relevant commit [here](https://github.com/goyder/autoreporting/blob/4f79514660bcc34a44c01be3076f757f1371f098/autoreporting.py). \n",
    "\n",
    "Of perhaps greatest importance is the `model_results` object. I spent a little bit of time debating whether to implement this as a class, dictionary, or named tuple: in the end, I landed on a class. The primary reason for this was that I was ended up bundling the *data* and the *methods* in one: and when you have data and methods combined, a class is a good approach. \n",
    "\n",
    "**`autoreporting.py - ModelResults class`**\n",
    "```python\n",
    "# ...\n",
    "\n",
    "class ModelResults:\n",
    "    \"\"\"\n",
    "    Class to store the results of a model run and associated data.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, model_name, filepath):\n",
    "        \"\"\"\n",
    "        :param model_name: Name of model.\n",
    "        :param filepath: Filepath to results .csv.\n",
    "        \"\"\"\n",
    "        self.model_name = model_name\n",
    "        self.filepath = filepath\n",
    "        \n",
    "        self.dataset = os.path.split(filepath)[-1]\n",
    "        self.df_results = csv_to_df(filepath)  # Filesystem access\n",
    "        \n",
    "        self.number_of_images = len(self.df_results)\n",
    "        \n",
    "        self.accuracy = self._calculate_accuracy()\n",
    "        \n",
    "        self.misidentified_images = self._get_misidentified_images()\n",
    "        self.number_misidentified = len(self.misidentified_images)\n",
    "\n",
    "    # Class methods not shown... \n",
    "    \n",
    "# ...\n",
    "```\n",
    "\n",
    "Now, with this `ModelResults` class, we can call in our results files only once and have our relevant information readily available, solving the DRY problem mentioned back in Step Five.\n",
    "\n",
    "**`autoreporting.py - main() function`**\n",
    "```python\n",
    "# ...\n",
    "\n",
    "def main():\n",
    "    \"\"\"\n",
    "    Entry point for the script.\n",
    "    Render a template and write it to file.\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    # Content to be published\n",
    "    title = \"Model Report\"\n",
    "    vgg19_results = ModelResults(\"VGG19\", \"datasets/VGG19_results.csv\")\n",
    "    mobilenet_results = ModelResults(\"MobileNet\", \"datasets/MobileNet_results.csv\")\n",
    "    number_misidentified = len(set(vgg19_results.misidentified_images) & set(mobilenet_results.misidentified_images))\n",
    "\n",
    "    # Produce our section blocks\n",
    "    sections = list()\n",
    "    sections.append(summary_section_template.render(\n",
    "        model_results_list=[vgg19_results, mobilenet_results],\n",
    "        number_misidentified=number_misidentified\n",
    "    ))\n",
    "    sections.append(table_section_template.render(\n",
    "        model=vgg19_results.model_name,\n",
    "        dataset=vgg19_results.dataset,\n",
    "        table=vgg19_results.get_results_df_as_html())\n",
    "    )\n",
    "    sections.append(table_section_template.render(\n",
    "        model=mobilenet_results.model_name,\n",
    "        dataset=mobilenet_results.dataset,\n",
    "        table=mobilenet_results.get_results_df_as_html())\n",
    "    )\n",
    "\n",
    "    # Produce and write the report to file\n",
    "    with open(\"outputs/report.html\", \"w\") as f:\n",
    "        f.write(base_template.render(\n",
    "            title=title,\n",
    "            sections=sections\n",
    "        ))\n",
    "```\n",
    "\n",
    "Now, we'll run our revised `autoreporting.py`, and:\n",
    "\n",
    "<img src=\"Static/Step Six End.png\">\n",
    "\n",
    "A neat summary is what we get!\n",
    "\n",
    "From this output, we can see:\n",
    "\n",
    "* Both models achieve approximately the same level of accuracy. This is surprising in and off itself - I would have expected VGG19 to be far more accurate.\n",
    "* We have a modest amount of overlap in the misidentified images - about 2/3s are common between the models. This suggests that those images are particularly challenging in some way. It may be worthwhile digging into these and examining whether these images are flawed or unsuitable in some way.\n",
    "\n",
    "Or, to summarise the above dot points: we're actually getting useful insights from the report. Nice!\n",
    "\n",
    "`<Include an image here for pay-off. Summarise the process!>`\n",
    "\n",
    "### GitHub status\n",
    "\n",
    "At this point, the project will probably look like [this](https://github.com/goyder/autoreporting/tree/4f79514660bcc34a44c01be3076f757f1371f098)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "We have a functioning report that produces useful information! That's a good start. However, there are many things to look at next, such as:\n",
    "* The datasets that we want to run the report are currently hard-coded into the script. How can we turn this into a more useful, flexible tool?\n",
    "* It's hard to dig into the tables in any meaningful fashion - the output is very static. Can we improve the model such that we can filter and tweak the data?\n",
    "* The report looks *exceedingly* dull. How can we spruce it up?\n",
    "\n",
    "There's always more work to be done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}