Add info on pngcrush/mogrify to fix bad pngs

ThomasOli · Mar 14, 2022 · 55ce545 · 55ce545
1 parent cc37cbb
commit 55ce545
Show file tree

Hide file tree

Showing 2 changed files with 187 additions and 2,183 deletions.
diff --git a/18_Transfer_Learning.ipynb b/18_Transfer_Learning.ipynb
@@ -591,6 +591,14 @@
     "    print(images[i].shape)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "1dcb98d9-c23b-49c8-86c5-6fb58dad8207",
+   "metadata": {},
+   "source": [
+    "## Step 6. Fit the model"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -704,6 +712,53 @@
     "labels['00000061_003.png']"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "722848b8-06ba-4ba6-b01f-15d847669273",
+   "metadata": {},
+   "source": [
+    "### Fix image errors\n",
+    "\n",
+    "Using the full dataset, there were two images that were porly formatted. These would throw the error:\n",
+    "\n",
+    " `W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: profile 'ICC Profile': 'GRAY': Gray color space not permitted on RGB PNG`\n",
+    "\n",
+    "To fix this, I found [this post](https://stackoverflow.com/questions/22745076/libpng-warning-iccp-known-incorrect-srgb-profile) that suggested identifying the problematic images with [`pngcrush`](https://pmt.sourceforge.io/pngcrush/) and fixing them with `mogrify` from [`imagemagick`](https://imagemagick.org/index.php).\n",
+    "\n",
+    "There were too many images in the main `images` folder to process all at once, so I used the folders split by diagnosis:\n",
+    "\n",
+    "```bash\n",
+    "[magitz@c0903a-s35 pcr010813]$ for dir in `ls -d ../../share/ChestXray-NIHCC/images_14_cat/*`; do echo \"Working on $dir; ./pngcrush -warn -n ${dir}/*.png; done\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Atelectasis\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Cardiomegaly\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Consolidation\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Edema\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Effusion\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Emphysema\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Fibrosis\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Hernia\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Infiltration\n",
+    "../../share/ChestXray-NIHCC/images_14_cat/Infiltration/00002846_012.png: iCCP: profile 'ICC Profile': 'GRAY': Gray color space not permitted on RGB PNG\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Mass\n",
+    "../../share/ChestXray-NIHCC/images_14_cat/Mass/00000004_000.png: iCCP: profile 'ICC Profile': 'GRAY': Gray color space not permitted on RGB PNG\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Nodule\n",
+    "../../share/ChestXray-NIHCC/images_14_cat/Nodule/00000004_000.png: iCCP: profile 'ICC Profile': 'GRAY': Gray color space not permitted on RGB PNG\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/No_Finding\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Pleural_Thickening\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Pneumonia\n",
+    "Working on ../../share/ChestXray-NIHCC/images_14_cat/Pneumothorax\n",
+    "[magitz@c0903a-s35 pcr010813]$\n",
+    "```\n",
+    "\n",
+    "So, images `00002846_012.png` and `00000004_000.png` are the culpets. Fixing them with:\n",
+    "   \n",
+    "```bash\n",
+    "[magitz@c0903a-s35 pcr010813]$ mogrify ../../share/ChestXray-NIHCC/images/00002846_012.png\n",
+    "[magitz@c0903a-s35 pcr010813]$ mogrify ../../share/ChestXray-NIHCC/images/00000004_000.png\n",
+    "```\n",
+    "Seems to have fixed things..."
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "a08d8ee6-72a4-4b0e-9241-010515e27f0e",