Download_Image_Files_from_Pteridoportal.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Download Image Files from Pteridoportal",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/emcdona1/fmnh_scripts/blob/main/Download_Image_Files_from_Pteridoportal.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dUWMcAdsFlOE"
      },
      "source": [
        "### Setup -- download necessary files:\n",
        "1. Go to [The Pteridophyte Collections Consortium](https://www.pteridoportal.org/portal/index.php).\n",
        "1. In the top menu, navigate to *Search* > *Search Collections*.\n",
        "1. Click *Select/Deselect All* to deselect everything, then choose your source (e.g. 'Field Museum of Natural History Pteridophyte Collection').\n",
        "1. Click *Search >* in the upper right.\n",
        "1. Fill in your search parameters and click *List Display*.\n",
        "1. In the top right, click the *download button* (it looks like a down arrow into an open box).\n",
        "1. In the pop-up window, choose the following settings:\n",
        "    * **Structure:** Darwin Core\n",
        "    * **Data Extensions**: Keep both boxes checked\n",
        "    * **File Format**: Comma Delimited (CSV)\n",
        "    * **Character Set**: ISO-8859-1 (western)  **note this setting is different than previously**\n",
        "    * **Compression**: Check this box\n",
        "1. Click 'Download Data.'\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GsNZHVij57_h"
      },
      "source": [
        "### How to download the images:\n",
        "1.    Hit play on the cell of code below to get Google Colab set up.  (Hover over the [ ], and a play button will appear.  It will show a spinning circle around a stop sign, until the cell is complete. After complete, you can move onto each next step.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3RXI1TuDFH7m"
      },
      "source": [
        "import csv\n",
        "import pandas as pd\n",
        "import requests\n",
        "import argparse\n",
        "from zipfile import ZipFile\n",
        "from google.colab import files\n",
        "import io\n",
        "import os\n",
        "import shutil"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RSmq5Z5p3fZ1"
      },
      "source": [
        "2.    Once the file has downloaded to your computer, hit play on the next cell, and upload the ZIP file to Google Colab  (click *Choose Files*)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lkvqDPHdH1L4"
      },
      "source": [
        "uploaded = files.upload()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1BXFffA47HyR"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zg2Onk8O3WEk"
      },
      "source": [
        "2.    Hit play on the cell below, and type in the name of your dataset.  (Something useful!)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cHTaxxHA2rx5"
      },
      "source": [
        "name = input('Type in the name of your dataset, then hit Enter: ')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gGIucA6X7KYB"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PtQTSSkq3up2"
      },
      "source": [
        "3.    Click play on the cell below.  If there are a lot of images, this may take a while.\n",
        "4.    Scroll to the bottom of the cell to view the progress.  When the program is finished, it will show you the name of the saved ZIP file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DwDp40JmFVpF"
      },
      "source": [
        "def unpack_zip_file() -> (pd.DataFrame, pd.DataFrame):\n",
        "    uploaded_filename = list(uploaded.keys())[0]\n",
        "    with ZipFile(io.BytesIO(uploaded[uploaded_filename]), 'r') as zipped:\n",
        "        occur_bytes = zipped.read('occurrences.csv')\n",
        "        occur = pd.read_csv(io.BytesIO(occur_bytes), encoding='ISO-8859-1', \n",
        "                            usecols=['id','catalogNumber'])\n",
        "        images_bytes = zipped.read('images.csv')\n",
        "        images = pd.read_csv(io.BytesIO(images_bytes), encoding='ISO-8859-1', \n",
        "                             usecols=['coreid','identifier','goodQualityAccessURI', 'format'])\n",
        "    return images, occur\n",
        "\n",
        "\n",
        "def download_images_from_csv(image_rows, occ_df):\n",
        "    download_location = 'images'\n",
        "    if not os.path.exists(download_location):\n",
        "        os.makedirs(download_location)\n",
        "    #delete the duplicate rows\n",
        "    # reindex and drop the rows that are NaN\n",
        "    image_rows = image_rows.reset_index(drop=True)\n",
        "    image_rows = image_rows.drop('format', axis=1)\n",
        "    print('Filtered duplicates.')\n",
        "\n",
        "    barcode_dict = occ_df.set_index('id').T.to_dict('list')\n",
        "    num_not_found = 0\n",
        "    not_found = [['Barcode', 'Core ID']]\n",
        "    for i in range(len(image_rows)):\n",
        "        image_url = image_rows.identifier[i]\n",
        "        result = requests.get(image_url)\n",
        "        coreid = image_rows.coreid[i]\n",
        "        barcode = barcode_dict.get(coreid)[0]\n",
        "        if (result.status_code != 200):\n",
        "            # if the identifier link fails, try the goodQualityAccessURI link\n",
        "            image_url = image_rows.goodQualityAccessURI[i]\n",
        "            result = requests.get(image_url)\n",
        "        if (result.status_code == 200):\n",
        "            with open(os.path.join(download_location, \n",
        "                                   str(barcode) + '_' + str(coreid) + '.jpg'), \n",
        "                      'wb') as download_image:\n",
        "                download_image.write(result.content)\n",
        "        else:\n",
        "            num_not_found = num_not_found + 1\n",
        "            not_found.append([barcode, coreid])\n",
        "        \n",
        "        if i > 0 and i % 25 == 0:\n",
        "            print('%i images downloaded.' % i)\n",
        "    print('All images downloaded.')\n",
        "    if (num_not_found > 0):\n",
        "        print('Some images were not able to be downloaded:')\n",
        "        for row in not_found:\n",
        "            print(str(row[0]) + '\\t\\t' + str(row[1]))\n",
        "\n",
        "\n",
        "images, occur = unpack_zip_file()\n",
        "download_images_from_csv(images, occur)\n",
        "name = name.lower().replace(' ', '_')\n",
        "save_location = shutil.make_archive(name, 'zip', 'images')\n",
        "print('Your zip file is saved as: %s' % save_location)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S6l9JipE4Ia2"
      },
      "source": [
        "---\n",
        "\n",
        "5.    In the left menu, click the folder icon (called *Files*).  You will see the ZIP file that was generated.  Right click this file, and click *Download* to save this file to your computer.\n",
        "6.    Celebrate!"
      ]
    }
  ]
}