2019-06-10-1-macj-instagram-tags.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pUBtnyHpLYQ8"
   },
   "source": [
    "# README\n",
    "\n",
    "This notebook is a scratch to implementation of Instagram web scraping.\n",
    "\n",
    "The goals are test and design scracthes of functions to get data from Instagram.\n",
    "\n",
    "This notebook contains the implementation and tests for a snowballing effect function to get data about tags in Instagram posts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ESrRL5xvMG1_"
   },
   "source": [
    "**Some references**\n",
    "\n",
    "https://www.eatthis.com/biggest-fast-food-chains-america/\n",
    "\n",
    "https://medium.com/@h4t0n/instagram-data-scraping-550c5f2fb6f1\n",
    "\n",
    "https://medium.com/@srujana.rao2/scraping-instagram-with-python-using-selenium-and-beautiful-soup-8b72c186a058\n",
    "\n",
    "https://towardsdatascience.com/social-network-analysis-of-related-hashtags-on-instagram-using-instacrawlr-46c397cb3dbe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yF6jYv8BPzyO"
   },
   "source": [
    "**Endpoint to User Information**\n",
    "\n",
    "https://i.instagram.com/api/v1/users/{USER_ID}/info/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Requesting Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "K3Gsb0qiMI53"
   },
   "outputs": [],
   "source": [
    "# Instagram tag search base url preffix\n",
    "tagurl_prefix = 'https://www.instagram.com/explore/tags/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0kNtKBHWNQSY"
   },
   "outputs": [],
   "source": [
    "# suffix to append to tag request url to retrieve data in JSON format\n",
    "tagurl_suffix = '/?__a=1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jhQa-MJajEGM"
   },
   "outputs": [],
   "source": [
    "# parameter to be appended to the url in order to search a new set of posts after a previous set\n",
    "tagurl_endcursor = '&max_id='"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EIxjdxtSezBS"
   },
   "outputs": [],
   "source": [
    "# a generic media post preffix (concat with media shortcode to view the post)\n",
    "posturl_prefix = 'https://www.instagram.com/p/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "o-MAqIYZLWaY"
   },
   "outputs": [],
   "source": [
    "# target initial tags\n",
    "tags = ['bolsonaro', 'haddad', 'dilma', 'ciro', 'guedes', 'moro']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-7Xzqi4TMqfB"
   },
   "outputs": [],
   "source": [
    "# target url to initial test\n",
    "tagurl = tagurl_prefix + tags[0] + tagurl_suffix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 678,
     "status": "ok",
     "timestamp": 1560206292277,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "dtb0XYV5MuX8",
    "outputId": "e2ee750d-85b7-48bb-cd1e-b2a8fcc9a042"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://www.instagram.com/explore/tags/bolsonaro/?__a=1'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking target url\n",
    "tagurl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Tid_jgkpM1qh"
   },
   "outputs": [],
   "source": [
    "# needed module\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wsXEFHhdM38y"
   },
   "outputs": [],
   "source": [
    "# requesting JSON information\n",
    "json_info = requests.get(tagurl).json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "95GveiSsbOXR"
   },
   "outputs": [],
   "source": [
    "# retrieving a list of posts\n",
    "posts_list = json_info['graphql']['hashtag']['edge_hashtag_to_media']['edges']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 436,
     "status": "ok",
     "timestamp": 1560206309492,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "PY74jKd5ccin",
    "outputId": "a25084ca-8508-4b04-f616-e0d62c30fb18"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking lenght of the list\n",
    "len(posts_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 573
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 654,
     "status": "ok",
     "timestamp": 1560206310431,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "LSa3hmuwbgUE",
    "outputId": "669b4644-6ebf-4899-dfed-33900532b659"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'node': {'__typename': 'GraphImage',\n",
       "  'accessibility_caption': 'Image may contain: one or more people and text',\n",
       "  'comments_disabled': False,\n",
       "  'dimensions': {'height': 1080, 'width': 1080},\n",
       "  'display_url': 'https://instagram.fnat1-1.fna.fbcdn.net/vp/2230dc04fde10320fb8bac0be9f26169/5DACCDC1/t51.2885-15/e35/s1080x1080/65137552_151111176028672_2662746227723959449_n.jpg?_nc_ht=instagram.fnat1-1.fna.fbcdn.net',\n",
       "  'edge_liked_by': {'count': 1},\n",
       "  'edge_media_preview_like': {'count': 1},\n",
       "  'edge_media_to_caption': {'edges': [{'node': {'text': 'Câmara municipal do Rio livra Marcelo Crivella de Impeachment.\\n🇧🇷 SIGA @PELONOVOBR'}}]},\n",
       "  'edge_media_to_comment': {'count': 1},\n",
       "  'id': '2074411265925452949',\n",
       "  'is_video': False,\n",
       "  'owner': {'id': '1100159559'},\n",
       "  'shortcode': 'BzJyg4_jhSV',\n",
       "  'taken_at_timestamp': 1561509114,\n",
       "  'thumbnail_resources': [{'config_height': 150,\n",
       "    'config_width': 150,\n",
       "    'src': 'https://instagram.fnat1-1.fna.fbcdn.net/vp/cbd76321a28c832b7169cde3edba47ed/5D8A16D1/t51.2885-15/e35/s150x150/65137552_151111176028672_2662746227723959449_n.jpg?_nc_ht=instagram.fnat1-1.fna.fbcdn.net'},\n",
       "   {'config_height': 240,\n",
       "    'config_width': 240,\n",
       "    'src': 'https://instagram.fnat1-1.fna.fbcdn.net/vp/e4f0d4769717969d7bd166377cde7dfa/5DB67A9B/t51.2885-15/e35/s240x240/65137552_151111176028672_2662746227723959449_n.jpg?_nc_ht=instagram.fnat1-1.fna.fbcdn.net'},\n",
       "   {'config_height': 320,\n",
       "    'config_width': 320,\n",
       "    'src': 'https://instagram.fnat1-1.fna.fbcdn.net/vp/5ab7004037ebe00f07cc27e8e625eec4/5DA17021/t51.2885-15/e35/s320x320/65137552_151111176028672_2662746227723959449_n.jpg?_nc_ht=instagram.fnat1-1.fna.fbcdn.net'},\n",
       "   {'config_height': 480,\n",
       "    'config_width': 480,\n",
       "    'src': 'https://instagram.fnat1-1.fna.fbcdn.net/vp/637c87257b9a402243e117c3659f3f94/5DAA017B/t51.2885-15/e35/s480x480/65137552_151111176028672_2662746227723959449_n.jpg?_nc_ht=instagram.fnat1-1.fna.fbcdn.net'},\n",
       "   {'config_height': 640,\n",
       "    'config_width': 640,\n",
       "    'src': 'https://instagram.fnat1-1.fna.fbcdn.net/vp/1a2574c3051a94f360f61772c42c4e34/5DC0ED76/t51.2885-15/sh0.08/e35/s640x640/65137552_151111176028672_2662746227723959449_n.jpg?_nc_ht=instagram.fnat1-1.fna.fbcdn.net'}],\n",
       "  'thumbnail_src': 'https://instagram.fnat1-1.fna.fbcdn.net/vp/1a2574c3051a94f360f61772c42c4e34/5DC0ED76/t51.2885-15/sh0.08/e35/s640x640/65137552_151111176028672_2662746227723959449_n.jpg?_nc_ht=instagram.fnat1-1.fna.fbcdn.net'}}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking details about one media\n",
    "posts_list[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oNopgOdCcBmU"
   },
   "source": [
    "With this JSON data, all the relevant information to the analysis can be retrieved. Some of them are:\n",
    "\n",
    "- shortcode of the post (in case of need to visualize)\n",
    "- text (at edge_media_to_caption > edges[0] > node > text)\n",
    "- owner (can be the username or the userid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 305
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 703,
     "status": "ok",
     "timestamp": 1560206312539,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "KX1y0jCucU9a",
    "outputId": "8e72bab6-71ba-48de-a6d3-a360fb5ce35a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'id_owner': '1100159559',\n",
       "  'id_post': '2074411265925452949',\n",
       "  'post_url': 'https://www.instagram.com/p/BzJyg4_jhSV/',\n",
       "  'shortcode': 'BzJyg4_jhSV',\n",
       "  'text': 'Câmara municipal do Rio livra Marcelo Crivella de Impeachment.🇧🇷 SIGA @PELONOVOBR'},\n",
       " {'id_owner': '3545180235',\n",
       "  'id_post': '2074411220988297909',\n",
       "  'post_url': 'https://www.instagram.com/p/BzJygPJF4K1/',\n",
       "  'shortcode': 'BzJygPJF4K1',\n",
       "  'text': 'O presidente Jair Bolsonaro (PSL) voltou atrás e decidiu\\xa0revogar o decreto\\xa0que facilitou o porte de armas de fogo após derrota no Senado. Em reunião com senadores na tarde desta terça-feira (25), o ministro da Casa Civil, Onyx Lorenzoni, anunciou a decisão..Em maio, Bolsonaro editou dois decretos sobre posse e porte de armas de fogo e uso de munições. O pacote de mudanças foi alvo de críticas. Na semana passada o plenário do Senado aprovou parecer da Comissão de Constituição e Justiça (CCJ) que pede a suspensão dos decretos. O parecer seguiu para análise da Câmara dos Deputados..As regras previstas nos decretos continuarão valendo até que a Câmara vote o tema e, eventualmente, aprove a suspensão dos decretos. O direito ao porte é a autorização para transportar a arma fora de casa. É diferente da posse, que só permite manter a arma dentro de casa..#boanoite #política #radardabahia#nadaescapadoradar #Bolsonaro #Brasil #'},\n",
       " {'id_owner': '1397932081',\n",
       "  'id_post': '2074411133443568749',\n",
       "  'post_url': 'https://www.instagram.com/p/BzJye9nBPRt/',\n",
       "  'shortcode': 'BzJye9nBPRt',\n",
       "  'text': 'Bolsonaro dá um drible da vaca nos desarmamentistas e atende apelo popular pró-armas e auto defesa, diz Flávio MorenoSegundo, o Presidente do PSL Alagoas, Flávio Moreno, o Presidente Bolsonaro mais uma vez atende o apelo popular na medida que dá um drible da vaca nos desarmentistas e edita 3 decretos e 1 projeto de lei pró-armas e em apoio a legítima defesa da população. A medida vem após o Senado aprovar parecer que pede a suspensão dos decretos e um dia antes de STF julgar pedido de anulação.http://www.federalnews.com.br/bolsonaro-da-um-drible-da-vaca-nos-desarmamentistas-e-atende-apelo-popular-pro-armas-e-auto-defesa-diz-flavio-moreno/Na semana passada, Senado chegou a rejeitar o texto de decreto assinado em maio. A maioria dos Senadores votaram contra a vontade popular em apoio ao porte de armas demostrada no referendo de 2005 e nas eleições que elegeram Bolsonaro.O presidente Jair Bolsonaro revogou dois decretos assinados em maio que facilitaram o porte de armas de fogo. No lugar, o presidente editou três novos decretos e enviou um projeto de lei ao Congresso Nacional sobre o mesmo tema.Bolsonaro editou, em maio, dois decretos sobre posse e porte de armas de fogo e uso de munições. O pacote de mudanças foi alvo de críticas e, na semana passada, o plenário do Senado aprovou parecer da Comissão de Constituição e Justiça (CCJ) que pede a suspensão dos decretos.Além disso, estava previsto para esta quarta (26) a análise, pelo Supremo Tribunal Federal (STF), de um pedido para anular os decretos.A informação sobre a revogação e sobre os novos decretos foi dada pelo ministro da Casa Civil, Onyx Lorenzoni, em reunião com senadores na tarde desta terça-feira (25). A publicação trouxe a revogação dos decretos sobre armas e munições editados por Bolsonaro em maio e que tiveram parecer do Senado pela suspensão. Além das revogações, o governo editou três novos decretos sobre o tema.Decreto 9.844: regulamenta lei sobre a aquisição, o cadastro, o registro, o porte e a comercialização de armas de fogo e de munição e sobre o Sistema Nacional de Armas e o Sistema de Gerenciamento Militar de Armas.... #gentedireita #bolsonaro #flaviomoreno #maceio #psl #age'}]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# list of media dictionaries (filtered and processed information)\n",
    "posts = []\n",
    "\n",
    "for post in posts_list:\n",
    "  \n",
    "    node = post['node']\n",
    "  \n",
    "    id_post = node['id']\n",
    "  \n",
    "    id_owner = node['owner']['id']\n",
    "  \n",
    "    edges = node['edge_media_to_caption']['edges']\n",
    "  \n",
    "    shortcode = node['shortcode']\n",
    "  \n",
    "    # not all medias have a text\n",
    "    text = edges[0]['node']['text'].replace('\\n','') if len(edges) else ''\n",
    "  \n",
    "    post_url =  posturl_prefix + shortcode + '/'\n",
    "  \n",
    "    post_dict = {\n",
    "        'id_post': id_post,\n",
    "        'id_owner': id_owner,\n",
    "        'shortcode': shortcode,\n",
    "        'text': text,\n",
    "        'post_url': post_url\n",
    "    }\n",
    "  \n",
    "    posts.append( post_dict )\n",
    "\n",
    "posts[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HbJlWOvBhTBE"
   },
   "source": [
    "**Note**\n",
    "\n",
    "We have a valid list of information to work with.\n",
    "\n",
    "The goal now is to create a function to retrieve a large amount of posts. Are we talking about **snowballing effect**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**\n",
    "\n",
    "Some of the attributes in the post dictionary had names changed.\n",
    "\n",
    "This does not affect the previous data analysis done with next notebooks with the file `data.json` generated in 2019-06-11."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Snowballing Effect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 778,
     "status": "ok",
     "timestamp": 1560206188380,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "gQHTYLxTiIsx",
    "outputId": "e7643a34-13d3-4747-de8d-cdb212ec0820"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QVFDSk9uQ2ZJSTZVMEhhRG1zcWY5SzZJOGJqbDZkNXRCZVUxSG1ZYl9SRENYSjhOdmpBR0FBbWtSN1BZQ3M0SFRyM1BTdkdDblNHMzNNeGFvaVJzVlVmaA=='"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking the end_cursor variable to iterate the search\n",
    "json_info['graphql']['hashtag']['edge_hashtag_to_media']['page_info']['end_cursor']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "D1JI8X-2nUYa"
   },
   "outputs": [],
   "source": [
    "def json2posts(json_info):\n",
    "\n",
    "    posts_list = json_info['graphql']['hashtag']['edge_hashtag_to_media']['edges']\n",
    "\n",
    "    posts_dicts = []\n",
    "\n",
    "    for post in posts_list:\n",
    "\n",
    "        node = post['node']\n",
    "\n",
    "        id_post = node['id']\n",
    "\n",
    "        id_owner = node['owner']['id']\n",
    "\n",
    "        shortcode = node['shortcode']\n",
    "\n",
    "        edges = node['edge_media_to_caption']['edges']\n",
    "        \n",
    "        text = edges[0]['node']['text'].replace('\\n','') if len(edges) else ''\n",
    "\n",
    "        post_url = posturl_prefix + shortcode + '/'\n",
    "\n",
    "        post_dict = {\n",
    "            'id_post': id_post,\n",
    "            'id_owner': id_owner,\n",
    "            'shortcode': shortcode,\n",
    "            'text': text,\n",
    "            'post_url': post_url,\n",
    "        }\n",
    "    \n",
    "        posts_dicts.append( post_dict )\n",
    "    \n",
    "    return posts_dicts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SMQJW9vIsemF"
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "def snowball(url, deep=1, end_cursor='', count=0, showurl=False, \n",
    "               sleep=0, forever=False, progress=False, pause=60 ):\n",
    "  \n",
    "    count = count + 1\n",
    "\n",
    "    request_url = url + tagurl_endcursor + end_cursor\n",
    "\n",
    "    if showurl :\n",
    "\n",
    "        print(request_url)\n",
    "\n",
    "    else:\n",
    "\n",
    "        if progress :\n",
    "\n",
    "            print( count )\n",
    "            # if count == 1 :\n",
    "            #  print( '*' * (deep-1) )\n",
    "            # else:\n",
    "            #  print( '*', end='' )\n",
    "\n",
    "    # TODO Involve the request in a try-except block\n",
    "    json_info = requests.get( request_url ).json()\n",
    "\n",
    "    end_cursor = json_info['graphql']['hashtag']['edge_hashtag_to_media']['page_info']['end_cursor']\n",
    "\n",
    "    posts = json2posts( json_info )\n",
    "\n",
    "    time.sleep(sleep)\n",
    "\n",
    "    if count < deep :\n",
    "\n",
    "        try:\n",
    "          \n",
    "          posts += snowball(\n",
    "                url=url, \n",
    "                deep=deep, \n",
    "                end_cursor=end_cursor, \n",
    "                count=count, \n",
    "                showurl=showurl, \n",
    "                sleep=sleep,\n",
    "                forever=forever,\n",
    "                progress=progress, \n",
    "                pause=pause)\n",
    "          \n",
    "        except:\n",
    "          \n",
    "          if forever :\n",
    "            \n",
    "            print( 'Fail, retrying in ' + str(pause) + ' seconds' )\n",
    "            \n",
    "            time.sleep(pause)\n",
    "            \n",
    "            posts += snowball(\n",
    "                url=url, \n",
    "                deep=deep, \n",
    "                end_cursor=end_cursor, \n",
    "                count=count, \n",
    "                showurl=showurl, \n",
    "                sleep=sleep,\n",
    "                forever=forever, \n",
    "                progress=progress, \n",
    "                pause=pause)\n",
    "          \n",
    "          else:\n",
    "            \n",
    "            print( 'Fail, ' + str(count) + ' requests done' )\n",
    "          \n",
    "        else:\n",
    "\n",
    "            pass\n",
    "\n",
    "    return posts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1979,
     "status": "ok",
     "timestamp": 1560208771271,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "1ESN0qy0jpqG",
    "outputId": "61b8b3b0-bbc7-4d2f-a963-576d0f932198"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65 posts retrieved\n"
     ]
    }
   ],
   "source": [
    "posts = snowball(tagurl, deep=1)\n",
    "\n",
    "print( str(len(posts)) + ' posts retrieved' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "Great! The function is working. Next, going deeper in search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4600,
     "status": "ok",
     "timestamp": 1560208774518,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "doGNdtQopqSw",
    "outputId": "9ddad355-9b6b-413b-db3f-bf7ec5e53e80"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "195 posts retrieved\n"
     ]
    }
   ],
   "source": [
    "posts = snowball(tagurl, deep=3)\n",
    "\n",
    "print( str(len(posts)) + ' posts retrieved' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "Great! The snowballing effect is working. Next, going deeper and testing the limits!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 14953,
     "status": "ok",
     "timestamp": 1560208785350,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "TAJTlWMqpxyP",
    "outputId": "13f9cfe0-ea96-4efa-8d1d-b6291278301d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "640 posts retrieved\n"
     ]
    }
   ],
   "source": [
    "posts = snowball(tagurl, deep=10)\n",
    "\n",
    "print( str(len(posts)) + ' posts retrieved' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "Some requests can take some time. Adding some feature to check the progress."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 234
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 25022,
     "status": "ok",
     "timestamp": 1560208795826,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "4QHwjEbXqcF0",
    "outputId": "38b4fcee-d61a-401f-d7a7-e86a76e0a7e2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFDdllDM1IxaU9fNXlCSUlzYTB1aGR2TkhlUXJidVk3NzhmUHRsdnVuUWo4aGE5Z0kzeDlGbFhfSHBDU092c3R3VHQxazVGVXBtUEs5c2oxcU5VazZQRQ==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFEU2ZXdGgtWFVHaTBYWk5BckhqVmVCZlpsR3FvNEdJOFlPU1BiSlhHcEtHTGFzMExucktKU2xoVS04VXdkYXdzS1lqWWpObUpOX1E5QkxsOUVJdGVPTw==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFCdk9rSUJwYlVpU281ZUVaelotekdrVVRHX1g0NkM2bHdTMkJQSnktVE42djdFLXdGZFpXbUR5Vkc2SHU3b0YwTGkzdW5IZzcxOWNuTUxxcjcxNUdJSA==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFDSDM2UGRieEhPZGFIZG83Rm5rMnp4cHZiZDBVU0NaUU1MN2VnLU1sMk1VUnQyNmxrX3ZDaXVndUNSV1EzMUtRVGswWVUwWWpwTkllVkxublRkN2N5bw==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFDUmtQYWxDT2xpT3RUUjR4QTd3ejZGWWdaYXl1Z0R1ZnJ0dHN6THFoN3lNYTc3S0s0VWxmVC02SWxtbEo3LWw3V0hrYkIxczZFRzNhQnlYVms2RnlIVQ==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFDV3Nndl9aMkQ3aFBFT1o5eDdEVGFHbk40NW0wcEptNHdzdk9Jdy03ek1Kc29pNTNhenJ5QnFqOF9yQkJHSWVudUxtRDNxcFJnVUpWcnVvTW9JTlVVNg==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFDNDdXVi1QcFJPSTdwdXNOak9oMG9wSUQxZUU0U1NlMEtqaGEwb29PSzJJS3luZUU0YmJoTmlKZHVDX2JBcmFxQWtCaklvbl8ta29XTzh0TlZrbzE0TQ==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFCOHRRVy1YNEVodWZDMGNnU0hYZXFxQWg2Vnl2Y0Z6M05zX0o0RXlkQjRmVDdIR2g4Z1Z6aVN6TVktZlNoV3d6S1JmSUxuU01ERzBhN0UyUExQUTBmcg==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFETHVqSnlpQkNqczBjTFlxcUh3M0l6akhFUEQyMTRKNlc1aWp6enJaN29NYVBzYWc5dTdfWlFVTXFLcXE1Qkp0ZkNzblo2SEE4dFVXRFFwS0NQZ1BoWg==\n",
      "640 posts retrieved\n"
     ]
    }
   ],
   "source": [
    "posts = snowball(tagurl, deep=10, showurl=True)\n",
    "\n",
    "print( str(len(posts)) + ' posts retrieved' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "Great! But some times Instagram may be blocking some requests depending on the frequency.\n",
    "\n",
    "Adding a parameter to smooth the requesting frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 448
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 44159,
     "status": "ok",
     "timestamp": 1560209565519,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "d_ELXFx6rAqo",
    "outputId": "e64218d3-3744-4997-c24e-74e3622a56c0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFBVFY0bklFZHNpWGV3Ul9jWXlyWmo4dEVBTk9qcnBFX2tXMWpXdVVhT2ktVWV3akg1WURTekRqOUNHcDQ0MVVJcFVYckpYU2RXUXR1NDhoejFEaDNWeA==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFEMlZaMDlYYW1jMkJEdFRUdERCcVJ0NWlsRDBtZEotS2JmeVJrc2FZY2VmOFo1WnlLaTRKXzlfaldmMGRjU0VWRktMdm5sYy1CV3d4WWVfTHQzakJSSA==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFBZmlMQTNhUmJLUUM5dmdfRnk1dDJIMEVGX1NMNHdGVzd5R0VNQ09TdHpfeVJIMmVlckRTdVRGT3lVYTBLblpqS0xUTXl6U21zQnJENC1mUGxtaW0tSQ==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFBNTh2TFJmc3hwekE3SjFvblN4dGRIQllmQnN0a0NidUdoazlESnBXUEE1bW9BUHJRSlFxcHNXeTlkVFJNbFFkVHY1dWNwZXltS0JnckRfdlVQTS1SVQ==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFBVndPaHlxeGQ3dnFlN19Oa2lOWXEzN05Cdms4cDV4SFdTNGxDaDhXSnRsLUljbUhRLVJ6R1VkZnVOcGxFTW1JS1AweFl5MDQ2cjVMVjRNbFl6M3lFcA==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFEV2s0MmFMdGtDUkZCcEwwV2ZkdWJxdUxqaXE1alVRbHUxVHd2Y2pUMlZZSm55RkhTRlpyN3ZjMlBOeUk1dl90WFNCWFlIWENyN3ZTLVZwVUhMbnExaA==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFDR283ZloxSG1hT0Q5ajVpOHZjblBWRU9JRktCc01MbGVQQVJnMHlMSGVDdjgydXk5Q0Z2MWdQLWtuZ1VHUXBleVU4ZnBCdDdCYjVOd3ZudE8zVWZmWQ==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFEZjd0c0Z0M3l2cmVTT0tPZk9PR2xLZzk1dnpaQm9iVi0yX19qTlNtdTV1UGZTeFpVNTZ6clFvSE5nVC1MWU81OGlVLWtnZkZjQURNSXFlZzJ2UlpFUQ==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFDYkNnTXBtSnBPVFV5UGo3QmdKU3BWTVFDUlJOcldLQmp4dDQ4b3BSVXpza3UwaXdEZEdWeWR6ZGVhLTMwWTdWNmdGcS1RdjQydUx0RXVBaXcxZ1dUcw==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFEdnVuMUJ3SWR1dW9zQXZzQkYyMzdueGtqMUxOWnlsZ3FsNXlOOW81YlJPcjhpdTFTTEVZVThQTl95MnRfN1BNQVdITThnbDVJaXBNMm5hakVoVzRTbA==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFCRk4taE1tVHQxbkhuMVFoLWZIaHV6NkZHWjZfZXZhVG9CRWFrTGQ2MzFONEd4STBDQ2R6UklvcERvbkJXMk0wd0pseldfdThoTUVVay1PeU01VUVjNA==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFDWHA2UFl2aTFjUjdjTmJpSmV5Y09UUlZ5UzZiMmc0anIzeEJURElBUVZJT1RNZWFGaWxQS0pnVWhOVW9VNHNqU2pEN1BITzZOOEk4LWNvdExDekJDVQ==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFBSUFGcTBsSkRsbklmdlhVQUZYekYzRU1CMDhPd05vQXQtNURJQ3hWQWMzOERSZy1rUnNWdktHaFVqYVNFZDJIUHp3UElnNEtpSU9xdmVvby1uWXp4WQ==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFCUU9FLVJRczdCbTItMlFwT0VIamtIZ24yWkNUYng3Z1VzaG43YkFtUnZuNHZtcUQxVFYxcjd5VFZXQlF4YWljakp5ZHZBV0tzampYT2RELWtwYllwcA==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFEU2VvVlpyc2RoMERDQ1RFTXNMUEFBV2JBYmo3eURJbWdTa0xjQTVNQzdUbXZkVVYyTWRsdHFlOFFCNy0xYUpFWG41c0ZjV2YzNk5qa1dlUEtWQ2NqYQ==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFDMndkOU5nVWtPdVhqQ1MtQmZlLU9QbThGdWdLaFpqamZqT29ZbVRpS0FtMF94NmpId29zaks2UEJ1R1UtS3M2NXRqZUVqdWEwbVZueHA3SWxUS0dWZA==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFBYkh5QmVJbThRbmNhN09oRHBJVnpIX1BPM0Mtek1BZU9YdXBZVURmbUp2TVhrVmZoMGlRaTVvSGtQb0ZISWZ2SG54clhFdVJOd09BUkFHQ0RBY0FTVQ==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFBYjhGM2tIWTBIMHFQMlNDRWhvVTRtb21vUUlRX19vWmh1RTc3d2Y1REJwWWYtak9veVVBeGFoUlNVa0swWUE1SEtxVTBmc08tSC1ZNTVmZEVDdGlfZg==\n",
      "https://www.instagram.com/explore/tags/bolsonaro/?__a=1&max_id=QVFDRDFTbnZqXzlrUnhNNldGMVMtazd2a1pXb1c2NVhRbXVkekxJcmJXOTRDVlFiUll6X0N1RHNEYlRvaHpxbTBsOG11d0h0MWtwQTd1UWZkVnZlZjlyXw==\n",
      "1266 posts retrieved\n",
      "CPU times: user 1.61 s, sys: 113 ms, total: 1.72 s\n",
      "Wall time: 52.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "posts = snowball(tagurl, deep=20, sleep=1, showurl=True)\n",
    "\n",
    "print( str(len(posts)) + ' posts retrieved' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "Great! Going deeper and checking the limits!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 535
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 49332,
     "status": "ok",
     "timestamp": 1560211425797,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "Dfo4Hhugp5z-",
    "outputId": "795320cc-d2a0-4a51-c010-aa028ea206d1",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n",
      "10\n",
      "11\n",
      "12\n",
      "13\n",
      "Fail, 12 requests done\n",
      "769 posts retrieved\n",
      "CPU times: user 1.07 s, sys: 93.4 ms, total: 1.16 s\n",
      "Wall time: 32.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "posts = snowball(tagurl, deep=50, sleep=1, progress=True)\n",
    "\n",
    "print( str(len(posts)) + ' posts retrieved' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "Good, but snowballing effect stops at the first block.\n",
    "\n",
    "Adding a feature to make it try again forever."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1018
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 167728,
     "status": "ok",
     "timestamp": 1560211748256,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "nNlxAa2KuVSB",
    "outputId": "7a7664de-36b1-49a4-be77-4eaefeccfa34"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n",
      "10\n",
      "11\n",
      "12\n",
      "13\n",
      "14\n",
      "15\n",
      "16\n",
      "17\n",
      "18\n",
      "19\n",
      "20\n",
      "21\n",
      "22\n",
      "23\n",
      "24\n",
      "25\n",
      "26\n",
      "27\n",
      "28\n",
      "29\n",
      "30\n",
      "31\n",
      "32\n",
      "33\n",
      "34\n",
      "35\n",
      "36\n",
      "37\n",
      "38\n",
      "39\n",
      "40\n",
      "41\n",
      "42\n",
      "43\n",
      "44\n",
      "45\n",
      "46\n",
      "47\n",
      "48\n",
      "49\n",
      "Fail, retrying in 60 seconds\n",
      "49\n",
      "50\n",
      "3176 posts retrieved\n",
      "CPU times: user 4.17 s, sys: 280 ms, total: 4.45 s\n",
      "Wall time: 3min 17s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "posts = snowball(tagurl, deep=50, sleep=1, progress=True, forever=True)\n",
    "\n",
    "print( str(len(posts)) + ' posts retrieved' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "Great! Finally, an ultimate test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 2000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 296342,
     "status": "ok",
     "timestamp": 1560212247770,
     "user": {
      "displayName": "MARCOS JUNIOR",
      "photoUrl": "",
      "userId": "02126454598504023629"
     },
     "user_tz": 180
    },
    "id": "tKsGtdYu3nhq",
    "outputId": "28526615-c51a-4b6d-9b07-1dd687e605ad",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n",
      "10\n",
      "11\n",
      "12\n",
      "13\n",
      "14\n",
      "15\n",
      "16\n",
      "17\n",
      "18\n",
      "19\n",
      "20\n",
      "21\n",
      "22\n",
      "23\n",
      "24\n",
      "25\n",
      "26\n",
      "27\n",
      "28\n",
      "29\n",
      "30\n",
      "31\n",
      "32\n",
      "33\n",
      "34\n",
      "35\n",
      "36\n",
      "37\n",
      "38\n",
      "39\n",
      "40\n",
      "41\n",
      "42\n",
      "43\n",
      "44\n",
      "45\n",
      "46\n",
      "47\n",
      "48\n",
      "49\n",
      "50\n",
      "51\n",
      "52\n",
      "Fail, retrying in 30 seconds\n",
      "52\n",
      "53\n",
      "54\n",
      "55\n",
      "56\n",
      "57\n",
      "58\n",
      "59\n",
      "60\n",
      "61\n",
      "62\n",
      "63\n",
      "64\n",
      "65\n",
      "66\n",
      "67\n",
      "68\n",
      "69\n",
      "70\n",
      "71\n",
      "72\n",
      "73\n",
      "74\n",
      "75\n",
      "76\n",
      "77\n",
      "78\n",
      "79\n",
      "80\n",
      "81\n",
      "82\n",
      "83\n",
      "84\n",
      "85\n",
      "86\n",
      "87\n",
      "88\n",
      "89\n",
      "90\n",
      "91\n",
      "92\n",
      "Fail, retrying in 30 seconds\n",
      "92\n",
      "Fail, retrying in 30 seconds\n",
      "91\n",
      "92\n",
      "93\n",
      "94\n",
      "95\n",
      "96\n",
      "97\n",
      "98\n",
      "99\n",
      "100\n",
      "6262 posts retrieved\n",
      "CPU times: user 8.63 s, sys: 629 ms, total: 9.26 s\n",
      "Wall time: 5min 14s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "posts = snowball(tagurl, deep=100, sleep=0.5, progress=True, forever=True, pause=30)\n",
    "\n",
    "print( str(len(posts)) + ' posts retrieved' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "Great time to retrieve a large amount of posts!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "2019-06-10-1-macj-instagram-tags.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}