Original LDA Code - Socrata API Dataset - Gensim LDA Groupings.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#standard imports\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "#API tools\n",
    "import requests\n",
    "import json\n",
    "from pandas.io.json import json_normalize\n",
    "from urllib.request import Request, urlopen\n",
    "import json\n",
    "\n",
    "#visuals\n",
    "import matplotlib as plt\n",
    "import seaborn as sns\n",
    "\n",
    "#text tools\n",
    "from collections import Counter #word frequency counts\n",
    "import re #regexs\n",
    "from ast import literal_eval #turn strings that look like lists into actual lists\n",
    "import itertools #use to return combinations of words in topic strings \n",
    "\n",
    "#Natural Language Processing\n",
    "import nltk\n",
    "import lda #Latent Dirichlet Allocation (create topics)\n",
    "import gensim\n",
    "from gensim import corpora, models #for constructing document term matrix\n",
    "#from stop_words import get_stop_words\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "from nltk import stem\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "#clustering tools\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.cluster import DBSCAN\n",
    "import hdbscan\n",
    "\n",
    "#magic\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.set_option('display.float_format', lambda x: '%.3f' % x) #otherwise we have scientific notation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access All Socrata Open Data Portal Meta-Data with Series of API Calls\n",
    "Socrata's Discovery API allows a user to access ample metadata and usage data for datasets from Socrata portals. However, the API does have some limitations:\n",
    "- The API itself is designed for queried searches, not returning all data about every Socrata open data portal. As such, you have to embrace some workarounds to truly access all available data.\n",
    "- Without changing the 'limit' parameter, the API only returns 100 datasets for a given search parameter (the datasets returned *do seem to be sorted by total views*)\n",
    "\n",
    "Additionally, trying to return more than 600 datasets at a time is very slow (on this machine).\n",
    "\n",
    "** To work around these limitations, we will create a list of URLs for city/state open government Socrata open data portal. We will iterate through this list, returning the top 600 pages of data for each city's portal. Then we will convert each returned result to pandas-readable JSON, and convert that JSON into a dataframe. Then we will concatenate each dataframe. **\n",
    "\n",
    "*Example API Call: http://api.us.socrata.com/api/catalog/v1?domains=data.seattle.gov&limit=600*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First, get a list of all Socrata open data portals (for governments in the U.S.)\n",
    "We will use an API call to get this list, which we will use for more API calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "url_request = requests.get('http://api.us.socrata.com/api/catalog/v1/domains') \n",
    "#this is an API call of basically only domain URLs\n",
    "\n",
    "url_json = url_request.json() #turn this requests.get object into a JSON-style dict\n",
    "url_df = pd.io.json.json_normalize(url_json, record_path = 'results')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>domain</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>2014bonds.cityofws.org</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>69</td>\n",
       "      <td>amopen.amo.on.ca</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>84</td>\n",
       "      <td>bchi.bigcitieshealth.org</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>bea.data.commerce.gov</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   count                    domain\n",
       "0      4    2014bonds.cityofws.org\n",
       "1     69          amopen.amo.on.ca\n",
       "2     84  bchi.bigcitieshealth.org\n",
       "3      3     bea.data.commerce.gov"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url_df.head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "urls = url_df.domain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** We have some .ca, .mx, etc. URLs that clearly aren't portals in the U.S. Let's remove them: **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/nathanzencey1/anaconda/lib/python3.6/site-packages/ipykernel_launcher.py:1: UserWarning: This pattern has match groups. To actually get the groups, use str.extract.\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "214              www.forsythfutures.org\n",
       "215    www.mdchildhungerpartnership.org\n",
       "216            www.metrochicagodata.org\n",
       "217                 www.opendatanyc.com\n",
       "218                   www.sudandata.org\n",
       "Name: domain, dtype: object"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "city_urls = urls[urls.str.contains('^[a-zA-Z0-9\\-\\.]+\\.(gov|org|com|us)$')] #use regex to match only certain URLs\n",
    "city_urls = city_urls.reset_index(drop=True)\n",
    "city_urls.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          2014bonds.cityofws.org\n",
       "1        bchi.bigcitieshealth.org\n",
       "2           bea.data.commerce.gov\n",
       "3           bis.data.commerce.gov\n",
       "4    brigades.opendatanetwork.com\n",
       "5         bythenumbers.sco.ca.gov\n",
       "6        census.data.commerce.gov\n",
       "7                chhs.data.ca.gov\n",
       "8             chronicdata.cdc.gov\n",
       "9         churned-data.awcnet.org\n",
       "Name: domain, dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "city_urls.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** As we can see, we have to manually datasets that the human eye can tell aren't related to U.S. cities or states. These include federal and non-profit open data. We could build a complex regex to do this, but there's only 220 or so datasets total. **\n",
    "\n",
    "(We will KEEP data from state portals)\n",
    "\n",
    "** CAUTION -- Must confirm these are the right index numbers, as the data returned by the API changes (obviously) **\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0              2014bonds.cityofws.org\n",
       "1            bchi.bigcitieshealth.org\n",
       "2               bea.data.commerce.gov\n",
       "3               bis.data.commerce.gov\n",
       "4        brigades.opendatanetwork.com\n",
       "5             bythenumbers.sco.ca.gov\n",
       "6            census.data.commerce.gov\n",
       "7                    chhs.data.ca.gov\n",
       "8                 chronicdata.cdc.gov\n",
       "9             churned-data.awcnet.org\n",
       "10                 cip.cityofnovi.org\n",
       "11          controllerdata.lacity.org\n",
       "12         dashboard.alexandriava.gov\n",
       "13               dashboard.hawaii.gov\n",
       "14                dashboard.plano.gov\n",
       "15                 dashboard.slco.org\n",
       "16                     data.acgov.org\n",
       "17                  data.albanyny.gov\n",
       "18                  data.auburnwa.gov\n",
       "19               data.austintexas.gov\n",
       "20             data.baltimorecity.gov\n",
       "21                      data.brla.gov\n",
       "22              data.burlingtonvt.gov\n",
       "23               data.cambridgema.gov\n",
       "24                       data.cdc.gov\n",
       "25              data.chattlibrary.org\n",
       "26             data.cincinnati-oh.gov\n",
       "27              data.cityofboston.gov\n",
       "28             data.cityofchicago.org\n",
       "29            data.cityofevanston.org\n",
       "30         data.cityofgainesville.org\n",
       "31                  data.cityofgp.com\n",
       "32             data.cityofmadison.com\n",
       "33              data.cityofnewyork.us\n",
       "34              data.cityoftacoma.org\n",
       "35      data.cityofwestsacramento.org\n",
       "36                       data.cms.gov\n",
       "37                   data.code4sa.org\n",
       "38                  data.colorado.gov\n",
       "39                  data.commerce.gov\n",
       "40       data.communityvitalsigns.org\n",
       "41           data.consumerfinance.gov\n",
       "42          data.countyofriverside.us\n",
       "43                      data.cstx.gov\n",
       "44                        data.ct.gov\n",
       "45                data.culvercity.org\n",
       "46                data.datamontana.us\n",
       "47                    data.dcpcsb.org\n",
       "48    data.debtwatch.treasurer.ca.gov\n",
       "49                  data.delaware.gov\n",
       "Name: domain, dtype: object"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "city_urls[0:50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/nathanzencey1/anaconda/lib/python3.6/site-packages/ipykernel_launcher.py:8: UserWarning: This pattern has match groups. To actually get the groups, use str.extract.\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "#went through df with human eye line by line\n",
    "clean_urls = city_urls.drop(city_urls.index[[1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 24, 25, 36, 40, 41, 47, 48, 52, 53, 54, \n",
    "                                             56, 57, 58, 66, 71, 79, 83, 84, 91, 94, 96, 97, 99, 103, 104, 108, 109, \n",
    "                                             110, 121, 128, 133, 140, 144, 146, 147, 148, 152, 153, 157, 159, 161, \n",
    "                                             162, 163, 164, 165, 171, 176, 199, 200, 201, 203, 204, 205, 209, 212, \n",
    "                                             214, 215, 218, 37, 39]]) #forgot two outlier datasets here\n",
    "\n",
    "clean_urls = clean_urls[~clean_urls.str.contains('^[a-zA-Z0-9\\-\\.]+\\.(cms.gov)$')] #make sure all CMS are gone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0            2014bonds.cityofws.org\n",
       "10               cip.cityofnovi.org\n",
       "12       dashboard.alexandriava.gov\n",
       "13             dashboard.hawaii.gov\n",
       "14              dashboard.plano.gov\n",
       "15               dashboard.slco.org\n",
       "16                   data.acgov.org\n",
       "17                data.albanyny.gov\n",
       "18                data.auburnwa.gov\n",
       "19             data.austintexas.gov\n",
       "20           data.baltimorecity.gov\n",
       "21                    data.brla.gov\n",
       "22            data.burlingtonvt.gov\n",
       "23             data.cambridgema.gov\n",
       "26           data.cincinnati-oh.gov\n",
       "27            data.cityofboston.gov\n",
       "28           data.cityofchicago.org\n",
       "29          data.cityofevanston.org\n",
       "30       data.cityofgainesville.org\n",
       "31                data.cityofgp.com\n",
       "32           data.cityofmadison.com\n",
       "33            data.cityofnewyork.us\n",
       "34            data.cityoftacoma.org\n",
       "35    data.cityofwestsacramento.org\n",
       "38                data.colorado.gov\n",
       "42        data.countyofriverside.us\n",
       "43                    data.cstx.gov\n",
       "44                      data.ct.gov\n",
       "45              data.culvercity.org\n",
       "46              data.datamontana.us\n",
       "49                data.delaware.gov\n",
       "50               data.detroitmi.gov\n",
       "51               data.douglas.co.us\n",
       "55          data.fortworthtexas.gov\n",
       "59           data.greensboro-nc.gov\n",
       "60                 data.hampton.gov\n",
       "61                data.hartford.gov\n",
       "62                  data.hawaii.gov\n",
       "63              data.healthcare.gov\n",
       "64                data.honolulu.gov\n",
       "65                data.illinois.gov\n",
       "67                    data.iowa.gov\n",
       "68               data.jacksonms.gov\n",
       "69                    data.kcmo.org\n",
       "70              data.kingcounty.gov\n",
       "72                  data.lacity.org\n",
       "73                data.lacounty.gov\n",
       "74                  data.lcb.wa.gov\n",
       "75              data.littlerock.gov\n",
       "76              data.livewellsd.org\n",
       "Name: domain, dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_urls[0:50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=149, step=1)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_urls = clean_urls.reset_index(drop=True)\n",
    "clean_urls.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a function to return JSON-formatted dicts of usage and meta-data for each URL's datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#a function that takes a pd.Series of Socrata open data portal URLs and returns data for EACH in list of JSON-format dicts\n",
    "def get_json(urls): #urls arg should be a Series of open data portal URLs\n",
    "    json_dicts = []\n",
    "    for url in urls:\n",
    "        request = requests.get('http://api.us.socrata.com/api/catalog/v1?domains={}&limit=600'.format(url))\n",
    "        raw_data = request.json()\n",
    "        json_dicts.append(raw_data)\n",
    "    return json_dicts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get JSON Dicts: (** One Minute+ Run Time Alert **)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "json_dicts = get_json(clean_urls)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "149"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(json_dicts) #we have JSON data from 164 cities/states"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a hulking list of JSON-style dictionaries. Let's see if pandas can handle making them into dfs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def json_to_df(json_dicts):\n",
    "    final_df = pd.DataFrame()\n",
    "    for d in json_dicts:\n",
    "        df = pd.io.json.json_normalize(d, record_path = 'results')\n",
    "        final_df = final_df.append(df)\n",
    "    return final_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = json_to_df(json_dicts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>classification</th>\n",
       "      <th>link</th>\n",
       "      <th>metadata</th>\n",
       "      <th>owner</th>\n",
       "      <th>permalink</th>\n",
       "      <th>preview_image_url</th>\n",
       "      <th>resource</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_categor...</td>\n",
       "      <td>https://2014bonds.cityofws.org/dataset/Bonds-P...</td>\n",
       "      <td>{'domain': '2014bonds.cityofws.org'}</td>\n",
       "      <td>{'id': '3xde-ipm7', 'display_name': 'mjmartin'}</td>\n",
       "      <td>https://2014bonds.cityofws.org/d/9csq-7i9t</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'name': 'Bonds Project Dataset (Official)', '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_tags': ...</td>\n",
       "      <td>https://2014bonds.cityofws.org/dataset/Bonds-A...</td>\n",
       "      <td>{'domain': '2014bonds.cityofws.org'}</td>\n",
       "      <td>{'id': 'd2yk-enzf', 'display_name': 'Christian...</td>\n",
       "      <td>https://2014bonds.cityofws.org/d/xwab-9myw</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'nbe_fxf': None, 'description': '', 'type': '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_tags': ...</td>\n",
       "      <td>https://2014bonds.cityofws.org/dataset/Wards-2...</td>\n",
       "      <td>{'domain': '2014bonds.cityofws.org'}</td>\n",
       "      <td>{'id': 'vkxb-3xjh', 'display_name': 'James Chu...</td>\n",
       "      <td>https://2014bonds.cityofws.org/d/963m-jiy8</td>\n",
       "      <td>https://2014bonds.cityofws.org/views/963m-jiy8...</td>\n",
       "      <td>{'nbe_fxf': None, 'description': '', 'type': '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_tags': ...</td>\n",
       "      <td>https://2014bonds.cityofws.org/dataset/Project...</td>\n",
       "      <td>{'domain': '2014bonds.cityofws.org'}</td>\n",
       "      <td>{'id': '3xde-ipm7', 'display_name': 'mjmartin'}</td>\n",
       "      <td>https://2014bonds.cityofws.org/d/hz5n-bkfx</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'nbe_fxf': None, 'description': '', 'type': '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_categor...</td>\n",
       "      <td>https://cip.cityofnovi.org/dataset/Capital-Pro...</td>\n",
       "      <td>{'domain': 'cip.cityofnovi.org'}</td>\n",
       "      <td>{'id': 'nc6z-mbfi', 'display_name': 'Jessica'}</td>\n",
       "      <td>https://cip.cityofnovi.org/d/7nqp-ya57</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'name': 'Capital Project Dataset', 'id': '7nq...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      classification  \\\n",
       "0  {'categories': [], 'tags': [], 'domain_categor...   \n",
       "1  {'categories': [], 'tags': [], 'domain_tags': ...   \n",
       "2  {'categories': [], 'tags': [], 'domain_tags': ...   \n",
       "3  {'categories': [], 'tags': [], 'domain_tags': ...   \n",
       "0  {'categories': [], 'tags': [], 'domain_categor...   \n",
       "\n",
       "                                                link  \\\n",
       "0  https://2014bonds.cityofws.org/dataset/Bonds-P...   \n",
       "1  https://2014bonds.cityofws.org/dataset/Bonds-A...   \n",
       "2  https://2014bonds.cityofws.org/dataset/Wards-2...   \n",
       "3  https://2014bonds.cityofws.org/dataset/Project...   \n",
       "0  https://cip.cityofnovi.org/dataset/Capital-Pro...   \n",
       "\n",
       "                               metadata  \\\n",
       "0  {'domain': '2014bonds.cityofws.org'}   \n",
       "1  {'domain': '2014bonds.cityofws.org'}   \n",
       "2  {'domain': '2014bonds.cityofws.org'}   \n",
       "3  {'domain': '2014bonds.cityofws.org'}   \n",
       "0      {'domain': 'cip.cityofnovi.org'}   \n",
       "\n",
       "                                               owner  \\\n",
       "0    {'id': '3xde-ipm7', 'display_name': 'mjmartin'}   \n",
       "1  {'id': 'd2yk-enzf', 'display_name': 'Christian...   \n",
       "2  {'id': 'vkxb-3xjh', 'display_name': 'James Chu...   \n",
       "3    {'id': '3xde-ipm7', 'display_name': 'mjmartin'}   \n",
       "0     {'id': 'nc6z-mbfi', 'display_name': 'Jessica'}   \n",
       "\n",
       "                                    permalink  \\\n",
       "0  https://2014bonds.cityofws.org/d/9csq-7i9t   \n",
       "1  https://2014bonds.cityofws.org/d/xwab-9myw   \n",
       "2  https://2014bonds.cityofws.org/d/963m-jiy8   \n",
       "3  https://2014bonds.cityofws.org/d/hz5n-bkfx   \n",
       "0      https://cip.cityofnovi.org/d/7nqp-ya57   \n",
       "\n",
       "                                   preview_image_url  \\\n",
       "0                                                NaN   \n",
       "1                                                NaN   \n",
       "2  https://2014bonds.cityofws.org/views/963m-jiy8...   \n",
       "3                                                NaN   \n",
       "0                                                NaN   \n",
       "\n",
       "                                            resource  \n",
       "0  {'name': 'Bonds Project Dataset (Official)', '...  \n",
       "1  {'nbe_fxf': None, 'description': '', 'type': '...  \n",
       "2  {'nbe_fxf': None, 'description': '', 'type': '...  \n",
       "3  {'nbe_fxf': None, 'description': '', 'type': '...  \n",
       "0  {'name': 'Capital Project Dataset', 'id': '7nq...  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(33452, 7)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape #we have data on 35,000+ datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Ok, now we just need to unpack the dicts in these columns and clean up our df **."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>classification</th>\n",
       "      <th>link</th>\n",
       "      <th>metadata</th>\n",
       "      <th>owner</th>\n",
       "      <th>permalink</th>\n",
       "      <th>preview_image_url</th>\n",
       "      <th>resource</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_categor...</td>\n",
       "      <td>https://2014bonds.cityofws.org/dataset/Bonds-P...</td>\n",
       "      <td>{'domain': '2014bonds.cityofws.org'}</td>\n",
       "      <td>{'id': '3xde-ipm7', 'display_name': 'mjmartin'}</td>\n",
       "      <td>https://2014bonds.cityofws.org/d/9csq-7i9t</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'name': 'Bonds Project Dataset (Official)', '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_tags': ...</td>\n",
       "      <td>https://2014bonds.cityofws.org/dataset/Bonds-A...</td>\n",
       "      <td>{'domain': '2014bonds.cityofws.org'}</td>\n",
       "      <td>{'id': 'd2yk-enzf', 'display_name': 'Christian...</td>\n",
       "      <td>https://2014bonds.cityofws.org/d/xwab-9myw</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'nbe_fxf': None, 'description': '', 'type': '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_tags': ...</td>\n",
       "      <td>https://2014bonds.cityofws.org/dataset/Wards-2...</td>\n",
       "      <td>{'domain': '2014bonds.cityofws.org'}</td>\n",
       "      <td>{'id': 'vkxb-3xjh', 'display_name': 'James Chu...</td>\n",
       "      <td>https://2014bonds.cityofws.org/d/963m-jiy8</td>\n",
       "      <td>https://2014bonds.cityofws.org/views/963m-jiy8...</td>\n",
       "      <td>{'nbe_fxf': None, 'description': '', 'type': '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_tags': ...</td>\n",
       "      <td>https://2014bonds.cityofws.org/dataset/Project...</td>\n",
       "      <td>{'domain': '2014bonds.cityofws.org'}</td>\n",
       "      <td>{'id': '3xde-ipm7', 'display_name': 'mjmartin'}</td>\n",
       "      <td>https://2014bonds.cityofws.org/d/hz5n-bkfx</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'nbe_fxf': None, 'description': '', 'type': '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_categor...</td>\n",
       "      <td>https://cip.cityofnovi.org/dataset/Capital-Pro...</td>\n",
       "      <td>{'domain': 'cip.cityofnovi.org'}</td>\n",
       "      <td>{'id': 'nc6z-mbfi', 'display_name': 'Jessica'}</td>\n",
       "      <td>https://cip.cityofnovi.org/d/7nqp-ya57</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'name': 'Capital Project Dataset', 'id': '7nq...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_tags': ...</td>\n",
       "      <td>https://cip.cityofnovi.org/dataset/Capital-Pro...</td>\n",
       "      <td>{'domain': 'cip.cityofnovi.org'}</td>\n",
       "      <td>{'id': 'vkxb-3xjh', 'display_name': 'James Chu...</td>\n",
       "      <td>https://cip.cityofnovi.org/d/kuue-gg3b</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'nbe_fxf': None, 'description': '', 'type': '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_categor...</td>\n",
       "      <td>https://cip.cityofnovi.org/dataset/Funding-Sou...</td>\n",
       "      <td>{'domain': 'cip.cityofnovi.org'}</td>\n",
       "      <td>{'id': 'trij-xrnq', 'display_name': 'Meredith ...</td>\n",
       "      <td>https://cip.cityofnovi.org/d/5h3i-jdcp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'name': 'Funding Sources (Pie Chart)', 'id': ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_categor...</td>\n",
       "      <td>https://cip.cityofnovi.org/dataset/Funding-Sou...</td>\n",
       "      <td>{'domain': 'cip.cityofnovi.org'}</td>\n",
       "      <td>{'id': 'trij-xrnq', 'display_name': 'Meredith ...</td>\n",
       "      <td>https://cip.cityofnovi.org/d/uuui-htp8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'name': 'Funding Sources', 'id': 'uuui-htp8',...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_tags': ...</td>\n",
       "      <td>https://cip.cityofnovi.org/dataset/City-Bounda...</td>\n",
       "      <td>{'domain': 'cip.cityofnovi.org'}</td>\n",
       "      <td>{'id': 'trij-xrnq', 'display_name': 'Meredith ...</td>\n",
       "      <td>https://cip.cityofnovi.org/d/9fp5-7dsy</td>\n",
       "      <td>https://cip.cityofnovi.org/views/9fp5-7dsy/fil...</td>\n",
       "      <td>{'description': '', 'type': 'map', 'download_c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>{'categories': [], 'tags': [], 'domain_tags': ...</td>\n",
       "      <td>https://cip.cityofnovi.org/dataset/Projects/cs...</td>\n",
       "      <td>{'domain': 'cip.cityofnovi.org'}</td>\n",
       "      <td>{'id': 'trij-xrnq', 'display_name': 'Meredith ...</td>\n",
       "      <td>https://cip.cityofnovi.org/d/cspu-rwqv</td>\n",
       "      <td>https://cip.cityofnovi.org/views/cspu-rwqv/fil...</td>\n",
       "      <td>{'description': '', 'type': 'map', 'download_c...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      classification  \\\n",
       "0  {'categories': [], 'tags': [], 'domain_categor...   \n",
       "1  {'categories': [], 'tags': [], 'domain_tags': ...   \n",
       "2  {'categories': [], 'tags': [], 'domain_tags': ...   \n",
       "3  {'categories': [], 'tags': [], 'domain_tags': ...   \n",
       "4  {'categories': [], 'tags': [], 'domain_categor...   \n",
       "5  {'categories': [], 'tags': [], 'domain_tags': ...   \n",
       "6  {'categories': [], 'tags': [], 'domain_categor...   \n",
       "7  {'categories': [], 'tags': [], 'domain_categor...   \n",
       "8  {'categories': [], 'tags': [], 'domain_tags': ...   \n",
       "9  {'categories': [], 'tags': [], 'domain_tags': ...   \n",
       "\n",
       "                                                link  \\\n",
       "0  https://2014bonds.cityofws.org/dataset/Bonds-P...   \n",
       "1  https://2014bonds.cityofws.org/dataset/Bonds-A...   \n",
       "2  https://2014bonds.cityofws.org/dataset/Wards-2...   \n",
       "3  https://2014bonds.cityofws.org/dataset/Project...   \n",
       "4  https://cip.cityofnovi.org/dataset/Capital-Pro...   \n",
       "5  https://cip.cityofnovi.org/dataset/Capital-Pro...   \n",
       "6  https://cip.cityofnovi.org/dataset/Funding-Sou...   \n",
       "7  https://cip.cityofnovi.org/dataset/Funding-Sou...   \n",
       "8  https://cip.cityofnovi.org/dataset/City-Bounda...   \n",
       "9  https://cip.cityofnovi.org/dataset/Projects/cs...   \n",
       "\n",
       "                               metadata  \\\n",
       "0  {'domain': '2014bonds.cityofws.org'}   \n",
       "1  {'domain': '2014bonds.cityofws.org'}   \n",
       "2  {'domain': '2014bonds.cityofws.org'}   \n",
       "3  {'domain': '2014bonds.cityofws.org'}   \n",
       "4      {'domain': 'cip.cityofnovi.org'}   \n",
       "5      {'domain': 'cip.cityofnovi.org'}   \n",
       "6      {'domain': 'cip.cityofnovi.org'}   \n",
       "7      {'domain': 'cip.cityofnovi.org'}   \n",
       "8      {'domain': 'cip.cityofnovi.org'}   \n",
       "9      {'domain': 'cip.cityofnovi.org'}   \n",
       "\n",
       "                                               owner  \\\n",
       "0    {'id': '3xde-ipm7', 'display_name': 'mjmartin'}   \n",
       "1  {'id': 'd2yk-enzf', 'display_name': 'Christian...   \n",
       "2  {'id': 'vkxb-3xjh', 'display_name': 'James Chu...   \n",
       "3    {'id': '3xde-ipm7', 'display_name': 'mjmartin'}   \n",
       "4     {'id': 'nc6z-mbfi', 'display_name': 'Jessica'}   \n",
       "5  {'id': 'vkxb-3xjh', 'display_name': 'James Chu...   \n",
       "6  {'id': 'trij-xrnq', 'display_name': 'Meredith ...   \n",
       "7  {'id': 'trij-xrnq', 'display_name': 'Meredith ...   \n",
       "8  {'id': 'trij-xrnq', 'display_name': 'Meredith ...   \n",
       "9  {'id': 'trij-xrnq', 'display_name': 'Meredith ...   \n",
       "\n",
       "                                    permalink  \\\n",
       "0  https://2014bonds.cityofws.org/d/9csq-7i9t   \n",
       "1  https://2014bonds.cityofws.org/d/xwab-9myw   \n",
       "2  https://2014bonds.cityofws.org/d/963m-jiy8   \n",
       "3  https://2014bonds.cityofws.org/d/hz5n-bkfx   \n",
       "4      https://cip.cityofnovi.org/d/7nqp-ya57   \n",
       "5      https://cip.cityofnovi.org/d/kuue-gg3b   \n",
       "6      https://cip.cityofnovi.org/d/5h3i-jdcp   \n",
       "7      https://cip.cityofnovi.org/d/uuui-htp8   \n",
       "8      https://cip.cityofnovi.org/d/9fp5-7dsy   \n",
       "9      https://cip.cityofnovi.org/d/cspu-rwqv   \n",
       "\n",
       "                                   preview_image_url  \\\n",
       "0                                                NaN   \n",
       "1                                                NaN   \n",
       "2  https://2014bonds.cityofws.org/views/963m-jiy8...   \n",
       "3                                                NaN   \n",
       "4                                                NaN   \n",
       "5                                                NaN   \n",
       "6                                                NaN   \n",
       "7                                                NaN   \n",
       "8  https://cip.cityofnovi.org/views/9fp5-7dsy/fil...   \n",
       "9  https://cip.cityofnovi.org/views/cspu-rwqv/fil...   \n",
       "\n",
       "                                            resource  \n",
       "0  {'name': 'Bonds Project Dataset (Official)', '...  \n",
       "1  {'nbe_fxf': None, 'description': '', 'type': '...  \n",
       "2  {'nbe_fxf': None, 'description': '', 'type': '...  \n",
       "3  {'nbe_fxf': None, 'description': '', 'type': '...  \n",
       "4  {'name': 'Capital Project Dataset', 'id': '7nq...  \n",
       "5  {'nbe_fxf': None, 'description': '', 'type': '...  \n",
       "6  {'name': 'Funding Sources (Pie Chart)', 'id': ...  \n",
       "7  {'name': 'Funding Sources', 'id': 'uuui-htp8',...  \n",
       "8  {'description': '', 'type': 'map', 'download_c...  \n",
       "9  {'description': '', 'type': 'map', 'download_c...  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = data.reset_index(drop=True)\n",
    "data.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ++++ Computationally Intensive!! ++++"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#get each column's dict as separate df\n",
    "classification = data.classification.apply(pd.Series)\n",
    "metadata = data.metadata.apply(pd.Series)\n",
    "resource = data.resource.apply(pd.Series)\n",
    "\n",
    "#mash_data = pd.concat([data, classification, metadata, resource])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>attribution</th>\n",
       "      <th>columns_datatype</th>\n",
       "      <th>columns_description</th>\n",
       "      <th>columns_field_name</th>\n",
       "      <th>columns_name</th>\n",
       "      <th>createdAt</th>\n",
       "      <th>description</th>\n",
       "      <th>download_count</th>\n",
       "      <th>id</th>\n",
       "      <th>name</th>\n",
       "      <th>nbe_fxf</th>\n",
       "      <th>obe_fxf</th>\n",
       "      <th>page_views</th>\n",
       "      <th>parent_fxf</th>\n",
       "      <th>provenance</th>\n",
       "      <th>type</th>\n",
       "      <th>updatedAt</th>\n",
       "      <th>view_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>[location, text, photo, text, text, text, text...</td>\n",
       "      <td>[, , , , , , , , , , , , , , , ]</td>\n",
       "      <td>[location, icon, project_image, category_id, l...</td>\n",
       "      <td>[Automated Geocoding, Icon, Project Image, Cat...</td>\n",
       "      <td>2015-02-12T19:14:36.000Z</td>\n",
       "      <td></td>\n",
       "      <td>15.000</td>\n",
       "      <td>9csq-7i9t</td>\n",
       "      <td>Bonds Project Dataset (Official)</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>{'page_views_last_week': 8, 'page_views_last_m...</td>\n",
       "      <td>None</td>\n",
       "      <td>official</td>\n",
       "      <td>dataset</td>\n",
       "      <td>2017-08-11T19:53:36.000Z</td>\n",
       "      <td>{'page_views_last_week': 8, 'page_views_last_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>None</td>\n",
       "      <td>[text, text, text]</td>\n",
       "      <td>[, , ]</td>\n",
       "      <td>[icon, name, id]</td>\n",
       "      <td>[Icon, name, id]</td>\n",
       "      <td>2015-02-05T18:41:29.000Z</td>\n",
       "      <td></td>\n",
       "      <td>7.000</td>\n",
       "      <td>xwab-9myw</td>\n",
       "      <td>Bonds Application - Reference Table</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>{'page_views_total': 66, 'page_views_total_log...</td>\n",
       "      <td>None</td>\n",
       "      <td>official</td>\n",
       "      <td>dataset</td>\n",
       "      <td>2015-04-17T15:30:47.000Z</td>\n",
       "      <td>{'page_views_total': 66, 'page_views_total_log...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>2015-02-09T20:48:39.000Z</td>\n",
       "      <td></td>\n",
       "      <td>7.000</td>\n",
       "      <td>963m-jiy8</td>\n",
       "      <td>Wards 2011.shp</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>{'page_views_total': 23, 'page_views_total_log...</td>\n",
       "      <td>None</td>\n",
       "      <td>official</td>\n",
       "      <td>map</td>\n",
       "      <td>2016-08-30T22:00:21.000Z</td>\n",
       "      <td>{'page_views_total': 23, 'page_views_total_log...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>None</td>\n",
       "      <td>[text, text, text, text, text, number, text, t...</td>\n",
       "      <td>[, , , , , , , , , , , , , ]</td>\n",
       "      <td>[city, location_id, project_id, project_name, ...</td>\n",
       "      <td>[City, Location Id, Project Id, Project Name, ...</td>\n",
       "      <td>2015-04-15T05:15:46.000Z</td>\n",
       "      <td></td>\n",
       "      <td>3.000</td>\n",
       "      <td>hz5n-bkfx</td>\n",
       "      <td>Project Dataset (Staging)</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>{'page_views_total': 15, 'page_views_total_log...</td>\n",
       "      <td>None</td>\n",
       "      <td>official</td>\n",
       "      <td>dataset</td>\n",
       "      <td>2015-04-15T05:20:19.000Z</td>\n",
       "      <td>{'page_views_total': 15, 'page_views_total_log...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>None</td>\n",
       "      <td>[text, text, text, text, text, text, text, tex...</td>\n",
       "      <td>[, , , , , , , , , Please be sure to include \"...</td>\n",
       "      <td>[document_10, document_9, document_8, document...</td>\n",
       "      <td>[Document 10, Document 9, Document 8, Document...</td>\n",
       "      <td>2015-08-28T12:03:36.000Z</td>\n",
       "      <td></td>\n",
       "      <td>25.000</td>\n",
       "      <td>7nqp-ya57</td>\n",
       "      <td>Capital Project Dataset</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>{'page_views_last_week': 0, 'page_views_last_m...</td>\n",
       "      <td>None</td>\n",
       "      <td>official</td>\n",
       "      <td>dataset</td>\n",
       "      <td>2017-07-27T14:24:01.000Z</td>\n",
       "      <td>{'page_views_last_week': 0, 'page_views_last_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>None</td>\n",
       "      <td>[text, text, text]</td>\n",
       "      <td>[, , ]</td>\n",
       "      <td>[id, name, icon]</td>\n",
       "      <td>[id, name, Icon]</td>\n",
       "      <td>2015-08-28T12:01:45.000Z</td>\n",
       "      <td></td>\n",
       "      <td>22.000</td>\n",
       "      <td>kuue-gg3b</td>\n",
       "      <td>Capital Project Application - Reference Table</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>{'page_views_total': 63, 'page_views_total_log...</td>\n",
       "      <td>None</td>\n",
       "      <td>official</td>\n",
       "      <td>dataset</td>\n",
       "      <td>2015-10-02T23:07:35.000Z</td>\n",
       "      <td>{'page_views_total': 63, 'page_views_total_log...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>None</td>\n",
       "      <td>[text, money]</td>\n",
       "      <td>[, ]</td>\n",
       "      <td>[funding_source, budget]</td>\n",
       "      <td>[Funding Source, Budget]</td>\n",
       "      <td>2016-08-09T19:40:33.000Z</td>\n",
       "      <td></td>\n",
       "      <td>32.000</td>\n",
       "      <td>5h3i-jdcp</td>\n",
       "      <td>Funding Sources (Pie Chart)</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>{'page_views_last_week': 2, 'page_views_last_m...</td>\n",
       "      <td>[7nqp-ya57, kf75-36tp]</td>\n",
       "      <td>official</td>\n",
       "      <td>chart</td>\n",
       "      <td>2017-07-27T14:23:39.000Z</td>\n",
       "      <td>{'page_views_last_week': 2, 'page_views_last_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>None</td>\n",
       "      <td>[text, money]</td>\n",
       "      <td>[, ]</td>\n",
       "      <td>[funding_source, budget]</td>\n",
       "      <td>[Funding Source, Budget]</td>\n",
       "      <td>2016-07-13T16:52:30.000Z</td>\n",
       "      <td></td>\n",
       "      <td>21.000</td>\n",
       "      <td>uuui-htp8</td>\n",
       "      <td>Funding Sources</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>{'page_views_last_week': 4, 'page_views_last_m...</td>\n",
       "      <td>[7nqp-ya57, kf75-36tp]</td>\n",
       "      <td>official</td>\n",
       "      <td>filter</td>\n",
       "      <td>2017-07-27T14:23:39.000Z</td>\n",
       "      <td>{'page_views_last_week': 4, 'page_views_last_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>2016-10-07T23:19:08.000Z</td>\n",
       "      <td></td>\n",
       "      <td>0.000</td>\n",
       "      <td>9fp5-7dsy</td>\n",
       "      <td>City Boundary</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>{'page_views_total': 27, 'page_views_total_log...</td>\n",
       "      <td>None</td>\n",
       "      <td>official</td>\n",
       "      <td>map</td>\n",
       "      <td>2016-10-07T23:20:31.000Z</td>\n",
       "      <td>{'page_views_total': 27, 'page_views_total_log...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>2016-10-07T23:22:05.000Z</td>\n",
       "      <td></td>\n",
       "      <td>0.000</td>\n",
       "      <td>cspu-rwqv</td>\n",
       "      <td>Projects</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>{'page_views_total': 20, 'page_views_total_log...</td>\n",
       "      <td>None</td>\n",
       "      <td>official</td>\n",
       "      <td>map</td>\n",
       "      <td>2016-10-07T23:25:10.000Z</td>\n",
       "      <td>{'page_views_total': 20, 'page_views_total_log...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  attribution                                   columns_datatype  \\\n",
       "0        None  [location, text, photo, text, text, text, text...   \n",
       "1        None                                 [text, text, text]   \n",
       "2        None                                                 []   \n",
       "3        None  [text, text, text, text, text, number, text, t...   \n",
       "4        None  [text, text, text, text, text, text, text, tex...   \n",
       "5        None                                 [text, text, text]   \n",
       "6        None                                      [text, money]   \n",
       "7        None                                      [text, money]   \n",
       "8        None                                                 []   \n",
       "9        None                                                 []   \n",
       "\n",
       "                                 columns_description  \\\n",
       "0                   [, , , , , , , , , , , , , , , ]   \n",
       "1                                             [, , ]   \n",
       "2                                                 []   \n",
       "3                       [, , , , , , , , , , , , , ]   \n",
       "4  [, , , , , , , , , Please be sure to include \"...   \n",
       "5                                             [, , ]   \n",
       "6                                               [, ]   \n",
       "7                                               [, ]   \n",
       "8                                                 []   \n",
       "9                                                 []   \n",
       "\n",
       "                                  columns_field_name  \\\n",
       "0  [location, icon, project_image, category_id, l...   \n",
       "1                                   [icon, name, id]   \n",
       "2                                                 []   \n",
       "3  [city, location_id, project_id, project_name, ...   \n",
       "4  [document_10, document_9, document_8, document...   \n",
       "5                                   [id, name, icon]   \n",
       "6                           [funding_source, budget]   \n",
       "7                           [funding_source, budget]   \n",
       "8                                                 []   \n",
       "9                                                 []   \n",
       "\n",
       "                                        columns_name  \\\n",
       "0  [Automated Geocoding, Icon, Project Image, Cat...   \n",
       "1                                   [Icon, name, id]   \n",
       "2                                                 []   \n",
       "3  [City, Location Id, Project Id, Project Name, ...   \n",
       "4  [Document 10, Document 9, Document 8, Document...   \n",
       "5                                   [id, name, Icon]   \n",
       "6                           [Funding Source, Budget]   \n",
       "7                           [Funding Source, Budget]   \n",
       "8                                                 []   \n",
       "9                                                 []   \n",
       "\n",
       "                  createdAt description  download_count         id  \\\n",
       "0  2015-02-12T19:14:36.000Z                      15.000  9csq-7i9t   \n",
       "1  2015-02-05T18:41:29.000Z                       7.000  xwab-9myw   \n",
       "2  2015-02-09T20:48:39.000Z                       7.000  963m-jiy8   \n",
       "3  2015-04-15T05:15:46.000Z                       3.000  hz5n-bkfx   \n",
       "4  2015-08-28T12:03:36.000Z                      25.000  7nqp-ya57   \n",
       "5  2015-08-28T12:01:45.000Z                      22.000  kuue-gg3b   \n",
       "6  2016-08-09T19:40:33.000Z                      32.000  5h3i-jdcp   \n",
       "7  2016-07-13T16:52:30.000Z                      21.000  uuui-htp8   \n",
       "8  2016-10-07T23:19:08.000Z                       0.000  9fp5-7dsy   \n",
       "9  2016-10-07T23:22:05.000Z                       0.000  cspu-rwqv   \n",
       "\n",
       "                                            name  nbe_fxf  obe_fxf  \\\n",
       "0               Bonds Project Dataset (Official)      nan      nan   \n",
       "1            Bonds Application - Reference Table      nan      nan   \n",
       "2                                 Wards 2011.shp      nan      nan   \n",
       "3                      Project Dataset (Staging)      nan      nan   \n",
       "4                        Capital Project Dataset      nan      nan   \n",
       "5  Capital Project Application - Reference Table      nan      nan   \n",
       "6                    Funding Sources (Pie Chart)      nan      nan   \n",
       "7                                Funding Sources      nan      nan   \n",
       "8                                  City Boundary      nan      nan   \n",
       "9                                       Projects      nan      nan   \n",
       "\n",
       "                                          page_views              parent_fxf  \\\n",
       "0  {'page_views_last_week': 8, 'page_views_last_m...                    None   \n",
       "1  {'page_views_total': 66, 'page_views_total_log...                    None   \n",
       "2  {'page_views_total': 23, 'page_views_total_log...                    None   \n",
       "3  {'page_views_total': 15, 'page_views_total_log...                    None   \n",
       "4  {'page_views_last_week': 0, 'page_views_last_m...                    None   \n",
       "5  {'page_views_total': 63, 'page_views_total_log...                    None   \n",
       "6  {'page_views_last_week': 2, 'page_views_last_m...  [7nqp-ya57, kf75-36tp]   \n",
       "7  {'page_views_last_week': 4, 'page_views_last_m...  [7nqp-ya57, kf75-36tp]   \n",
       "8  {'page_views_total': 27, 'page_views_total_log...                    None   \n",
       "9  {'page_views_total': 20, 'page_views_total_log...                    None   \n",
       "\n",
       "  provenance     type                 updatedAt  \\\n",
       "0   official  dataset  2017-08-11T19:53:36.000Z   \n",
       "1   official  dataset  2015-04-17T15:30:47.000Z   \n",
       "2   official      map  2016-08-30T22:00:21.000Z   \n",
       "3   official  dataset  2015-04-15T05:20:19.000Z   \n",
       "4   official  dataset  2017-07-27T14:24:01.000Z   \n",
       "5   official  dataset  2015-10-02T23:07:35.000Z   \n",
       "6   official    chart  2017-07-27T14:23:39.000Z   \n",
       "7   official   filter  2017-07-27T14:23:39.000Z   \n",
       "8   official      map  2016-10-07T23:20:31.000Z   \n",
       "9   official      map  2016-10-07T23:25:10.000Z   \n",
       "\n",
       "                                          view_count  \n",
       "0  {'page_views_last_week': 8, 'page_views_last_m...  \n",
       "1  {'page_views_total': 66, 'page_views_total_log...  \n",
       "2  {'page_views_total': 23, 'page_views_total_log...  \n",
       "3  {'page_views_total': 15, 'page_views_total_log...  \n",
       "4  {'page_views_last_week': 0, 'page_views_last_m...  \n",
       "5  {'page_views_total': 63, 'page_views_total_log...  \n",
       "6  {'page_views_last_week': 2, 'page_views_last_m...  \n",
       "7  {'page_views_last_week': 4, 'page_views_last_m...  \n",
       "8  {'page_views_total': 27, 'page_views_total_log...  \n",
       "9  {'page_views_total': 20, 'page_views_total_log...  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resource.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>page_views_last_month</th>\n",
       "      <th>page_views_last_month_log</th>\n",
       "      <th>page_views_last_week</th>\n",
       "      <th>page_views_last_week_log</th>\n",
       "      <th>page_views_total</th>\n",
       "      <th>page_views_total_log</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>34.000</td>\n",
       "      <td>5.129</td>\n",
       "      <td>8.000</td>\n",
       "      <td>3.170</td>\n",
       "      <td>464.000</td>\n",
       "      <td>8.861</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>66.000</td>\n",
       "      <td>6.066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>23.000</td>\n",
       "      <td>4.585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>15.000</td>\n",
       "      <td>4.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.000</td>\n",
       "      <td>2.322</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>393.000</td>\n",
       "      <td>8.622</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>63.000</td>\n",
       "      <td>6.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6.000</td>\n",
       "      <td>2.807</td>\n",
       "      <td>2.000</td>\n",
       "      <td>1.585</td>\n",
       "      <td>62.000</td>\n",
       "      <td>5.977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>10.000</td>\n",
       "      <td>3.459</td>\n",
       "      <td>4.000</td>\n",
       "      <td>2.322</td>\n",
       "      <td>59.000</td>\n",
       "      <td>5.907</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>4.000</td>\n",
       "      <td>2.322</td>\n",
       "      <td>4.000</td>\n",
       "      <td>2.322</td>\n",
       "      <td>27.000</td>\n",
       "      <td>4.807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>20.000</td>\n",
       "      <td>4.392</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   page_views_last_month  page_views_last_month_log  page_views_last_week  \\\n",
       "0                 34.000                      5.129                 8.000   \n",
       "1                  1.000                      1.000                 0.000   \n",
       "2                  0.000                      0.000                 0.000   \n",
       "3                  0.000                      0.000                 0.000   \n",
       "4                  4.000                      2.322                 0.000   \n",
       "5                  0.000                      0.000                 0.000   \n",
       "6                  6.000                      2.807                 2.000   \n",
       "7                 10.000                      3.459                 4.000   \n",
       "8                  4.000                      2.322                 4.000   \n",
       "9                  1.000                      1.000                 1.000   \n",
       "\n",
       "   page_views_last_week_log  page_views_total  page_views_total_log  \n",
       "0                     3.170           464.000                 8.861  \n",
       "1                     0.000            66.000                 6.066  \n",
       "2                     0.000            23.000                 4.585  \n",
       "3                     0.000            15.000                 4.000  \n",
       "4                     0.000           393.000                 8.622  \n",
       "5                     0.000            63.000                 6.000  \n",
       "6                     1.585            62.000                 5.977  \n",
       "7                     2.322            59.000                 5.907  \n",
       "8                     2.322            27.000                 4.807  \n",
       "9                     1.000            20.000                 4.392  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#unpack views:\n",
    "views = resource.view_count.apply(pd.Series)\n",
    "views.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Concatenate all these dfs into one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>attribution</th>\n",
       "      <th>columns_datatype</th>\n",
       "      <th>columns_description</th>\n",
       "      <th>columns_field_name</th>\n",
       "      <th>columns_name</th>\n",
       "      <th>createdAt</th>\n",
       "      <th>description</th>\n",
       "      <th>download_count</th>\n",
       "      <th>id</th>\n",
       "      <th>name</th>\n",
       "      <th>...</th>\n",
       "      <th>domain_tags</th>\n",
       "      <th>tags</th>\n",
       "      <th>page_views_last_month</th>\n",
       "      <th>page_views_last_month_log</th>\n",
       "      <th>page_views_last_week</th>\n",
       "      <th>page_views_last_week_log</th>\n",
       "      <th>page_views_total</th>\n",
       "      <th>page_views_total_log</th>\n",
       "      <th>domain</th>\n",
       "      <th>license</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>None</td>\n",
       "      <td>[percent, number, number, calendar_date]</td>\n",
       "      <td>[, , , ]</td>\n",
       "      <td>[percent_of_valid_parking_meter_problem_servic...</td>\n",
       "      <td>[Percent of valid parking meter problem servic...</td>\n",
       "      <td>2016-02-22T20:36:45.000Z</td>\n",
       "      <td></td>\n",
       "      <td>17.000</td>\n",
       "      <td>gnby-xhxz</td>\n",
       "      <td>Parking Complaints Bar Chart</td>\n",
       "      <td>...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>17.000</td>\n",
       "      <td>4.170</td>\n",
       "      <td>4.000</td>\n",
       "      <td>2.322</td>\n",
       "      <td>274.000</td>\n",
       "      <td>8.103</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>None</td>\n",
       "      <td>[calendar_date, number]</td>\n",
       "      <td>[, ]</td>\n",
       "      <td>[fiscal_year_originated, count]</td>\n",
       "      <td>[Fiscal Year Originated, Count]</td>\n",
       "      <td>2016-08-09T20:05:43.000Z</td>\n",
       "      <td></td>\n",
       "      <td>10.000</td>\n",
       "      <td>mzr8-ks94</td>\n",
       "      <td>Home Buyer Loans Graph</td>\n",
       "      <td>...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>18.000</td>\n",
       "      <td>4.248</td>\n",
       "      <td>6.000</td>\n",
       "      <td>2.807</td>\n",
       "      <td>272.000</td>\n",
       "      <td>8.093</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>None</td>\n",
       "      <td>[number, number, percent, calendar_date]</td>\n",
       "      <td>[, , , ]</td>\n",
       "      <td>[potholes_filled, lane_miles_swept, percent_of...</td>\n",
       "      <td>[Potholes Filled, Lane Miles Repaved, Percent ...</td>\n",
       "      <td>2015-10-09T16:19:29.000Z</td>\n",
       "      <td></td>\n",
       "      <td>7.000</td>\n",
       "      <td>5kha-4jsz</td>\n",
       "      <td>Potholes Filled by FY (Bar Chart)</td>\n",
       "      <td>...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>38.000</td>\n",
       "      <td>5.285</td>\n",
       "      <td>3.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>269.000</td>\n",
       "      <td>8.077</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>None</td>\n",
       "      <td>[number, text]</td>\n",
       "      <td>[, ]</td>\n",
       "      <td>[number_of_units, fiscal_year]</td>\n",
       "      <td>[NUMBER OF UNITS, Fiscal Year]</td>\n",
       "      <td>2015-09-04T11:17:55.000Z</td>\n",
       "      <td></td>\n",
       "      <td>5.000</td>\n",
       "      <td>ikzj-e4dt</td>\n",
       "      <td>Lane Miles Repaved</td>\n",
       "      <td>...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>28.000</td>\n",
       "      <td>4.858</td>\n",
       "      <td>5.000</td>\n",
       "      <td>2.585</td>\n",
       "      <td>264.000</td>\n",
       "      <td>8.050</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>None</td>\n",
       "      <td>[number, number, text]</td>\n",
       "      <td>[, , ]</td>\n",
       "      <td>[lipos_admissions_per_100k, nvmhi_admission_pe...</td>\n",
       "      <td>[LIPOS Admissions per 100K, NVMHI Admission pe...</td>\n",
       "      <td>2016-06-27T20:10:49.000Z</td>\n",
       "      <td></td>\n",
       "      <td>5.000</td>\n",
       "      <td>e5v5-i25b</td>\n",
       "      <td>NVMHI Admissions</td>\n",
       "      <td>...</td>\n",
       "      <td>[delete, dchs]</td>\n",
       "      <td>[]</td>\n",
       "      <td>38.000</td>\n",
       "      <td>5.285</td>\n",
       "      <td>5.000</td>\n",
       "      <td>2.585</td>\n",
       "      <td>263.000</td>\n",
       "      <td>8.044</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>None</td>\n",
       "      <td>[percent, calendar_date, number, percent, number]</td>\n",
       "      <td>[, , , , ]</td>\n",
       "      <td>[diversion_rate, calendar_year, missed_collect...</td>\n",
       "      <td>[Diversion rate, Calendar Year, Missed Collect...</td>\n",
       "      <td>2015-09-04T13:38:06.000Z</td>\n",
       "      <td></td>\n",
       "      <td>6.000</td>\n",
       "      <td>ayk9-7672</td>\n",
       "      <td>Tons Recycling Dropoff Centers</td>\n",
       "      <td>...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>15.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>1.585</td>\n",
       "      <td>246.000</td>\n",
       "      <td>7.948</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>None</td>\n",
       "      <td>[number, percent, calendar_date, number, number]</td>\n",
       "      <td>[, , , , ]</td>\n",
       "      <td>[number_of_property_owners_trained, percent_of...</td>\n",
       "      <td>[Number of property owners trained, Percent of...</td>\n",
       "      <td>2015-10-29T19:42:42.000Z</td>\n",
       "      <td>Office of Housing Data</td>\n",
       "      <td>11.000</td>\n",
       "      <td>9vuk-i6y2</td>\n",
       "      <td>Property Owners Trainined</td>\n",
       "      <td>...</td>\n",
       "      <td>[housing]</td>\n",
       "      <td>[]</td>\n",
       "      <td>30.000</td>\n",
       "      <td>4.954</td>\n",
       "      <td>4.000</td>\n",
       "      <td>2.322</td>\n",
       "      <td>246.000</td>\n",
       "      <td>7.948</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>None</td>\n",
       "      <td>[percent, text, text, percent, text, calendar_...</td>\n",
       "      <td>[, , , , , , , , , , , , ]</td>\n",
       "      <td>[problem, protected_class_group, action, prob_...</td>\n",
       "      <td>[Percent Problem, Protected Class Group, Actio...</td>\n",
       "      <td>2016-01-13T17:58:05.000Z</td>\n",
       "      <td></td>\n",
       "      <td>17.000</td>\n",
       "      <td>mfwu-24pf</td>\n",
       "      <td>Fair Housing Sites Tested Per Year</td>\n",
       "      <td>...</td>\n",
       "      <td>[fair housing, housing]</td>\n",
       "      <td>[]</td>\n",
       "      <td>20.000</td>\n",
       "      <td>4.392</td>\n",
       "      <td>3.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>237.000</td>\n",
       "      <td>7.895</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>None</td>\n",
       "      <td>[number, calendar_date]</td>\n",
       "      <td>[, ]</td>\n",
       "      <td>[count, fyear]</td>\n",
       "      <td>[Count, FYEAR]</td>\n",
       "      <td>2016-07-12T20:20:28.000Z</td>\n",
       "      <td>DCHS Rent Relief Data. Well Being and Safety f...</td>\n",
       "      <td>7.000</td>\n",
       "      <td>gr5b-bvxa</td>\n",
       "      <td>Number of Residents Assisted Through Rent Relief</td>\n",
       "      <td>...</td>\n",
       "      <td>[well, rent relief, well-being and safety for ...</td>\n",
       "      <td>[]</td>\n",
       "      <td>26.000</td>\n",
       "      <td>4.755</td>\n",
       "      <td>3.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>233.000</td>\n",
       "      <td>7.870</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>None</td>\n",
       "      <td>[calendar_date, number, text]</td>\n",
       "      <td>[, , ]</td>\n",
       "      <td>[season, pts, outcome]</td>\n",
       "      <td>[Season, PTS, Outcome]</td>\n",
       "      <td>2015-07-24T18:52:32.000Z</td>\n",
       "      <td>Lakers game outcomes and Kobe box scores, 1996...</td>\n",
       "      <td>3.000</td>\n",
       "      <td>22w3-jbsg</td>\n",
       "      <td>Kobe average points in wins and losses, by season</td>\n",
       "      <td>...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>31.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>226.000</td>\n",
       "      <td>7.827</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   attribution                                   columns_datatype  \\\n",
       "15        None           [percent, number, number, calendar_date]   \n",
       "16        None                            [calendar_date, number]   \n",
       "17        None           [number, number, percent, calendar_date]   \n",
       "18        None                                     [number, text]   \n",
       "19        None                             [number, number, text]   \n",
       "20        None  [percent, calendar_date, number, percent, number]   \n",
       "21        None   [number, percent, calendar_date, number, number]   \n",
       "22        None  [percent, text, text, percent, text, calendar_...   \n",
       "23        None                            [number, calendar_date]   \n",
       "24        None                      [calendar_date, number, text]   \n",
       "\n",
       "           columns_description  \\\n",
       "15                    [, , , ]   \n",
       "16                        [, ]   \n",
       "17                    [, , , ]   \n",
       "18                        [, ]   \n",
       "19                      [, , ]   \n",
       "20                  [, , , , ]   \n",
       "21                  [, , , , ]   \n",
       "22  [, , , , , , , , , , , , ]   \n",
       "23                        [, ]   \n",
       "24                      [, , ]   \n",
       "\n",
       "                                   columns_field_name  \\\n",
       "15  [percent_of_valid_parking_meter_problem_servic...   \n",
       "16                    [fiscal_year_originated, count]   \n",
       "17  [potholes_filled, lane_miles_swept, percent_of...   \n",
       "18                     [number_of_units, fiscal_year]   \n",
       "19  [lipos_admissions_per_100k, nvmhi_admission_pe...   \n",
       "20  [diversion_rate, calendar_year, missed_collect...   \n",
       "21  [number_of_property_owners_trained, percent_of...   \n",
       "22  [problem, protected_class_group, action, prob_...   \n",
       "23                                     [count, fyear]   \n",
       "24                             [season, pts, outcome]   \n",
       "\n",
       "                                         columns_name  \\\n",
       "15  [Percent of valid parking meter problem servic...   \n",
       "16                    [Fiscal Year Originated, Count]   \n",
       "17  [Potholes Filled, Lane Miles Repaved, Percent ...   \n",
       "18                     [NUMBER OF UNITS, Fiscal Year]   \n",
       "19  [LIPOS Admissions per 100K, NVMHI Admission pe...   \n",
       "20  [Diversion rate, Calendar Year, Missed Collect...   \n",
       "21  [Number of property owners trained, Percent of...   \n",
       "22  [Percent Problem, Protected Class Group, Actio...   \n",
       "23                                     [Count, FYEAR]   \n",
       "24                             [Season, PTS, Outcome]   \n",
       "\n",
       "                   createdAt  \\\n",
       "15  2016-02-22T20:36:45.000Z   \n",
       "16  2016-08-09T20:05:43.000Z   \n",
       "17  2015-10-09T16:19:29.000Z   \n",
       "18  2015-09-04T11:17:55.000Z   \n",
       "19  2016-06-27T20:10:49.000Z   \n",
       "20  2015-09-04T13:38:06.000Z   \n",
       "21  2015-10-29T19:42:42.000Z   \n",
       "22  2016-01-13T17:58:05.000Z   \n",
       "23  2016-07-12T20:20:28.000Z   \n",
       "24  2015-07-24T18:52:32.000Z   \n",
       "\n",
       "                                          description  download_count  \\\n",
       "15                                                             17.000   \n",
       "16                                                             10.000   \n",
       "17                                                              7.000   \n",
       "18                                                              5.000   \n",
       "19                                                              5.000   \n",
       "20                                                              6.000   \n",
       "21                             Office of Housing Data          11.000   \n",
       "22                                                             17.000   \n",
       "23  DCHS Rent Relief Data. Well Being and Safety f...           7.000   \n",
       "24  Lakers game outcomes and Kobe box scores, 1996...           3.000   \n",
       "\n",
       "           id                                               name   ...    \\\n",
       "15  gnby-xhxz                       Parking Complaints Bar Chart   ...     \n",
       "16  mzr8-ks94                             Home Buyer Loans Graph   ...     \n",
       "17  5kha-4jsz                  Potholes Filled by FY (Bar Chart)   ...     \n",
       "18  ikzj-e4dt                                 Lane Miles Repaved   ...     \n",
       "19  e5v5-i25b                                   NVMHI Admissions   ...     \n",
       "20  ayk9-7672                     Tons Recycling Dropoff Centers   ...     \n",
       "21  9vuk-i6y2                          Property Owners Trainined   ...     \n",
       "22  mfwu-24pf                 Fair Housing Sites Tested Per Year   ...     \n",
       "23  gr5b-bvxa   Number of Residents Assisted Through Rent Relief   ...     \n",
       "24  22w3-jbsg  Kobe average points in wins and losses, by season   ...     \n",
       "\n",
       "                                          domain_tags  tags  \\\n",
       "15                                                 []    []   \n",
       "16                                                 []    []   \n",
       "17                                                 []    []   \n",
       "18                                                 []    []   \n",
       "19                                     [delete, dchs]    []   \n",
       "20                                                 []    []   \n",
       "21                                          [housing]    []   \n",
       "22                            [fair housing, housing]    []   \n",
       "23  [well, rent relief, well-being and safety for ...    []   \n",
       "24                                                 []    []   \n",
       "\n",
       "   page_views_last_month page_views_last_month_log page_views_last_week  \\\n",
       "15                17.000                     4.170                4.000   \n",
       "16                18.000                     4.248                6.000   \n",
       "17                38.000                     5.285                3.000   \n",
       "18                28.000                     4.858                5.000   \n",
       "19                38.000                     5.285                5.000   \n",
       "20                15.000                     4.000                2.000   \n",
       "21                30.000                     4.954                4.000   \n",
       "22                20.000                     4.392                3.000   \n",
       "23                26.000                     4.755                3.000   \n",
       "24                31.000                     5.000                0.000   \n",
       "\n",
       "   page_views_last_week_log page_views_total page_views_total_log  \\\n",
       "15                    2.322          274.000                8.103   \n",
       "16                    2.807          272.000                8.093   \n",
       "17                    2.000          269.000                8.077   \n",
       "18                    2.585          264.000                8.050   \n",
       "19                    2.585          263.000                8.044   \n",
       "20                    1.585          246.000                7.948   \n",
       "21                    2.322          246.000                7.948   \n",
       "22                    2.000          237.000                7.895   \n",
       "23                    2.000          233.000                7.870   \n",
       "24                    0.000          226.000                7.827   \n",
       "\n",
       "                        domain license  \n",
       "15  dashboard.alexandriava.gov     NaN  \n",
       "16  dashboard.alexandriava.gov     NaN  \n",
       "17  dashboard.alexandriava.gov     NaN  \n",
       "18  dashboard.alexandriava.gov     NaN  \n",
       "19  dashboard.alexandriava.gov     NaN  \n",
       "20  dashboard.alexandriava.gov     NaN  \n",
       "21  dashboard.alexandriava.gov     NaN  \n",
       "22  dashboard.alexandriava.gov     NaN  \n",
       "23  dashboard.alexandriava.gov     NaN  \n",
       "24  dashboard.alexandriava.gov     NaN  \n",
       "\n",
       "[10 rows x 31 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_mash = pd.concat([resource, classification, views, metadata], axis=1)\n",
    "data_mash[15:25]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['attribution', 'columns_datatype', 'columns_description',\n",
       "       'columns_field_name', 'columns_name', 'createdAt', 'description',\n",
       "       'download_count', 'id', 'name', 'nbe_fxf', 'obe_fxf', 'page_views',\n",
       "       'parent_fxf', 'provenance', 'type', 'updatedAt', 'view_count',\n",
       "       'categories', 'domain_category', 'domain_metadata', 'domain_tags',\n",
       "       'tags', 'page_views_last_month', 'page_views_last_month_log',\n",
       "       'page_views_last_week', 'page_views_last_week_log', 'page_views_total',\n",
       "       'page_views_total_log', 'domain', 'license'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_mash.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "      <th>attribution</th>\n",
       "      <th>columns_field_name</th>\n",
       "      <th>columns_name</th>\n",
       "      <th>type</th>\n",
       "      <th>categories</th>\n",
       "      <th>domain_category</th>\n",
       "      <th>domain_tags</th>\n",
       "      <th>provenance</th>\n",
       "      <th>download_count</th>\n",
       "      <th>page_views_last_month</th>\n",
       "      <th>page_views_last_week</th>\n",
       "      <th>page_views_total</th>\n",
       "      <th>page_views_total_log</th>\n",
       "      <th>domain</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Resident Survey: Air Quality</td>\n",
       "      <td>For the relevant summary reports and technical...</td>\n",
       "      <td>None</td>\n",
       "      <td>[of_total_weight, answergroupstep1_group, comp...</td>\n",
       "      <td>[Response Percent (weighted), Answer, Comparis...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[finance]</td>\n",
       "      <td>None</td>\n",
       "      <td>[resident survey, city strategic plan]</td>\n",
       "      <td>official</td>\n",
       "      <td>4.000</td>\n",
       "      <td>60.000</td>\n",
       "      <td>8.000</td>\n",
       "      <td>220.000</td>\n",
       "      <td>7.788</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Rental Unit Gain and Loss Graph</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>[fiscal_year, units_lost, new_units_added]</td>\n",
       "      <td>[Fiscal Year, Rental Units Lost, New Rental Un...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[assisted rental, housing]</td>\n",
       "      <td>official</td>\n",
       "      <td>14.000</td>\n",
       "      <td>6.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>217.000</td>\n",
       "      <td>7.768</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Education Level- Filter</td>\n",
       "      <td>Worfkforce Development Center Education Ranges</td>\n",
       "      <td>None</td>\n",
       "      <td>[fystartdate, count, geom, fiscalyear, fyendda...</td>\n",
       "      <td>[FYStartDate, count, geom, FiscalYear, FYEndDa...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[education level, increase employment, workfor...</td>\n",
       "      <td>official</td>\n",
       "      <td>5.000</td>\n",
       "      <td>22.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>216.000</td>\n",
       "      <td>7.762</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Issue types, as a percent of all types</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>[fiscal_year, percent_unsubstantiated, percent...</td>\n",
       "      <td>[Fiscal Year, Percent Unsubstantiated, Percent...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>5.000</td>\n",
       "      <td>31.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>214.000</td>\n",
       "      <td>7.748</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Resident Survey: Ease of Travel by Bike</td>\n",
       "      <td>For the relevant summary reports and technical...</td>\n",
       "      <td>None</td>\n",
       "      <td>[of_total_weight, answergroupstep1_group, comp...</td>\n",
       "      <td>[Response Percent (weighted), Answer, Comparis...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[finance]</td>\n",
       "      <td>None</td>\n",
       "      <td>[resident survey, city strategic plan]</td>\n",
       "      <td>official</td>\n",
       "      <td>2.000</td>\n",
       "      <td>61.000</td>\n",
       "      <td>12.000</td>\n",
       "      <td>214.000</td>\n",
       "      <td>7.748</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Pothole Service Requests by Fiscal Year</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>[fiscal_year, count]</td>\n",
       "      <td>[Fiscal Year, Count]</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>2.000</td>\n",
       "      <td>34.000</td>\n",
       "      <td>10.000</td>\n",
       "      <td>209.000</td>\n",
       "      <td>7.714</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Signal Services Requests Column Graph</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>[percent_of_signals_receiving_preventative_mai...</td>\n",
       "      <td>[Percent of signals receiving preventative mai...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>1.000</td>\n",
       "      <td>30.000</td>\n",
       "      <td>11.000</td>\n",
       "      <td>207.000</td>\n",
       "      <td>7.700</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Local GDP 2016</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>[number_of_accounts, business_license_class_na...</td>\n",
       "      <td>[Number of Accounts, Business License Class Na...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[finance]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[local gross domestic product, city strategic ...</td>\n",
       "      <td>official</td>\n",
       "      <td>4.000</td>\n",
       "      <td>39.000</td>\n",
       "      <td>6.000</td>\n",
       "      <td>207.000</td>\n",
       "      <td>7.700</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Number of Youth Counseled through the Workforc...</td>\n",
       "      <td>Work Force Development Center, Youth Participants</td>\n",
       "      <td>None</td>\n",
       "      <td>[fytarget, fyenddate, geom, count, fystartdate...</td>\n",
       "      <td>[FYTarget, FYEndDate, geom, count, FYStartDate...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[politics]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[youth work experience, work experience, workf...</td>\n",
       "      <td>official</td>\n",
       "      <td>4.000</td>\n",
       "      <td>44.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>206.000</td>\n",
       "      <td>7.693</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>DCHS_BH_Opioid Use</td>\n",
       "      <td>Drug use report from Alaina</td>\n",
       "      <td>None</td>\n",
       "      <td>[primary_opiates_category_combined, fiscal_yea...</td>\n",
       "      <td>[Opiates Category Combined, Fiscal Year, Perce...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[behavioral health, detox, dchs]</td>\n",
       "      <td>official</td>\n",
       "      <td>4.000</td>\n",
       "      <td>9.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>204.000</td>\n",
       "      <td>7.679</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 name  \\\n",
       "25                       Resident Survey: Air Quality   \n",
       "26                    Rental Unit Gain and Loss Graph   \n",
       "27                            Education Level- Filter   \n",
       "28             Issue types, as a percent of all types   \n",
       "29            Resident Survey: Ease of Travel by Bike   \n",
       "30            Pothole Service Requests by Fiscal Year   \n",
       "31              Signal Services Requests Column Graph   \n",
       "32                                     Local GDP 2016   \n",
       "33  Number of Youth Counseled through the Workforc...   \n",
       "34                                 DCHS_BH_Opioid Use   \n",
       "\n",
       "                                          description attribution  \\\n",
       "25  For the relevant summary reports and technical...        None   \n",
       "26                                                           None   \n",
       "27     Worfkforce Development Center Education Ranges        None   \n",
       "28                                                           None   \n",
       "29  For the relevant summary reports and technical...        None   \n",
       "30                                                           None   \n",
       "31                                                           None   \n",
       "32                                                           None   \n",
       "33  Work Force Development Center, Youth Participants        None   \n",
       "34                        Drug use report from Alaina        None   \n",
       "\n",
       "                                   columns_field_name  \\\n",
       "25  [of_total_weight, answergroupstep1_group, comp...   \n",
       "26         [fiscal_year, units_lost, new_units_added]   \n",
       "27  [fystartdate, count, geom, fiscalyear, fyendda...   \n",
       "28  [fiscal_year, percent_unsubstantiated, percent...   \n",
       "29  [of_total_weight, answergroupstep1_group, comp...   \n",
       "30                               [fiscal_year, count]   \n",
       "31  [percent_of_signals_receiving_preventative_mai...   \n",
       "32  [number_of_accounts, business_license_class_na...   \n",
       "33  [fytarget, fyenddate, geom, count, fystartdate...   \n",
       "34  [primary_opiates_category_combined, fiscal_yea...   \n",
       "\n",
       "                                         columns_name   type  categories  \\\n",
       "25  [Response Percent (weighted), Answer, Comparis...  chart   [finance]   \n",
       "26  [Fiscal Year, Rental Units Lost, New Rental Un...  chart          []   \n",
       "27  [FYStartDate, count, geom, FiscalYear, FYEndDa...  chart          []   \n",
       "28  [Fiscal Year, Percent Unsubstantiated, Percent...  chart          []   \n",
       "29  [Response Percent (weighted), Answer, Comparis...  chart   [finance]   \n",
       "30                               [Fiscal Year, Count]  chart          []   \n",
       "31  [Percent of signals receiving preventative mai...  chart          []   \n",
       "32  [Number of Accounts, Business License Class Na...  chart   [finance]   \n",
       "33  [FYTarget, FYEndDate, geom, count, FYStartDate...  chart  [politics]   \n",
       "34  [Opiates Category Combined, Fiscal Year, Perce...  chart          []   \n",
       "\n",
       "   domain_category                                        domain_tags  \\\n",
       "25            None             [resident survey, city strategic plan]   \n",
       "26             NaN                         [assisted rental, housing]   \n",
       "27             NaN  [education level, increase employment, workfor...   \n",
       "28            None                                                 []   \n",
       "29            None             [resident survey, city strategic plan]   \n",
       "30             NaN                                                 []   \n",
       "31             NaN                                                 []   \n",
       "32             NaN  [local gross domestic product, city strategic ...   \n",
       "33             NaN  [youth work experience, work experience, workf...   \n",
       "34             NaN                   [behavioral health, detox, dchs]   \n",
       "\n",
       "   provenance  download_count  page_views_last_month  page_views_last_week  \\\n",
       "25   official           4.000                 60.000                 8.000   \n",
       "26   official          14.000                  6.000                 1.000   \n",
       "27   official           5.000                 22.000                 1.000   \n",
       "28   official           5.000                 31.000                 1.000   \n",
       "29   official           2.000                 61.000                12.000   \n",
       "30   official           2.000                 34.000                10.000   \n",
       "31   official           1.000                 30.000                11.000   \n",
       "32   official           4.000                 39.000                 6.000   \n",
       "33   official           4.000                 44.000                 4.000   \n",
       "34   official           4.000                  9.000                 0.000   \n",
       "\n",
       "    page_views_total  page_views_total_log                      domain  \n",
       "25           220.000                 7.788  dashboard.alexandriava.gov  \n",
       "26           217.000                 7.768  dashboard.alexandriava.gov  \n",
       "27           216.000                 7.762  dashboard.alexandriava.gov  \n",
       "28           214.000                 7.748  dashboard.alexandriava.gov  \n",
       "29           214.000                 7.748  dashboard.alexandriava.gov  \n",
       "30           209.000                 7.714  dashboard.alexandriava.gov  \n",
       "31           207.000                 7.700  dashboard.alexandriava.gov  \n",
       "32           207.000                 7.700  dashboard.alexandriava.gov  \n",
       "33           206.000                 7.693  dashboard.alexandriava.gov  \n",
       "34           204.000                 7.679  dashboard.alexandriava.gov  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols_to_keep = ['name', 'description', 'attribution', 'columns_field_name', 'columns_name', 'type', 'categories', \n",
    "                'domain_category','domain_tags', 'provenance', 'download_count','page_views_last_month',\n",
    "                'page_views_last_week','page_views_total', \n",
    "                'page_views_total_log', 'domain']\n",
    "cities = data_mash[cols_to_keep]\n",
    "cities[25:35]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(33452, 16)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cities.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA/Data Cleaning\n",
    "We are going to cluster these datasets by content and use those clusters for quantitative analysis. However, we first need to see what we're dealing with and drop and NaN data for views. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** See how many NaN view counts there are - we'll want to drop these **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NAs = cities[cities.page_views_total.isnull()] \n",
    "len(NAs.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.float64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(cities.iloc[0, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our page views column are floats, not lists (which could potentially be empty, as in other columns), so we're hopefully getting the true number of NaNs (0). This would square with what we expect - unless something went seriously wrong, Socrata's API will return view data for every dataset. However, not every city/state tags their datasets with categories/tags -- hence, some are missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NoneType"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(cities.attribution[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like the \"attribution\" attribute has many NoneTypes. That's ok - we don't want to use attribution for clustering, as it's too specific - we want to be able to cluster similar datasets across cities, whereas including attribution - e.g. \"Dallas Police Department\" -- will just skew our clusters to clustering within a given city."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we need to choose what attributes we'll use to create a text 'mash' from which we will use natural language processing and clustering tools. \n",
    "\n",
    "We need to choose carefully - as noted with attribution, the model we will build to do this is not a genius. It basically comes down to word appearance frequency is giant vectors, and how close these vectors are to each other in mathematical space. So, we want to include text attributes that won't skew how a dataset is classified."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create \"Mash\" from categories and tags:\n",
    "\n",
    "** This actually takes a fair amount of cleaning **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mash_df = cities.copy()\n",
    "\n",
    "#domain category is tricky because it contains both NaNs and NoneTypes. We'll replace both with a nothing str\n",
    "mash_df['domain_category'] = mash_df['domain_category'].replace([None], '') \n",
    "\n",
    "#combine domain_tags and categories lists of strings columns - easy enough\n",
    "mash_df['mash'] = mash_df.domain_tags + mash_df.categories \n",
    "\n",
    "#CONVERT lists of strings into actual string\n",
    "mash_df['mash'] = mash_df['mash'].apply(lambda x: ','.join(map(str, x)))\n",
    "\n",
    "#now add domain_category string to our mash column\n",
    "mash_df['mash'] = mash_df.mash.str.cat(mash_df.domain_category, sep=' ') #this adds blank space if domain_category blank\n",
    "\n",
    "#NOW our mash column is real-life words separated by commas and spaces. we need to split them\n",
    "mash_df.mash = mash_df.mash.str.strip() #remove trailing whitespace\n",
    "mash_df.mash = mash_df.mash.str.replace(' ', \",\") #now replace all whitespaces with a comma to make splitting easier\n",
    "mash_df.mash = mash_df.mash.str.lower() #we need strings to all be lowercase for vectorizing purposes later\n",
    "mash_df.mash = mash_df.mash.str.split(',') #split big string on comma into actual words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['environmental', 'protection']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mash_df.iloc[433].mash"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "      <th>attribution</th>\n",
       "      <th>columns_field_name</th>\n",
       "      <th>columns_name</th>\n",
       "      <th>type</th>\n",
       "      <th>categories</th>\n",
       "      <th>domain_category</th>\n",
       "      <th>domain_tags</th>\n",
       "      <th>provenance</th>\n",
       "      <th>download_count</th>\n",
       "      <th>page_views_last_month</th>\n",
       "      <th>page_views_last_week</th>\n",
       "      <th>page_views_total</th>\n",
       "      <th>page_views_total_log</th>\n",
       "      <th>domain</th>\n",
       "      <th>mash</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>430</th>\n",
       "      <td>Children Who Are Confirmed By Child Protective...</td>\n",
       "      <td></td>\n",
       "      <td>http://datacenter.kidscount.org/</td>\n",
       "      <td>[location, timeframe, dataformat, date_time, g...</td>\n",
       "      <td>[Location, TimeFrame, DataFormat, Date Time, G...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>23.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>304.000</td>\n",
       "      <td>8.253</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>431</th>\n",
       "      <td>AABD Client Age - June 2013</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>[pctn, n, characteristics]</td>\n",
       "      <td>[PctN, N, Characteristics]</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>14.000</td>\n",
       "      <td>33.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>304.000</td>\n",
       "      <td>8.253</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>432</th>\n",
       "      <td>Reading Proficiency Changes over Time Line Chart</td>\n",
       "      <td></td>\n",
       "      <td>Socrata</td>\n",
       "      <td>[year_text, year_date, reading_proficiency_mee...</td>\n",
       "      <td>[Year, Date Time, Meets, Exceeds, Approaches, ...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[education]</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>21.000</td>\n",
       "      <td>25.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>302.000</td>\n",
       "      <td>8.243</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>[education]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>433</th>\n",
       "      <td>Wastewater Percentage Reused</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>[percentage_reused, wastewater_reused_mgd, tot...</td>\n",
       "      <td>[Percentage Reused, Wastewater Reused (MGD), T...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>Environmental Protection</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>26.000</td>\n",
       "      <td>31.000</td>\n",
       "      <td>7.000</td>\n",
       "      <td>302.000</td>\n",
       "      <td>8.243</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>[environmental, protection]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>434</th>\n",
       "      <td>TImeLIne of Class Size</td>\n",
       "      <td>This data reflects average class sizes from 20...</td>\n",
       "      <td>None</td>\n",
       "      <td>[year_string, class_size, year]</td>\n",
       "      <td>[Year String, Class Size, Year]</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[class size]</td>\n",
       "      <td>official</td>\n",
       "      <td>22.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>301.000</td>\n",
       "      <td>8.238</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>[class, size]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  name  \\\n",
       "430  Children Who Are Confirmed By Child Protective...   \n",
       "431                        AABD Client Age - June 2013   \n",
       "432   Reading Proficiency Changes over Time Line Chart   \n",
       "433                       Wastewater Percentage Reused   \n",
       "434                             TImeLIne of Class Size   \n",
       "\n",
       "                                           description  \\\n",
       "430                                                      \n",
       "431                                                      \n",
       "432                                                      \n",
       "433                                                      \n",
       "434  This data reflects average class sizes from 20...   \n",
       "\n",
       "                          attribution  \\\n",
       "430  http://datacenter.kidscount.org/   \n",
       "431                              None   \n",
       "432                           Socrata   \n",
       "433                              None   \n",
       "434                              None   \n",
       "\n",
       "                                    columns_field_name  \\\n",
       "430  [location, timeframe, dataformat, date_time, g...   \n",
       "431                         [pctn, n, characteristics]   \n",
       "432  [year_text, year_date, reading_proficiency_mee...   \n",
       "433  [percentage_reused, wastewater_reused_mgd, tot...   \n",
       "434                    [year_string, class_size, year]   \n",
       "\n",
       "                                          columns_name     type   categories  \\\n",
       "430  [Location, TimeFrame, DataFormat, Date Time, G...  dataset           []   \n",
       "431                         [PctN, N, Characteristics]    chart           []   \n",
       "432  [Year, Date Time, Meets, Exceeds, Approaches, ...    chart  [education]   \n",
       "433  [Percentage Reused, Wastewater Reused (MGD), T...    chart           []   \n",
       "434                    [Year String, Class Size, Year]    chart           []   \n",
       "\n",
       "              domain_category   domain_tags provenance  download_count  \\\n",
       "430                                      []   official          23.000   \n",
       "431                                      []   official          14.000   \n",
       "432                                      []   official          21.000   \n",
       "433  Environmental Protection            []   official          26.000   \n",
       "434                            [class size]   official          22.000   \n",
       "\n",
       "     page_views_last_month  page_views_last_week  page_views_total  \\\n",
       "430                  0.000                 0.000           304.000   \n",
       "431                 33.000                 0.000           304.000   \n",
       "432                 25.000                 1.000           302.000   \n",
       "433                 31.000                 7.000           302.000   \n",
       "434                  4.000                 0.000           301.000   \n",
       "\n",
       "     page_views_total_log                domain                         mash  \n",
       "430                 8.253  dashboard.hawaii.gov                           []  \n",
       "431                 8.253  dashboard.hawaii.gov                           []  \n",
       "432                 8.243  dashboard.hawaii.gov                  [education]  \n",
       "433                 8.243  dashboard.hawaii.gov  [environmental, protection]  \n",
       "434                 8.238  dashboard.hawaii.gov                [class, size]  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mash_df[430:435]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning\n",
    "- Drop type \"filter\", which essentially counts datasets twice\n",
    "- Drop any blank \"mash\" value - can't analyze data we don't have\n",
    "- Get counts of the data we deleted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "33452"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#before drop\n",
    "len(mash_df.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filtered Views Dropped: 6732\n"
     ]
    }
   ],
   "source": [
    "cleaning_df = mash_df.copy()\n",
    "\n",
    "#drop filtered views\n",
    "cleaning_df = cleaning_df[cleaning_df.type != 'filter']\n",
    "\n",
    "print(\"Filtered Views Dropped: {}\".format(len(mash_df.index) - len(cleaning_df.index)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "official     25615\n",
       "community     1105\n",
       "Name: provenance, dtype: int64"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleaning_df.provenance.value_counts() #check what's here; we want only official"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleaning_df.provenance.isnull().sum() #no NaNs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cleaning_df = cleaning_df[cleaning_df.provenance == 'official'] #make sure we have only gov datasets here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25615"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(cleaning_df.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                                                       []\n",
       "1                                                       []\n",
       "2                                                       []\n",
       "3                                                       []\n",
       "4                                                       []\n",
       "5                                                       []\n",
       "6                                                       []\n",
       "8                                                       []\n",
       "9                                                       []\n",
       "10                                                      []\n",
       "11                                                      []\n",
       "12                                                      []\n",
       "13               [point, in, time, homelessness, ss, dchs]\n",
       "14                                               [housing]\n",
       "15                                        [transportation]\n",
       "16                                                      []\n",
       "17                                                      []\n",
       "18                                                      []\n",
       "19                                          [delete, dchs]\n",
       "20                                                      []\n",
       "21                                               [housing]\n",
       "22                                [fair, housing, housing]\n",
       "23       [well, rent, relief, well-being, and, safety, ...\n",
       "24                                                      []\n",
       "25       [resident, survey, city, strategic, plan, fina...\n",
       "26                             [assisted, rental, housing]\n",
       "27       [education, level, increase, employment, workf...\n",
       "28                                                      []\n",
       "29       [resident, survey, city, strategic, plan, fina...\n",
       "30                                                      []\n",
       "                               ...                        \n",
       "33326                     [public, safety, public, safety]\n",
       "33328                     [public, safety, public, safety]\n",
       "33330                                                   []\n",
       "33335               [finance, economy, budget, &, finance]\n",
       "33336                     [public, safety, public, safety]\n",
       "33344                                                   []\n",
       "33347    [shelter, adopt, animal, adoption, dog, shelte...\n",
       "33359    [dogs, loose, dog, animal, animals, animal, fi...\n",
       "33369                                                   []\n",
       "33376                                   [finance, economy]\n",
       "33381                                     [public, safety]\n",
       "33389                     [public, safety, public, safety]\n",
       "33400                     [public, safety, public, safety]\n",
       "33405                           [geography, &, boundaries]\n",
       "33408                                                   []\n",
       "33411        [dpd, police, use, of, force, public, safety]\n",
       "33412                                                   []\n",
       "33414    [adopt, shelter, animal, shelter, dog, animal,...\n",
       "33418    [dpd, rms, police, incident, shooting, shootin...\n",
       "33438                                     [public, safety]\n",
       "33442    [environment, public, safety, social, services...\n",
       "33443                                                   []\n",
       "33444                                                   []\n",
       "33445                                                   []\n",
       "33446    [local, law, 63, procurement, plan, fy, 2014, ...\n",
       "33447    [calendar, year, 2013, local, law, 11, complia...\n",
       "33448                  [quarterly, reports, 2015, finance]\n",
       "33449                           [quarterly, reports, 2014]\n",
       "33450    [meeting, agendas, &, performance, 2014, polic...\n",
       "33451                  [quarterly, reports, 2012, finance]\n",
       "Name: mash, Length: 25615, dtype: object"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleaning_df.mash"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "      <th>attribution</th>\n",
       "      <th>columns_field_name</th>\n",
       "      <th>columns_name</th>\n",
       "      <th>type</th>\n",
       "      <th>categories</th>\n",
       "      <th>domain_category</th>\n",
       "      <th>domain_tags</th>\n",
       "      <th>provenance</th>\n",
       "      <th>download_count</th>\n",
       "      <th>page_views_last_month</th>\n",
       "      <th>page_views_last_week</th>\n",
       "      <th>page_views_total</th>\n",
       "      <th>page_views_total_log</th>\n",
       "      <th>domain</th>\n",
       "      <th>mash</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Bonds Project Dataset (Official)</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>[location, icon, project_image, category_id, l...</td>\n",
       "      <td>[Automated Geocoding, Icon, Project Image, Cat...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>15.000</td>\n",
       "      <td>34.000</td>\n",
       "      <td>8.000</td>\n",
       "      <td>464.000</td>\n",
       "      <td>8.861</td>\n",
       "      <td>2014bonds.cityofws.org</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Bonds Application - Reference Table</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>[icon, name, id]</td>\n",
       "      <td>[Icon, name, id]</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>7.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>66.000</td>\n",
       "      <td>6.066</td>\n",
       "      <td>2014bonds.cityofws.org</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Wards 2011.shp</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>map</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>7.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>23.000</td>\n",
       "      <td>4.585</td>\n",
       "      <td>2014bonds.cityofws.org</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Project Dataset (Staging)</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>[city, location_id, project_id, project_name, ...</td>\n",
       "      <td>[City, Location Id, Project Id, Project Name, ...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>3.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>15.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>2014bonds.cityofws.org</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Capital Project Dataset</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>[document_10, document_9, document_8, document...</td>\n",
       "      <td>[Document 10, Document 9, Document 8, Document...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>25.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>393.000</td>\n",
       "      <td>8.622</td>\n",
       "      <td>cip.cityofnovi.org</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  name description attribution  \\\n",
       "0     Bonds Project Dataset (Official)                    None   \n",
       "1  Bonds Application - Reference Table                    None   \n",
       "2                       Wards 2011.shp                    None   \n",
       "3            Project Dataset (Staging)                    None   \n",
       "4              Capital Project Dataset                    None   \n",
       "\n",
       "                                  columns_field_name  \\\n",
       "0  [location, icon, project_image, category_id, l...   \n",
       "1                                   [icon, name, id]   \n",
       "2                                                 []   \n",
       "3  [city, location_id, project_id, project_name, ...   \n",
       "4  [document_10, document_9, document_8, document...   \n",
       "\n",
       "                                        columns_name     type categories  \\\n",
       "0  [Automated Geocoding, Icon, Project Image, Cat...  dataset         []   \n",
       "1                                   [Icon, name, id]  dataset         []   \n",
       "2                                                 []      map         []   \n",
       "3  [City, Location Id, Project Id, Project Name, ...  dataset         []   \n",
       "4  [Document 10, Document 9, Document 8, Document...  dataset         []   \n",
       "\n",
       "  domain_category domain_tags provenance  download_count  \\\n",
       "0                          []   official          15.000   \n",
       "1                          []   official           7.000   \n",
       "2                          []   official           7.000   \n",
       "3                          []   official           3.000   \n",
       "4                          []   official          25.000   \n",
       "\n",
       "   page_views_last_month  page_views_last_week  page_views_total  \\\n",
       "0                 34.000                 8.000           464.000   \n",
       "1                  1.000                 0.000            66.000   \n",
       "2                  0.000                 0.000            23.000   \n",
       "3                  0.000                 0.000            15.000   \n",
       "4                  4.000                 0.000           393.000   \n",
       "\n",
       "   page_views_total_log                  domain mash  \n",
       "0                 8.861  2014bonds.cityofws.org   []  \n",
       "1                 6.066  2014bonds.cityofws.org   []  \n",
       "2                 4.585  2014bonds.cityofws.org   []  \n",
       "3                 4.000  2014bonds.cityofws.org   []  \n",
       "4                 8.622      cip.cityofnovi.org   []  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleaning_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove all blank lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = cleaning_df.copy()\n",
    "df = df[df.astype(str).mash != \"['']\"] #drops all blank lists\n",
    "df = df.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Records in big_mash df: 21793\n"
     ]
    }
   ],
   "source": [
    "big_mash = df.copy()\n",
    "print(\"Number of Records in big_mash df: {}\".format(len(big_mash.index)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Clean up description string and get it into a tokenized list that we can easily add to the existing mash list of tokens: **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "big_mash.description = big_mash.description.str.replace(',', ' ') #replace natural commas with white space\n",
    "big_mash.description = big_mash.description.str.replace('-', ' ') \n",
    "big_mash.description = big_mash.description.str.replace('.', ' ')\n",
    "big_mash.description = big_mash.description.str.replace('&', ' ') \n",
    "big_mash.description = big_mash.description.str.replace(':', ' ')\n",
    "big_mash.description = big_mash.description.str.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "big_mash['big_mash'] = big_mash.mash + big_mash.description "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Now we must reset our index or our later topic percentage comparisons won't match! **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "      <th>attribution</th>\n",
       "      <th>columns_field_name</th>\n",
       "      <th>columns_name</th>\n",
       "      <th>type</th>\n",
       "      <th>categories</th>\n",
       "      <th>domain_category</th>\n",
       "      <th>domain_tags</th>\n",
       "      <th>provenance</th>\n",
       "      <th>download_count</th>\n",
       "      <th>page_views_last_month</th>\n",
       "      <th>page_views_last_week</th>\n",
       "      <th>page_views_total</th>\n",
       "      <th>page_views_total_log</th>\n",
       "      <th>domain</th>\n",
       "      <th>mash</th>\n",
       "      <th>big_mash</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Homelessness PIT Transitional Age Youth</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[location_on_the_night_of_the_count_total_pers...</td>\n",
       "      <td>[Location on the night of the count, Total Per...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[point in time, homelessness, ss, dchs]</td>\n",
       "      <td>official</td>\n",
       "      <td>4.000</td>\n",
       "      <td>46.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>319.000</td>\n",
       "      <td>8.322</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[point, in, time, homelessness, ss, dchs]</td>\n",
       "      <td>[point, in, time, homelessness, ss, dchs]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fair Housing Complaints</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[violations, percent_found_to_be_compliant, si...</td>\n",
       "      <td>[Number of complaints, Percent of sites found ...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[housing]</td>\n",
       "      <td>official</td>\n",
       "      <td>21.000</td>\n",
       "      <td>31.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>278.000</td>\n",
       "      <td>8.124</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[housing]</td>\n",
       "      <td>[housing]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Parking Complaints Bar Chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[percent_of_valid_parking_meter_problem_servic...</td>\n",
       "      <td>[Percent of valid parking meter problem servic...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[transportation]</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>17.000</td>\n",
       "      <td>17.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>274.000</td>\n",
       "      <td>8.103</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[transportation]</td>\n",
       "      <td>[transportation]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NVMHI Admissions</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[lipos_admissions_per_100k, nvmhi_admission_pe...</td>\n",
       "      <td>[LIPOS Admissions per 100K, NVMHI Admission pe...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[delete, dchs]</td>\n",
       "      <td>official</td>\n",
       "      <td>5.000</td>\n",
       "      <td>38.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>263.000</td>\n",
       "      <td>8.044</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[delete, dchs]</td>\n",
       "      <td>[delete, dchs]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Property Owners Trainined</td>\n",
       "      <td>[Office, of, Housing, Data]</td>\n",
       "      <td>None</td>\n",
       "      <td>[number_of_property_owners_trained, percent_of...</td>\n",
       "      <td>[Number of property owners trained, Percent of...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[housing]</td>\n",
       "      <td>official</td>\n",
       "      <td>11.000</td>\n",
       "      <td>30.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>246.000</td>\n",
       "      <td>7.948</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[housing]</td>\n",
       "      <td>[housing, Office, of, Housing, Data]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      name                  description  \\\n",
       "0  Homelessness PIT Transitional Age Youth                           []   \n",
       "1                  Fair Housing Complaints                           []   \n",
       "2             Parking Complaints Bar Chart                           []   \n",
       "3                         NVMHI Admissions                           []   \n",
       "4                Property Owners Trainined  [Office, of, Housing, Data]   \n",
       "\n",
       "  attribution                                 columns_field_name  \\\n",
       "0        None  [location_on_the_night_of_the_count_total_pers...   \n",
       "1        None  [violations, percent_found_to_be_compliant, si...   \n",
       "2        None  [percent_of_valid_parking_meter_problem_servic...   \n",
       "3        None  [lipos_admissions_per_100k, nvmhi_admission_pe...   \n",
       "4        None  [number_of_property_owners_trained, percent_of...   \n",
       "\n",
       "                                        columns_name   type        categories  \\\n",
       "0  [Location on the night of the count, Total Per...  chart                []   \n",
       "1  [Number of complaints, Percent of sites found ...  chart                []   \n",
       "2  [Percent of valid parking meter problem servic...  chart  [transportation]   \n",
       "3  [LIPOS Admissions per 100K, NVMHI Admission pe...  chart                []   \n",
       "4  [Number of property owners trained, Percent of...  chart                []   \n",
       "\n",
       "  domain_category                              domain_tags provenance  \\\n",
       "0                  [point in time, homelessness, ss, dchs]   official   \n",
       "1                                                [housing]   official   \n",
       "2                                                       []   official   \n",
       "3                                           [delete, dchs]   official   \n",
       "4                                                [housing]   official   \n",
       "\n",
       "   download_count  page_views_last_month  page_views_last_week  \\\n",
       "0           4.000                 46.000                 5.000   \n",
       "1          21.000                 31.000                 1.000   \n",
       "2          17.000                 17.000                 4.000   \n",
       "3           5.000                 38.000                 5.000   \n",
       "4          11.000                 30.000                 4.000   \n",
       "\n",
       "   page_views_total  page_views_total_log                      domain  \\\n",
       "0           319.000                 8.322  dashboard.alexandriava.gov   \n",
       "1           278.000                 8.124  dashboard.alexandriava.gov   \n",
       "2           274.000                 8.103  dashboard.alexandriava.gov   \n",
       "3           263.000                 8.044  dashboard.alexandriava.gov   \n",
       "4           246.000                 7.948  dashboard.alexandriava.gov   \n",
       "\n",
       "                                        mash  \\\n",
       "0  [point, in, time, homelessness, ss, dchs]   \n",
       "1                                  [housing]   \n",
       "2                           [transportation]   \n",
       "3                             [delete, dchs]   \n",
       "4                                  [housing]   \n",
       "\n",
       "                                    big_mash  \n",
       "0  [point, in, time, homelessness, ss, dchs]  \n",
       "1                                  [housing]  \n",
       "2                           [transportation]  \n",
       "3                             [delete, dchs]  \n",
       "4       [housing, Office, of, Housing, Data]  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "big_mash = big_mash.reset_index(drop=True)\n",
    "big_mash.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['youth',\n",
       " 'work',\n",
       " 'experience',\n",
       " 'work',\n",
       " 'experience',\n",
       " 'workforce',\n",
       " 'development',\n",
       " 'center',\n",
       " 'dchs',\n",
       " 'politics',\n",
       " 'Work',\n",
       " 'Force',\n",
       " 'Development',\n",
       " 'Center',\n",
       " 'Youth',\n",
       " 'Participants']"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "big_mash.big_mash[12] #this looks great; do have to remember to make it all lowercase"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA on full dataset (minus filtered views)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Missing domain_category tags: 0\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of Missing domain_category tags: {}\".format(big_mash.domain_category.isnull().sum()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Empty Mash Rows Dropped: 3822\n"
     ]
    }
   ],
   "source": [
    "print(\"Empty Mash Rows Dropped: {}\".format(len(cleaning_df.index) - len(big_mash.index)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clean DataFrame Length: 21793\n"
     ]
    }
   ],
   "source": [
    "print(\"Clean DataFrame Length: {}\".format(len(big_mash.index)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def median_mash_len(df):\n",
    "    lengths = []\n",
    "    for ls in df.big_mash:\n",
    "        lengths.append(len(ls))\n",
    "    return np.median(lengths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Median Mash Length: 23.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Median Mash Length: {}\".format(median_mash_len(big_mash)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NTLK Topic Analysis to find latent topics in \"mash\" categories\n",
    "- Goal: use topic word \"umbrellas\" to count views/downloads by umbrella.\n",
    "- Drawbacks: LDA assumes a document has multiple topics. This may be true of our mash - e.g. government and finance - but sometimes it might be just a single topic being arbitrarily split.\n",
    "\n",
    "Resources: \n",
    "- https://rstudio-pubs-static.s3.amazonaws.com/79360_850b2a69980c4488b1db95987a24867a.html\n",
    "- https://algobeans.com/2015/06/21/laymans-explanation-of-topic-modeling-with-lda-2/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit an LDA Model\n",
    "- This is very computationally intensive.\n",
    "- We pick the number of topics that we want the algorithm to find. This is a very important parameter.\n",
    "- LDA Model fitting is an iterative process. The algorithm starts out by assigning every word to a temporary topic. Then, for *every* word, it updates the topics by calculating:\n",
    "    - How prevalent is that word across topics? Topics with a high prevalance of the word in question get a higher weight for that word's assignment.\n",
    "     - How prevalent are topics within a document? If one topic within a document is more prevalent, it gets a higher weight.\n",
    "     - Based on these two criteria, LDA then updates a word's topic and document assignment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Therefore, the more iterations of LDA you can run, the more accurate it gets **\n",
    "- Unfortunately, 40 passes takes about an hour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mash = big_mash.big_mash\n",
    "\n",
    "stop_words = stopwords.words('english') #list of stop words\n",
    "\n",
    "stop = list(stop_words) #copy ls since we're gonna mess with it\n",
    "stop.append('&') #this ampersand is giving us fits\n",
    "stop.extend(['', 'data', 'dataset', 'datasets', '//data', 'http', \n",
    "             'https', 'html', 'www', '//www', \"=\", \"gov\", \"gov)\", \"(gov\"]) #add words that don't specify given subject\n",
    "stop.extend(\"•\") #special char that showed up in earlier model iterations\n",
    "stop.extend(\"–\") #ditto as above\n",
    "stop.extend(np.arange(101).astype(str)) #remove common numbers 0-100\n",
    "stop.extend(np.arange(1980, 2025).astype(str)) #remove common years 1979-2024\n",
    "\n",
    "texts = [] #blank list to append to\n",
    "\n",
    "for ls in mash:\n",
    "    lowers = [word.lower() for word in ls]\n",
    "    stopped_tokens = [word for word in lowers if not word in stop]\n",
    "    texts.append(stopped_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dictionary = corpora.Dictionary(texts)\n",
    "corpus = [dictionary.doc2bow(text) for text in texts]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# !!! ~One hour run time warning !!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 50 Topics & 45 passes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lda_50_45 = gensim.models.ldamodel.LdaModel(corpus, num_topics=50, id2word = dictionary, \n",
    "                                         passes = 45, random_state=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've (hopefully) created groups of words that tend to co-occur -- \"topics\".\n",
    "\n",
    "One can assume the decimals returned for each word is the distribution of that word over the topic itself. So obviously, larger decimals are more important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  [('new', 0.10316757798764456),\n",
       "   ('york', 0.071569777494110365),\n",
       "   ('state', 0.053206212403621514),\n",
       "   ('prevention', 0.032032819059224719),\n",
       "   ('environment', 0.0259229189793997),\n",
       "   ('waste', 0.023565802574361239),\n",
       "   ('section', 0.019080124537761374),\n",
       "   ('taxes', 0.018383886834403274),\n",
       "   ('environmental', 0.011596182265864707),\n",
       "   ('site', 0.011555240674199723)]),\n",
       " (1,\n",
       "  [('food', 0.054992544288208108),\n",
       "   ('cost', 0.042269310008768754),\n",
       "   ('facility', 0.042127112534745534),\n",
       "   ('inspection', 0.03847589835551201),\n",
       "   ('inspections', 0.032707388728011114),\n",
       "   ('health', 0.028980025847417542),\n",
       "   ('release', 0.027962755993273639),\n",
       "   ('county', 0.027773584270816991),\n",
       "   ('facilities', 0.026376692147202847),\n",
       "   ('weekly', 0.021455540181697839)]),\n",
       " (2,\n",
       "  [('recreation', 0.087322849450364337),\n",
       "   ('fire', 0.066376996472196026),\n",
       "   ('emergency', 0.050154897104409624),\n",
       "   ('public', 0.049353844405411552),\n",
       "   ('parks', 0.043831285217989721),\n",
       "   ('safety', 0.04053988618797006),\n",
       "   ('department', 0.028218975205801335),\n",
       "   ('calls', 0.023501036326813732),\n",
       "   ('response', 0.0221884321283717),\n",
       "   ('call', 0.021742523512329054)]),\n",
       " (3,\n",
       "  [('education', 0.16311929093846347),\n",
       "   ('school', 0.085287846944022),\n",
       "   ('students', 0.03461332112734046),\n",
       "   ('schools', 0.033209240133260133),\n",
       "   ('high', 0.030255853189387116),\n",
       "   ('year', 0.027657043942472811),\n",
       "   ('district', 0.017314479264753192),\n",
       "   ('bus', 0.017249366502765814),\n",
       "   ('student', 0.016889376815752342),\n",
       "   ('bike', 0.016390407690068231)]),\n",
       " (4,\n",
       "  [('michigan', 0.1009563557683765),\n",
       "   ('state', 0.062854323061637948),\n",
       "   ('open', 0.03231549537446804),\n",
       "   ('concerning', 0.029447966298628805),\n",
       "   ('health', 0.029351912624166622),\n",
       "   ('official', 0.029025867312879396),\n",
       "   ('created', 0.027940432913339426),\n",
       "   ('information', 0.027509005044946543),\n",
       "   ('user', 0.025920267009938817),\n",
       "   ('(openmichigan@michigan', 0.022875942826386286)]),\n",
       " (5,\n",
       "  [('us', 0.064604545229042895),\n",
       "   ('unit', 0.032682265469764667),\n",
       "   ('standard', 0.031684353055512648),\n",
       "   ('region', 0.031104753903138851),\n",
       "   ('two', 0.026407764750120181),\n",
       "   ('info', 0.019327255631807025),\n",
       "   ('measurement', 0.01818072949232101),\n",
       "   ('master', 0.017844497353860518),\n",
       "   ('field', 0.015874143767345677),\n",
       "   ('meet', 0.015687736042869105)]),\n",
       " (6,\n",
       "  [('endorsed', 0.058311256534216413),\n",
       "   ('crest', 0.04877399769662473),\n",
       "   ('metadata', 0.037883700992413026),\n",
       "   ('seattle', 0.03296610395160942),\n",
       "   ('vermont', 0.030891276225383214),\n",
       "   ('el', 0.025944545590156212),\n",
       "   (\"governor's\", 0.022087760513498693),\n",
       "   ('control', 0.017365929722780751),\n",
       "   ('documentation', 0.013961357672725928),\n",
       "   ('national', 0.012185934246447578)]),\n",
       " (7,\n",
       "  [('financial', 0.11865652962593821),\n",
       "   ('state', 0.095392874384892357),\n",
       "   ('government', 0.059800851382918796),\n",
       "   ('expenditures', 0.051899332198620175),\n",
       "   ('agencies', 0.037599754405829897),\n",
       "   ('reports', 0.035915268340985303),\n",
       "   ('june', 0.03426303126395918),\n",
       "   ('july', 0.033908987245403061),\n",
       "   ('agency', 0.03042040137442201),\n",
       "   ('provides', 0.029610785401747747)]),\n",
       " (8,\n",
       "  [('historic', 0.045779047689639453),\n",
       "   ('governance', 0.032182194853617306),\n",
       "   ('api', 0.029906575390854079),\n",
       "   ('national', 0.028470957923760119),\n",
       "   ('cause', 0.022147450725259671),\n",
       "   ('sample', 0.020934747142705608),\n",
       "   ('community', 0.019528669579044096),\n",
       "   ('centers', 0.01874436719573843),\n",
       "   ('agriculture', 0.017839816543946375),\n",
       "   ('behavioral', 0.016504675797445436)]),\n",
       " (9,\n",
       "  [('government', 0.15065762671999411),\n",
       "   ('city', 0.036129224846505451),\n",
       "   ('authority', 0.03180576554707782),\n",
       "   ('county', 0.031406244072705827),\n",
       "   ('cities', 0.021963104843726187),\n",
       "   ('includes', 0.021609727989906815),\n",
       "   ('municipal', 0.02025919699951587),\n",
       "   ('fuel', 0.020128452014600832),\n",
       "   ('reduction', 0.018586710368985185),\n",
       "   ('top', 0.017207857236242276)]),\n",
       " (10,\n",
       "  [('performance', 0.12674666693229911),\n",
       "   ('indicators', 0.080891295973778282),\n",
       "   ('city', 0.057020585869640067),\n",
       "   ('government', 0.053424511056798282),\n",
       "   ('administration', 0.05276813411162079),\n",
       "   ('plan', 0.049337230856492671),\n",
       "   ('resultsnola', 0.046264962382329092),\n",
       "   ('management', 0.043213214411401031),\n",
       "   ('includes', 0.041729656246536989),\n",
       "   (\"city's\", 0.032118529409401415)]),\n",
       " (11,\n",
       "  [('communities', 0.03819205470995915),\n",
       "   ('family', 0.025140387063851091),\n",
       "   ('disclosure', 0.018428401516323939),\n",
       "   ('properties', 0.017967119003472511),\n",
       "   ('manager', 0.016117197721432795),\n",
       "   ('certified', 0.014842013978236535),\n",
       "   ('women', 0.014091568024169433),\n",
       "   ('c', 0.013996473876942826),\n",
       "   ('opportunity', 0.012734807713539915),\n",
       "   ('star', 0.012461094556004158)]),\n",
       " (12,\n",
       "  [('police', 0.080209872411013586),\n",
       "   ('information', 0.036953960068332826),\n",
       "   ('department', 0.034734395610500658),\n",
       "   ('public', 0.034009067224003112),\n",
       "   ('safety', 0.033507286077474191),\n",
       "   ('use', 0.020753464675038882),\n",
       "   ('injury', 0.017928530773194541),\n",
       "   ('officer', 0.014934215492799763),\n",
       "   ('incident', 0.014166382659714781),\n",
       "   ('gender', 0.013273875613552684)]),\n",
       " (13,\n",
       "  [('income', 0.056671119165198715),\n",
       "   ('average', 0.050865625847585208),\n",
       "   ('statewide', 0.047990441647605422),\n",
       "   ('resources', 0.042632316156013289),\n",
       "   ('per', 0.027081483391081365),\n",
       "   ('median', 0.021641531043170237),\n",
       "   ('access', 0.020831824053170734),\n",
       "   ('natural', 0.02040855601931936),\n",
       "   ('salt', 0.018911766549354427),\n",
       "   ('year', 0.018857836358946233)]),\n",
       " (14,\n",
       "  [('county', 0.10599215065122963),\n",
       "   ('health', 0.055757125536669352),\n",
       "   ('rates', 0.050122063968434299),\n",
       "   ('public', 0.032700452104773653),\n",
       "   ('population', 0.031883126892740463),\n",
       "   ('000', 0.029052877913418425),\n",
       "   ('per', 0.027345663586926718),\n",
       "   ('diego', 0.024120564853999545),\n",
       "   ('harbison', 0.02247568512054578),\n",
       "   ('cajon', 0.020763886441815822)]),\n",
       " (15,\n",
       "  [('health', 0.20778016257399401),\n",
       "   ('county', 0.021487279429586802),\n",
       "   ('\"about\"', 0.020609848138436242),\n",
       "   ('death', 0.017907707044705918),\n",
       "   ('hospital', 0.017669929770858368),\n",
       "   ('rates', 0.015896005751459972),\n",
       "   ('details', 0.015604922943462005),\n",
       "   ('chart', 0.0135226840515623),\n",
       "   ('per', 0.013492293024783874),\n",
       "   ('000', 0.011847060985784781)]),\n",
       " (16,\n",
       "  [('economy', 0.065331816468157958),\n",
       "   ('employment', 0.047391861196077731),\n",
       "   ('fund', 0.041342965535565469),\n",
       "   ('labor', 0.03036523332519208),\n",
       "   ('unemployment', 0.024624116246304624),\n",
       "   ('workforce', 0.019363855657420843),\n",
       "   ('annual', 0.017346360040827031),\n",
       "   ('investment', 0.014743832646056956),\n",
       "   ('insurance', 0.013528875018372903),\n",
       "   ('economic', 0.013279890926263353)]),\n",
       " (17,\n",
       "  [('environment', 0.040496525353180754),\n",
       "   ('animal', 0.0256820371802517),\n",
       "   ('licensing', 0.025660493735719599),\n",
       "   ('priority', 0.021471708874639006),\n",
       "   ('agenda', 0.020606449350560385),\n",
       "   ('costs', 0.018612550200760736),\n",
       "   ('fair', 0.018435392579043176),\n",
       "   ('year', 0.018071670497530761),\n",
       "   ('reduce', 0.017583266285989566),\n",
       "   ('ecology', 0.016375604954623358)]),\n",
       " (18,\n",
       "  [('ny', 0.082710242768851944),\n",
       "   ('indicator', 0.066378098134037278),\n",
       "   ('additional', 0.04822816101758802),\n",
       "   ('metrics', 0.036720088157035931),\n",
       "   ('contains', 0.022503529323130902),\n",
       "   ('vehicle', 0.02157955168294055),\n",
       "   ('tools', 0.021241226946291054),\n",
       "   ('motor', 0.020197420446146305),\n",
       "   ('column', 0.019688194254133949),\n",
       "   ('crash', 0.017342454899806826)]),\n",
       " (19,\n",
       "  [('based', 0.024848251299990461),\n",
       "   ('days', 0.024093690427869231),\n",
       "   ('values', 0.022357698869000606),\n",
       "   ('hours', 0.020456170951929947),\n",
       "   ('wa', 0.019882045568109553),\n",
       "   ('benefits', 0.019138060227712812),\n",
       "   ('metric', 0.017221451077064198),\n",
       "   ('within', 0.017186429615022595),\n",
       "   ('e', 0.016664706451323414),\n",
       "   ('physical', 0.015675880507999274)]),\n",
       " (20,\n",
       "  [('statistics', 0.080154965751574409),\n",
       "   ('percent', 0.047464400167813871),\n",
       "   ('township', 0.036045079344124945),\n",
       "   ('codes', 0.032416496505884941),\n",
       "   ('mortality', 0.03202389165537832),\n",
       "   ('code', 0.028578620531188717),\n",
       "   ('zip', 0.027721758988494754),\n",
       "   ('stops', 0.022876949527224508),\n",
       "   ('miles', 0.017245330107219313),\n",
       "   ('department', 0.014376282922058198)]),\n",
       " (21,\n",
       "  [('measures', 0.050264274407219312),\n",
       "   ('time', 0.031581178540664896),\n",
       "   ('number', 0.023000568319138701),\n",
       "   ('period', 0.014439234451407348),\n",
       "   ('employees', 0.014254795294823656),\n",
       "   ('employee', 0.01379812571940634),\n",
       "   ('check', 0.013390608053717078),\n",
       "   ('update', 0.012917781903205196),\n",
       "   ('set', 0.012398736166264785),\n",
       "   ('every', 0.011560337270440875)]),\n",
       " (22,\n",
       "  [('demographics', 0.10842340328390644),\n",
       "   ('census', 0.084025180348806544),\n",
       "   ('population', 0.075448282388353119),\n",
       "   ('estimates', 0.042775190267543908),\n",
       "   ('counties', 0.038684291973256325),\n",
       "   ('age', 0.038492201012047624),\n",
       "   ('bureau', 0.025857466008132685),\n",
       "   ('source', 0.02334511191186936),\n",
       "   ('county', 0.016994669037237814),\n",
       "   ('race', 0.016497297675593433)]),\n",
       " (23,\n",
       "  [('report', 0.18500215938737025),\n",
       "   ('key', 0.033302296868038012),\n",
       "   ('new', 0.031665092201614953),\n",
       "   ('orleans', 0.021482096917624909),\n",
       "   ('annual', 0.021034631560886214),\n",
       "   ('file', 0.018857775344335394),\n",
       "   ('activity', 0.017456635273900289),\n",
       "   ('web', 0.017064634154530137),\n",
       "   ('clear', 0.016644744287487686),\n",
       "   ('city', 0.015454089392417106)]),\n",
       " (24,\n",
       "  [('business', 0.10111420655314741),\n",
       "   ('license', 0.037884199812961206),\n",
       "   ('jobs', 0.030364225025840633),\n",
       "   ('job', 0.026662232524529639),\n",
       "   ('list', 0.026019677046940602),\n",
       "   ('active', 0.024978838526874252),\n",
       "   ('economy', 0.024783169235003403),\n",
       "   ('businesses', 0.022452543619367438),\n",
       "   ('city', 0.021194780888722089),\n",
       "   ('consumer', 0.020842406539920092)]),\n",
       " (25,\n",
       "  [('tax', 0.12429906590319924),\n",
       "   ('property', 0.072659745851672189),\n",
       "   ('cook', 0.055653561779072561),\n",
       "   ('sales', 0.041653399695410404),\n",
       "   ('county', 0.032894912554528255),\n",
       "   ('finance', 0.024123471401332721),\n",
       "   ('credit', 0.021795845901248716),\n",
       "   ('local', 0.019582284303989297),\n",
       "   ('gis/maps', 0.01768783140541837),\n",
       "   ('government', 0.017429257522269394)]),\n",
       " (26,\n",
       "  [('disease', 0.068077868214435691),\n",
       "   ('rate', 0.05944115528597789),\n",
       "   ('chronic', 0.039340487105275955),\n",
       "   ('risk', 0.034634503717806027),\n",
       "   ('calculated', 0.031857305404497764),\n",
       "   ('cancer', 0.028802282273485309),\n",
       "   ('due', 0.021384201186404692),\n",
       "   ('events', 0.019817338776413656),\n",
       "   ('software', 0.017674417717273808),\n",
       "   ('asthma', 0.015210941319345844)]),\n",
       " (27,\n",
       "  [('water', 0.084393734594413683),\n",
       "   ('dfps', 0.064235565768264946),\n",
       "   ('quality', 0.042030426409406428),\n",
       "   ('information', 0.033350025128925036),\n",
       "   ('provider', 0.028887620501586494),\n",
       "   ('visit', 0.026718955646236206),\n",
       "   ('environmental', 0.020898272248362184),\n",
       "   ('tx', 0.020679531614405246),\n",
       "   ('programs', 0.020511794910074977),\n",
       "   ('lead', 0.019566854976587097)]),\n",
       " (28,\n",
       "  [('children', 0.040775136098806251),\n",
       "   ('quality', 0.038216726213425749),\n",
       "   ('home', 0.037468131811124752),\n",
       "   ('maryland', 0.028655590002890207),\n",
       "   ('care', 0.026328381662908566),\n",
       "   ('life', 0.019870762967494498),\n",
       "   ('nursing', 0.01684856993265452),\n",
       "   ('homes', 0.016797019275272822),\n",
       "   ('arts', 0.016249420903489489),\n",
       "   ('health', 0.015073936687275158)]),\n",
       " (29,\n",
       "  [('map', 0.056568367203415917),\n",
       "   ('gis', 0.038189846941413891),\n",
       "   ('city', 0.036600176633187952),\n",
       "   ('boundaries', 0.03235138346097291),\n",
       "   ('districts', 0.022778376666423048),\n",
       "   ('parking', 0.021006537172764211),\n",
       "   ('district', 0.019094535532542627),\n",
       "   ('locations', 0.015517533310976326),\n",
       "   ('county', 0.014699186361133126),\n",
       "   ('zoning', 0.014572942825322037)]),\n",
       " (30,\n",
       "  [('jersey', 0.041112273952947788),\n",
       "   ('class', 0.02153522672107086),\n",
       "   ('land', 0.020861253090580456),\n",
       "   ('use', 0.020237240431203448),\n",
       "   ('inpatient', 0.020055361924225173),\n",
       "   ('address', 0.018773264849191368),\n",
       "   ('parcel', 0.018088599604757154),\n",
       "   ('infrastructure', 0.01742137497090741),\n",
       "   ('city', 0.016604092286366775),\n",
       "   ('austin', 0.014395949814092037)]),\n",
       " (31,\n",
       "  [('support', 0.025420459429903921),\n",
       "   ('state', 0.023255578138466355),\n",
       "   ('government', 0.02324529057005037),\n",
       "   ('survey', 0.021294920669962081),\n",
       "   ('citizens', 0.019880918142698684),\n",
       "   ('states', 0.017820097393126035),\n",
       "   ('areas', 0.017352733649766614),\n",
       "   ('service', 0.016821403326025178),\n",
       "   ('private', 0.01637387744646758),\n",
       "   ('inventory', 0.014872144536696741)]),\n",
       " (32,\n",
       "  [('public', 0.20955359976967117),\n",
       "   ('safety', 0.16595132200721041),\n",
       "   ('crime', 0.058139999358857199),\n",
       "   ('police', 0.031942349308622005),\n",
       "   ('reported', 0.014409578621096906),\n",
       "   ('crimes', 0.011296411789044327),\n",
       "   ('part', 0.01082262387918027),\n",
       "   ('criminal', 0.010562273758408692),\n",
       "   ('law', 0.010396860671166333),\n",
       "   ('violent', 0.010156403193244725)]),\n",
       " (33,\n",
       "  [('dashboard', 0.058136216084580528),\n",
       "   ('finance', 0.057826890967880892),\n",
       "   ('ethics', 0.044238328620496946),\n",
       "   ('form', 0.039076068454969311),\n",
       "   ('summary', 0.034342536329770648),\n",
       "   ('board', 0.026900548844217474),\n",
       "   ('campaign', 0.026749315929666763),\n",
       "   ('city', 0.024123677157651104),\n",
       "   ('management', 0.023609527677092672),\n",
       "   ('commission', 0.022954467179594727)]),\n",
       " (34,\n",
       "  [('san', 0.10648235523538448),\n",
       "   ('county', 0.065283398685495958),\n",
       "   ('mateo', 0.052098740322732658),\n",
       "   ('survey', 0.048542181335179575),\n",
       "   ('action', 0.024519703554556637),\n",
       "   ('climate', 0.024184201019546197),\n",
       "   ('results', 0.019766986388204469),\n",
       "   ('francisco', 0.017965456098035928),\n",
       "   ('developed', 0.014801146467849023),\n",
       "   ('see', 0.012795370295093806)]),\n",
       " (35,\n",
       "  [('colorado', 0.041913513470847742),\n",
       "   ('demographic', 0.030440940555409669),\n",
       "   ('department', 0.030375848543688206),\n",
       "   ('cdc', 0.029048202923849992),\n",
       "   ('college', 0.022390859399501101),\n",
       "   ('corrections', 0.0183852127390009),\n",
       "   ('american', 0.017953948632142499),\n",
       "   ('technology', 0.017343057144835989),\n",
       "   ('shows', 0.016040151629317215),\n",
       "   ('institutional', 0.014718484496470317)]),\n",
       " (36,\n",
       "  [('goal', 0.047757562286705531),\n",
       "   ('healthy', 0.043310977860052141),\n",
       "   ('safe', 0.040034916594301244),\n",
       "   ('retirement', 0.02754999091205584),\n",
       "   ('pension', 0.023215214171492995),\n",
       "   ('discharges', 0.021137860763265043),\n",
       "   ('percentage', 0.020252816727427615),\n",
       "   ('condition', 0.01820482675059347),\n",
       "   ('technical', 0.017090282608134198),\n",
       "   ('system', 0.016636451106579041)]),\n",
       " (37,\n",
       "  [('library', 0.038589016480956383),\n",
       "   ('guide', 0.038009474612647004),\n",
       "   ('air', 0.036193042521417307),\n",
       "   ('standards', 0.035672411081621801),\n",
       "   ('maps', 0.032157086628986406),\n",
       "   ('limitations', 0.023366263280603854),\n",
       "   ('poverty', 0.023227875393901823),\n",
       "   ('community', 0.014418348952081227),\n",
       "   ('important', 0.013777213147097846),\n",
       "   ('positions', 0.013562265261606055)]),\n",
       " (38,\n",
       "  [('information', 0.041168176418363983),\n",
       "   ('cases', 0.021610008511538423),\n",
       "   ('please', 0.019501348763342675),\n",
       "   ('may', 0.017994835922067041),\n",
       "   ('records', 0.015323268349852802),\n",
       "   ('location', 0.01507243888583157),\n",
       "   ('view', 0.013793315047680476),\n",
       "   ('click', 0.012562020889071745),\n",
       "   ('contains', 0.012313432112369998),\n",
       "   ('also', 0.010731460300829228)]),\n",
       " (39,\n",
       "  [('transportation', 0.223305018524142),\n",
       "   ('infrastructure', 0.071973213113117829),\n",
       "   ('traffic', 0.051731018740892025),\n",
       "   ('street', 0.029759447796036079),\n",
       "   ('transit', 0.024224930980839342),\n",
       "   ('streets', 0.020359417536286215),\n",
       "   ('network', 0.017022203827923195),\n",
       "   ('road', 0.014019932922684828),\n",
       "   ('city', 0.013114249619861787),\n",
       "   ('travel', 0.012831098241139953)]),\n",
       " (40,\n",
       "  [('finance', 0.12343853738991102),\n",
       "   ('budget', 0.079226007486914385),\n",
       "   ('year', 0.06154244980477918),\n",
       "   ('fiscal', 0.057438343819527525),\n",
       "   ('revenue', 0.030168585852365142),\n",
       "   ('account', 0.026632053831381307),\n",
       "   ('items', 0.024698939067817061),\n",
       "   ('accounts', 0.022840949570821147),\n",
       "   ('funds', 0.020058697387220017),\n",
       "   ('fy', 0.013713169867642287)]),\n",
       " (41,\n",
       "  [('code', 0.059576146155742866),\n",
       "   ('definition', 0.046329179138290211),\n",
       "   ('violations', 0.031618990615787657),\n",
       "   ('healthcare', 0.030528972633974145),\n",
       "   ('complaints', 0.025119692965920724),\n",
       "   ('unknown', 0.022227290778071587),\n",
       "   ('department', 0.019352800375606501),\n",
       "   ('violation', 0.019034668163001367),\n",
       "   ('observed', 0.018708740991499426),\n",
       "   ('potential', 0.014857102033686778)]),\n",
       " (42,\n",
       "  [('politics', 0.10785359353231141),\n",
       "   ('election', 0.040423302378762203),\n",
       "   ('government', 0.035636970997169004),\n",
       "   ('elections', 0.031374807284401146),\n",
       "   ('results', 0.028221414031932443),\n",
       "   ('county', 0.027570878538215462),\n",
       "   ('filter', 0.02633237016229744),\n",
       "   ('link', 0.02106635044829756),\n",
       "   ('page', 0.019826385822407425),\n",
       "   ('council', 0.01409228404186828)]),\n",
       " (43,\n",
       "  [('services', 0.20443150448981229),\n",
       "   ('social', 0.076817780894837345),\n",
       "   ('health', 0.061101122727494926),\n",
       "   ('care', 0.057867574619333756),\n",
       "   ('human', 0.045545366618153997),\n",
       "   ('agency', 0.026004692670449168),\n",
       "   ('state', 0.024052738881300557),\n",
       "   ('information', 0.023834754012025141),\n",
       "   ('child', 0.020054125283018084),\n",
       "   ('discharge', 0.019240020707699253)]),\n",
       " (44,\n",
       "  [('energy', 0.1117672180830399),\n",
       "   ('environment', 0.10716157511690279),\n",
       "   ('gas', 0.044125227905571551),\n",
       "   ('consumption', 0.025513602172885869),\n",
       "   ('electricity', 0.024274464800903454),\n",
       "   ('sustainable', 0.021002286507517648),\n",
       "   ('total', 0.020414292112832202),\n",
       "   ('greenhouse', 0.020285756996840529),\n",
       "   ('emissions', 0.017183810260189211),\n",
       "   ('infrastructure', 0.016633340942926563)]),\n",
       " (45,\n",
       "  [('community', 0.057622897492192605),\n",
       "   ('neighborhood', 0.046665296705475749),\n",
       "   ('book', 0.035343489294651968),\n",
       "   ('neighborhoods', 0.029835550450299755),\n",
       "   ('statistical', 0.028994561863055209),\n",
       "   ('real', 0.024324006204920964),\n",
       "   ('alone', 0.02124469657305824),\n",
       "   ('graphs', 0.02083217314457932),\n",
       "   ('highly', 0.015462921604095537),\n",
       "   ('infection', 0.014638441942477256)]),\n",
       " (46,\n",
       "  [('permits', 0.060110080574680175),\n",
       "   ('building', 0.051075348489021129),\n",
       "   ('permit', 0.048658671957110065),\n",
       "   ('tab', 0.046715386109904118),\n",
       "   ('medicaid', 0.025911714838122428),\n",
       "   ('plans', 0.024600096855178062),\n",
       "   ('managed', 0.024480351517300412),\n",
       "   ('construction', 0.018282871447904143),\n",
       "   ('issued', 0.017923269724343074),\n",
       "   ('plan', 0.015608145357567459)]),\n",
       " (47,\n",
       "  [('monthly', 0.071923395870210974),\n",
       "   ('finance', 0.067577285791939737),\n",
       "   ('service', 0.063607307677479019),\n",
       "   ('city', 0.061598381508498043),\n",
       "   ('requests', 0.029753810956411046),\n",
       "   ('topeka', 0.026411057755959977),\n",
       "   ('chart', 0.026209789741948597),\n",
       "   ('utah', 0.024530079194503406),\n",
       "   ('311', 0.022210053447613242),\n",
       "   ('used', 0.021882970953107542)]),\n",
       " (48,\n",
       "  [('payments', 0.051932170026863879),\n",
       "   ('state', 0.048669283498645667),\n",
       "   ('finance', 0.021868219788504738),\n",
       "   ('document', 0.021581796679442711),\n",
       "   ('contracts', 0.018981990098186512),\n",
       "   ('vendor', 0.016717070073027054),\n",
       "   ('contract', 0.016341512429004029),\n",
       "   ('quantities', 0.01488327339156594),\n",
       "   ('vendors', 0.014124802675758701),\n",
       "   ('modified', 0.013829679632168426)]),\n",
       " (49,\n",
       "  [('development', 0.16325374612445911),\n",
       "   ('housing', 0.13501404235166573),\n",
       "   ('economic', 0.0505651964679983),\n",
       "   ('projects', 0.031937906743941237),\n",
       "   ('project', 0.025706699083227834),\n",
       "   ('community', 0.022434301943472089),\n",
       "   ('infrastructure', 0.021768458637761726),\n",
       "   ('planning', 0.020183422978304468),\n",
       "   ('capital', 0.01790139061721599),\n",
       "   ('buildings', 0.014449572607823374)])]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_50_45.show_topics(num_topics=50, formatted=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lda_45_fifty = gensim.models.ldamodel.LdaModel(corpus, num_topics=45, id2word = dictionary, \n",
    "                                         passes = 50, random_state=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  [('health', 0.28889349052335084),\n",
       "   ('care', 0.038152752914044963),\n",
       "   ('county', 0.019615084197074441),\n",
       "   ('community', 0.019350158353068463),\n",
       "   ('home', 0.018423974405805475),\n",
       "   ('medicaid', 0.016124394901775481),\n",
       "   ('managed', 0.015703304770523105),\n",
       "   ('system', 0.012857080046039977),\n",
       "   ('state', 0.011904354997325322),\n",
       "   ('medical', 0.011775910916319483)]),\n",
       " (1,\n",
       "  [('chart', 0.069111335913462274),\n",
       "   ('county', 0.03741621452915532),\n",
       "   ('assistance', 0.034746374264368704),\n",
       "   ('historic', 0.032832750195864284),\n",
       "   ('income', 0.031630079455261646),\n",
       "   ('arts', 0.025084058341459166),\n",
       "   ('governance', 0.024586485403403292),\n",
       "   ('program', 0.023034810339547393),\n",
       "   ('finance', 0.017740679523431474),\n",
       "   ('insurance', 0.017411275747355712)]),\n",
       " (2,\n",
       "  [('census', 0.064906883684765576),\n",
       "   ('demographics', 0.037242956403767423),\n",
       "   ('community', 0.025265700083693006),\n",
       "   ('u', 0.015312570141907301),\n",
       "   ('bureau', 0.014669806766646965),\n",
       "   ('township', 0.014231272416557694),\n",
       "   ('neighborhood', 0.013800295421922192),\n",
       "   ('neighborhoods', 0.013623615775915627),\n",
       "   ('vermont', 0.011475650963802103),\n",
       "   ('survey', 0.01115289843501507)]),\n",
       " (3,\n",
       "  [('tab', 0.044873202101482927),\n",
       "   ('parking', 0.026636033014142201),\n",
       "   ('information', 0.024520168432947603),\n",
       "   ('additional', 0.018006107441283423),\n",
       "   ('check', 0.01655981951942747),\n",
       "   ('current', 0.014485149591957087),\n",
       "   ('city', 0.012980005385631398),\n",
       "   ('franklin', 0.012034727495733989),\n",
       "   ('complete', 0.011289804243761168),\n",
       "   ('initiatives', 0.010888972465781313)]),\n",
       " (4,\n",
       "  [('michigan', 0.090081344209961542),\n",
       "   ('recreation', 0.059007726579712955),\n",
       "   ('state', 0.04843949197694699),\n",
       "   ('parks', 0.035589032318943101),\n",
       "   ('open', 0.034909780248093047),\n",
       "   ('projects', 0.033306809494324255),\n",
       "   ('project', 0.028964972493622534),\n",
       "   ('official', 0.026540049293908741),\n",
       "   ('created', 0.023234346894895473),\n",
       "   ('park', 0.021912529387994727)]),\n",
       " (5,\n",
       "  [('public', 0.240688573774342),\n",
       "   ('safety', 0.18029586560481944),\n",
       "   ('police', 0.046072201258632831),\n",
       "   ('department', 0.019970026132568361),\n",
       "   ('calls', 0.015870016431420941),\n",
       "   ('incidents', 0.015343793127287748),\n",
       "   ('call', 0.012276843293079872),\n",
       "   ('works', 0.011842811732729054),\n",
       "   ('city', 0.011509274694162354),\n",
       "   ('incident', 0.010423763982869459)]),\n",
       " (6,\n",
       "  [('tax', 0.11575871012689749),\n",
       "   ('property', 0.084536666761607995),\n",
       "   ('sales', 0.038195453488971819),\n",
       "   ('taxes', 0.034677383653219783),\n",
       "   ('government', 0.025618563523409132),\n",
       "   ('value', 0.018774431651027712),\n",
       "   ('information', 0.016785397960819938),\n",
       "   ('sale', 0.015284046286165422),\n",
       "   ('properties', 0.014420850468959837),\n",
       "   ('investment', 0.014088629178492211)]),\n",
       " (7,\n",
       "  [('indicators', 0.10035081580580814),\n",
       "   ('administration', 0.065804228068173765),\n",
       "   ('adult', 0.032159841933492805),\n",
       "   ('employees', 0.029910512468092725),\n",
       "   ('city', 0.029507427431111882),\n",
       "   ('lead', 0.023402838229708421),\n",
       "   ('employee', 0.023012196523421524),\n",
       "   ('years', 0.021737591528888957),\n",
       "   ('state', 0.020493743562586861),\n",
       "   ('bike', 0.019397003489292491)]),\n",
       " (8,\n",
       "  [('health', 0.078631560469102177),\n",
       "   ('population', 0.041083636365164072),\n",
       "   ('disease', 0.038519976440391838),\n",
       "   ('000', 0.029258737927123343),\n",
       "   ('per', 0.028566813191114541),\n",
       "   ('estimates', 0.028145312159191988),\n",
       "   ('county', 0.02423354801830982),\n",
       "   ('chronic', 0.022188361259087217),\n",
       "   ('public', 0.018489480106327449),\n",
       "   ('diego', 0.017796592035991526)]),\n",
       " (9,\n",
       "  [('housing', 0.15022954648032388),\n",
       "   ('development', 0.12613730602526141),\n",
       "   ('building', 0.053491192457649668),\n",
       "   ('permits', 0.053310921577550019),\n",
       "   ('permit', 0.043255341281105038),\n",
       "   ('planning', 0.023919182941866724),\n",
       "   ('infrastructure', 0.021756839510381282),\n",
       "   ('city', 0.020104915204599585),\n",
       "   ('construction', 0.019180129085083777),\n",
       "   ('issued', 0.016689087668068144)]),\n",
       " (10,\n",
       "  [('performance', 0.15014282297314588),\n",
       "   ('measures', 0.084916810125499176),\n",
       "   ('quality', 0.082407480554709422),\n",
       "   ('government', 0.046282664549000041),\n",
       "   ('utah', 0.024726237688587407),\n",
       "   ('life', 0.024493035566701037),\n",
       "   ('environmental', 0.022190639354620373),\n",
       "   ('set', 0.021453160456235888),\n",
       "   ('results', 0.014404567691562294),\n",
       "   ('protection', 0.014186363478051179)]),\n",
       " (11,\n",
       "  [('energy', 0.05380723813451678),\n",
       "   ('state', 0.053280740916985515),\n",
       "   ('year', 0.041298512798120274),\n",
       "   ('payments', 0.036020417444420048),\n",
       "   ('electricity', 0.026847254742470631),\n",
       "   ('june', 0.023065022234194144),\n",
       "   ('calendar', 0.021870375486676235),\n",
       "   ('government', 0.018735809716720113),\n",
       "   ('july', 0.017126262924993831),\n",
       "   ('total', 0.015282139577827715)]),\n",
       " (12,\n",
       "  [('map', 0.065673608197020783),\n",
       "   ('gis', 0.050174838357732295),\n",
       "   ('view', 0.030715104163311073),\n",
       "   ('special', 0.028915595428765057),\n",
       "   ('go', 0.026425760555286969),\n",
       "   ('click', 0.025525461906458814),\n",
       "   ('boundaries', 0.023444240645811885),\n",
       "   ('use', 0.022386945048080399),\n",
       "   ('chicago', 0.022362437784027061),\n",
       "   ('files', 0.02023143276413511)]),\n",
       " (13,\n",
       "  [('food', 0.05262301687907428),\n",
       "   ('cost', 0.036789013950306142),\n",
       "   ('average', 0.03296634334636759),\n",
       "   ('statewide', 0.019919530082943591),\n",
       "   ('program', 0.019765711866778345),\n",
       "   ('contracts', 0.01958755956989915),\n",
       "   ('income', 0.019352930988438453),\n",
       "   ('median', 0.016664673214648932),\n",
       "   ('market', 0.0162591386504473),\n",
       "   ('contract', 0.015282705178735314)]),\n",
       " (14,\n",
       "  [('rates', 0.12661906256052477),\n",
       "   ('rate', 0.096880363456563645),\n",
       "   ('county', 0.050925856493528578),\n",
       "   ('statistics', 0.04922756905190067),\n",
       "   ('zip', 0.041784513123770982),\n",
       "   ('department', 0.039902394776945883),\n",
       "   ('codes', 0.033909660298362163),\n",
       "   ('unit', 0.02871332645173532),\n",
       "   ('prepared', 0.021158499212074338),\n",
       "   ('shown', 0.019098252964788846)]),\n",
       " (15,\n",
       "  [('health', 0.057786568406837539),\n",
       "   ('indicator', 0.037281930951645684),\n",
       "   ('death', 0.034003268599008631),\n",
       "   ('hospital', 0.033332125922179157),\n",
       "   ('contains', 0.020449288873698446),\n",
       "   ('hospitals', 0.01909333054091682),\n",
       "   ('vital', 0.017560524825048433),\n",
       "   ('deaths', 0.017023052986615821),\n",
       "   ('birth', 0.016482077560521346),\n",
       "   ('adjusted', 0.016179063997555126)]),\n",
       " (16,\n",
       "  [('environment', 0.11087873205891415),\n",
       "   ('energy', 0.061876219066803476),\n",
       "   ('gas', 0.04295892049113683),\n",
       "   ('natural', 0.02844958999543843),\n",
       "   ('consumption', 0.025027756264770308),\n",
       "   ('iowa', 0.024513874571871461),\n",
       "   ('air', 0.024300310464907805),\n",
       "   ('sustainable', 0.020720077828084653),\n",
       "   ('emissions', 0.016670393024089739),\n",
       "   ('training', 0.014258215866849567)]),\n",
       " (17,\n",
       "  [('survey', 0.055080504331346064),\n",
       "   ('prevention', 0.029798524604995949),\n",
       "   ('concerning', 0.026602570397414531),\n",
       "   ('facility', 0.023210853585343673),\n",
       "   ('definition', 0.022671619469323469),\n",
       "   ('center', 0.018594650582680124),\n",
       "   ('address', 0.014943240790432544),\n",
       "   ('bay', 0.013127252626146743),\n",
       "   ('site', 0.012493202627897411),\n",
       "   ('point', 0.012417599542331558)]),\n",
       " (18,\n",
       "  [('fire', 0.082392973126039382),\n",
       "   ('infrastructure', 0.073226883363776737),\n",
       "   ('city', 0.044527544947321619),\n",
       "   ('department', 0.022872586600753305),\n",
       "   ('emergency', 0.022068567597666656),\n",
       "   ('economy', 0.020159088051681661),\n",
       "   ('los', 0.019709479564177092),\n",
       "   ('stations', 0.019082080199846536),\n",
       "   ('angeles', 0.019005293789663692),\n",
       "   ('prosperous', 0.015708719171070331)]),\n",
       " (19,\n",
       "  [('code', 0.064378093482520146),\n",
       "   ('cook', 0.047866860211464232),\n",
       "   ('cases', 0.045852159327479609),\n",
       "   ('standard', 0.033628926147047856),\n",
       "   ('court', 0.02250294443453028),\n",
       "   ('risk', 0.020841045332955842),\n",
       "   ('violations', 0.020775778175224188),\n",
       "   ('condition', 0.017639813143572557),\n",
       "   ('discharges', 0.017128710733832583),\n",
       "   ('women', 0.015586229739573514)]),\n",
       " (20,\n",
       "  [('safe', 0.059335443871468781),\n",
       "   ('nutrition', 0.028404437786302351),\n",
       "   ('volume', 0.025405732200222013),\n",
       "   ('cases', 0.023819055393089941),\n",
       "   ('section', 0.019532190367069709),\n",
       "   ('trash', 0.017819693844525766),\n",
       "   ('st', 0.017505741036806938),\n",
       "   ('temporary', 0.017433071453655347),\n",
       "   ('aid', 0.016060412137513171),\n",
       "   ('program', 0.015305411253969253)]),\n",
       " (21,\n",
       "  [('report', 0.13681496598192433),\n",
       "   ('monthly', 0.080151580666805652),\n",
       "   ('finance', 0.07460567280380262),\n",
       "   ('city', 0.035221739315816361),\n",
       "   ('used', 0.031167547666955068),\n",
       "   ('campaign', 0.028008828965699843),\n",
       "   ('topeka', 0.026448886583274517),\n",
       "   ('case', 0.023935878027760989),\n",
       "   ('reports', 0.020724025549784062),\n",
       "   ('month', 0.020596141923020735)]),\n",
       " (22,\n",
       "  [('services', 0.21714635505927651),\n",
       "   ('social', 0.079656374069215857),\n",
       "   ('human', 0.047501865894170762),\n",
       "   ('dfps', 0.044794194931301261),\n",
       "   ('care', 0.038084585826505479),\n",
       "   ('programs', 0.030453187270098797),\n",
       "   ('children', 0.026890816546111185),\n",
       "   ('state', 0.023096741137266109),\n",
       "   ('information', 0.022518470129769926),\n",
       "   ('provider', 0.020131791134192903)]),\n",
       " (23,\n",
       "  [('information', 0.037518961688644596),\n",
       "   ('may', 0.029747532491448306),\n",
       "   ('police', 0.026840642940219085),\n",
       "   ('user', 0.024153108352139404),\n",
       "   ('department', 0.021670300303393775),\n",
       "   ('use', 0.017468104997984693),\n",
       "   ('filter', 0.014575930765739443),\n",
       "   ('injury', 0.012755344463863037),\n",
       "   ('date', 0.010124844228913861),\n",
       "   ('web', 0.0098070132647404759)]),\n",
       " (24,\n",
       "  [('business', 0.10075451106190153),\n",
       "   ('revenue', 0.045210841809066447),\n",
       "   ('license', 0.034293025545457358),\n",
       "   ('finance', 0.033992967813915538),\n",
       "   ('retirement', 0.025277183888041375),\n",
       "   ('economy', 0.025206489893493115),\n",
       "   ('active', 0.022450230732160962),\n",
       "   ('pension', 0.021640583549558615),\n",
       "   ('consumer', 0.018627225056309236),\n",
       "   ('licenses', 0.01806058257747703)]),\n",
       " (25,\n",
       "  [('new', 0.20926421860148925),\n",
       "   ('york', 0.086994586996435821),\n",
       "   ('waste', 0.028743912639261592),\n",
       "   ('plans', 0.019027392307964283),\n",
       "   ('buildings', 0.01790960339395975),\n",
       "   ('orleans', 0.015622794181023782),\n",
       "   ('city', 0.014669461628354441),\n",
       "   ('annually', 0.01391907544646385),\n",
       "   ('nys', 0.013726482686303245),\n",
       "   ('self', 0.012113267591408235)]),\n",
       " (26,\n",
       "  [('number', 0.0428770485860109),\n",
       "   ('population', 0.03853096054912037),\n",
       "   ('age', 0.037237612746123347),\n",
       "   ('percentage', 0.035155800251083405),\n",
       "   ('total', 0.033423397331042451),\n",
       "   ('demographics', 0.022027181730314024),\n",
       "   ('per', 0.01951324261371374),\n",
       "   ('year', 0.01923084092386608),\n",
       "   ('percent', 0.017587426975467276),\n",
       "   ('race', 0.016606086403182781)]),\n",
       " (27,\n",
       "  [('name', 0.024048053914557881),\n",
       "   ('board', 0.022623307125358535),\n",
       "   ('period', 0.019826450796043425),\n",
       "   ('type', 0.019685116587116826),\n",
       "   ('tx', 0.017691335843061085),\n",
       "   ('time', 0.016847853393518797),\n",
       "   ('agency', 0.01657110005713772),\n",
       "   ('number', 0.015635973960090085),\n",
       "   ('date', 0.014595767942192183),\n",
       "   ('includes', 0.013160769088616823)]),\n",
       " (28,\n",
       "  [('transportation', 0.16454273644072911),\n",
       "   ('maryland', 0.031619987773880046),\n",
       "   ('transit', 0.021968378334540423),\n",
       "   ('infrastructure', 0.020000343981233279),\n",
       "   ('standards', 0.019390761377249956),\n",
       "   ('nursing', 0.018192920410357156),\n",
       "   ('bus', 0.017862311087441768),\n",
       "   ('routes', 0.014219481017054773),\n",
       "   ('ecology', 0.011882653708446361),\n",
       "   ('adopted', 0.011728395004195409)]),\n",
       " (29,\n",
       "  [('economic', 0.074397456932756151),\n",
       "   ('development', 0.06746286260195794),\n",
       "   ('inspections', 0.022580242592300591),\n",
       "   ('vehicle', 0.021547911997648841),\n",
       "   ('jobs', 0.020765711354879549),\n",
       "   ('update', 0.019527995310979901),\n",
       "   ('inspection', 0.018786269438061359),\n",
       "   ('community', 0.018770795847216571),\n",
       "   ('grants', 0.018381613152950615),\n",
       "   ('economy', 0.017937030741879641)]),\n",
       " (30,\n",
       "  [('jersey', 0.039976877432761376),\n",
       "   ('city', 0.021014838403706312),\n",
       "   ('land', 0.020332826324993024),\n",
       "   ('parcel', 0.017069807609122536),\n",
       "   ('boundaries', 0.015752689339949937),\n",
       "   ('seattle', 0.013947032297332875),\n",
       "   ('feature', 0.013569101678596814),\n",
       "   ('planning', 0.013246184824297973),\n",
       "   ('analysis', 0.012481306645741114),\n",
       "   ('use', 0.012472410416204383)]),\n",
       " (31,\n",
       "  [('water', 0.096788092953541821),\n",
       "   ('environment', 0.033245543779339116),\n",
       "   ('animal', 0.017955429600823813),\n",
       "   ('monitoring', 0.016141761267292762),\n",
       "   ('sites', 0.014972152458093272),\n",
       "   ('lake', 0.013864652257256839),\n",
       "   ('including', 0.012532158178828176),\n",
       "   ('resources', 0.011936079978637779),\n",
       "   ('national', 0.011869231447825788),\n",
       "   ('salt', 0.010924992069101296)]),\n",
       " (32,\n",
       "  [('employment', 0.061736232807593279),\n",
       "   ('labor', 0.039438041774733326),\n",
       "   ('economy', 0.035445442651735139),\n",
       "   ('unemployment', 0.031980674325731942),\n",
       "   ('job', 0.02580207524674958),\n",
       "   ('weekly', 0.024120999528832312),\n",
       "   ('workforce', 0.018640542230883878),\n",
       "   ('force', 0.018525061218845396),\n",
       "   ('fee', 0.017504124753219812),\n",
       "   ('work', 0.015526474516951792)]),\n",
       " (33,\n",
       "  [('management', 0.050931876524556739),\n",
       "   ('city', 0.049317073798028313),\n",
       "   ('dashboard', 0.044715466355559147),\n",
       "   ('resultsnola', 0.039502910958274108),\n",
       "   ('report', 0.038527327548460631),\n",
       "   ('key', 0.03803146153518052),\n",
       "   ('includes', 0.037801558698075728),\n",
       "   ('goal', 0.032088460469994487),\n",
       "   ('form', 0.030537626476611599),\n",
       "   (\"city's\", 0.029181255214847823)]),\n",
       " (34,\n",
       "  [('education', 0.17144394756224091),\n",
       "   ('school', 0.077067470010658196),\n",
       "   ('students', 0.031165180988619822),\n",
       "   ('schools', 0.029975243254721912),\n",
       "   ('high', 0.025792695460581733),\n",
       "   ('colorado', 0.020219993563855596),\n",
       "   ('metrics', 0.01726546559390239),\n",
       "   ('year', 0.017175144224619007),\n",
       "   ('district', 0.016734522135330537),\n",
       "   ('state', 0.015850885186052512)]),\n",
       " (35,\n",
       "  [('local', 0.038244221184516938),\n",
       "   ('program', 0.034971978843397547),\n",
       "   ('agencies', 0.027340868267011348),\n",
       "   ('network', 0.026920815865666466),\n",
       "   ('state', 0.024523859704053574),\n",
       "   ('act', 0.022052449722116486),\n",
       "   ('dept', 0.015767611548752173),\n",
       "   ('homes', 0.015670453034813375),\n",
       "   ('funded', 0.014981276751193076),\n",
       "   ('funding', 0.013095429404343847)]),\n",
       " (36,\n",
       "  [('street', 0.029958959466164708),\n",
       "   ('streets', 0.021962981967140398),\n",
       "   ('city', 0.020464392882027466),\n",
       "   ('climate', 0.020064550788718528),\n",
       "   ('areas', 0.019680187846933352),\n",
       "   ('area', 0.017754180646546429),\n",
       "   ('region', 0.017736401602454224),\n",
       "   ('information', 0.015656224952851808),\n",
       "   ('plans', 0.015603068922544466),\n",
       "   ('developed', 0.013520490179222633)]),\n",
       " (37,\n",
       "  [('finance', 0.15699400981374406),\n",
       "   ('financial', 0.071012926187898201),\n",
       "   ('government', 0.032353904705819156),\n",
       "   ('expenditures', 0.031323279795486374),\n",
       "   ('fund', 0.02943086016834065),\n",
       "   ('summary', 0.02863886647643232),\n",
       "   ('account', 0.02586394819941833),\n",
       "   ('items', 0.024497132624645878),\n",
       "   ('accounts', 0.023970575655244183),\n",
       "   ('endorsed', 0.020926418892509138)]),\n",
       " (38,\n",
       "  [('crime', 0.071221214884884823),\n",
       "   ('strategic', 0.020992979135410938),\n",
       "   ('public', 0.020550677708716738),\n",
       "   ('plan', 0.019082424938131531),\n",
       "   ('information', 0.016474952532086782),\n",
       "   ('incident', 0.014581572580379926),\n",
       "   ('safety', 0.013710692501380196),\n",
       "   ('reported', 0.013646967997626847),\n",
       "   ('crimes', 0.013156917581380698),\n",
       "   ('police', 0.012895763414225591)]),\n",
       " (39,\n",
       "  [('transportation', 0.048347489734435367),\n",
       "   ('traffic', 0.046857059238468002),\n",
       "   ('counts', 0.036994357693818564),\n",
       "   ('api', 0.024565904144387855),\n",
       "   ('homeless', 0.024256714020289501),\n",
       "   ('access', 0.020506901853792493),\n",
       "   ('gender', 0.018019423644301161),\n",
       "   ('quantities', 0.017302528149327253),\n",
       "   ('count', 0.016858425614350851),\n",
       "   ('link', 0.016087289997948263)]),\n",
       " (40,\n",
       "  [('year', 0.076074895650487903),\n",
       "   ('budget', 0.075257226073174477),\n",
       "   ('fiscal', 0.070363618635769051),\n",
       "   ('finance', 0.056048658939604992),\n",
       "   ('funds', 0.017838940622045014),\n",
       "   ('fy', 0.016790880074999671),\n",
       "   ('spending', 0.01634146570253139),\n",
       "   ('department', 0.013968402815431621),\n",
       "   ('information', 0.013151830156292408),\n",
       "   ('city', 0.012771373130855745)]),\n",
       " (41,\n",
       "  [('service', 0.073177171513935851),\n",
       "   ('requests', 0.032649249934943739),\n",
       "   ('healthy', 0.031270215932668856),\n",
       "   ('311', 0.024429990046110846),\n",
       "   ('communities', 0.023207802724922112),\n",
       "   ('request', 0.02129406941912193),\n",
       "   ('days', 0.018984918522909639),\n",
       "   ('complaints', 0.017861005365830754),\n",
       "   ('zone', 0.01718332530342126),\n",
       "   ('goal', 0.01715733013367984)]),\n",
       " (42,\n",
       "  [('county', 0.11816624246646536),\n",
       "   ('government', 0.1012649647585219),\n",
       "   ('san', 0.079253227599115142),\n",
       "   ('politics', 0.069289429184738141),\n",
       "   ('mateo', 0.038441178987883684),\n",
       "   ('district', 0.028424760874036149),\n",
       "   ('election', 0.027105594957480855),\n",
       "   ('city', 0.022811642852013727),\n",
       "   ('districts', 0.022604434781291718),\n",
       "   ('elections', 0.02103436693216704)]),\n",
       " (43,\n",
       "  [('information', 0.084369740792923267),\n",
       "   ('ny', 0.04640898170629746),\n",
       "   ('us', 0.036791304860584878),\n",
       "   ('details', 0.029639049922539077),\n",
       "   ('locations', 0.026502526801779818),\n",
       "   ('discharge', 0.026326526607525446),\n",
       "   ('visit', 0.019272379856815961),\n",
       "   ('public', 0.01863904352139693),\n",
       "   ('facilities', 0.017916081374834435),\n",
       "   ('library', 0.017915464426800181)]),\n",
       " (44,\n",
       "  [('\"about\"', 0.069961847808411692),\n",
       "   ('counties', 0.050211754871259423),\n",
       "   ('county', 0.045286891765442719),\n",
       "   ('dallas', 0.026274904039437191),\n",
       "   ('metadata', 0.025201367410261164),\n",
       "   ('credit', 0.023431480711222943),\n",
       "   ('cities', 0.022110891751494454),\n",
       "   ('sources', 0.021952309073986355),\n",
       "   ('registry', 0.020120274323178845),\n",
       "   ('see', 0.014278366958971171)])]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_45_fifty.show_topics(num_topics=45, formatted=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Try 48 topics\n",
    "lda_48_forty = gensim.models.ldamodel.LdaModel(corpus, num_topics=48, id2word = dictionary, \n",
    "                                         passes = 40, random_state=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  [('health', 0.34111006609046857),\n",
       "   ('care', 0.042087730089601157),\n",
       "   ('\"about\"', 0.025842829053547376),\n",
       "   ('cases', 0.020041568984374279),\n",
       "   ('services', 0.018166861041868612),\n",
       "   ('department', 0.016766608007023197),\n",
       "   ('demographics', 0.01431613410975068),\n",
       "   ('medical', 0.013670961940252566),\n",
       "   ('facilities', 0.013508392154239044),\n",
       "   ('public', 0.012063311465395103)]),\n",
       " (1,\n",
       "  [('transportation', 0.3229756258776596),\n",
       "   ('transit', 0.034970034774981389),\n",
       "   ('infrastructure', 0.030128117457588435),\n",
       "   ('bay', 0.02689792222626438),\n",
       "   ('seattle', 0.02496197177494526),\n",
       "   ('routes', 0.022608948604884809),\n",
       "   ('treatment', 0.021597817911408388),\n",
       "   ('road', 0.020160147773203036),\n",
       "   ('longitude', 0.0119131594008866),\n",
       "   ('latitude', 0.011611884088580328)]),\n",
       " (2,\n",
       "  [('goal', 0.074563284390782775),\n",
       "   ('payments', 0.043503900545184188),\n",
       "   ('endorsed', 0.038016772158128598),\n",
       "   ('demographics', 0.033722379753645233),\n",
       "   ('crest', 0.030642760605548446),\n",
       "   ('economy', 0.029791192913146931),\n",
       "   ('action', 0.027215575677648787),\n",
       "   ('homeless', 0.024333308281024531),\n",
       "   ('clean', 0.021870430389775136),\n",
       "   ('gender', 0.021587528724597297)]),\n",
       " (3,\n",
       "  [('information', 0.060715801450161817),\n",
       "   ('go', 0.018923715910440321),\n",
       "   ('please', 0.018238009144050264),\n",
       "   ('improvement', 0.018191230180196534),\n",
       "   ('available', 0.015976894333269185),\n",
       "   ('currently', 0.01454539627811211),\n",
       "   ('bike', 0.013775122531773645),\n",
       "   ('provided', 0.010443746474331687),\n",
       "   ('provides', 0.010437103600618763),\n",
       "   ('real', 0.0096981092493160486)]),\n",
       " (4,\n",
       "  [('michigan', 0.10262556069679551),\n",
       "   ('recreation', 0.078840848374361241),\n",
       "   ('state', 0.067354154798435081),\n",
       "   ('open', 0.041118838141061778),\n",
       "   ('parks', 0.039077341294205752),\n",
       "   ('official', 0.033664246139273817),\n",
       "   ('concerning', 0.029712614857845342),\n",
       "   ('created', 0.028670008056070815),\n",
       "   ('park', 0.023830242177678342),\n",
       "   ('(openmichigan@michigan', 0.023265158699974361)]),\n",
       " (5,\n",
       "  [('county', 0.13280442042840196),\n",
       "   ('area', 0.042206249181729126),\n",
       "   ('map', 0.036646002729991037),\n",
       "   ('district', 0.035186383052680979),\n",
       "   ('mateo', 0.031482187065858022),\n",
       "   ('areas', 0.029640666572468336),\n",
       "   ('city', 0.026620242112115791),\n",
       "   ('gis', 0.0233408047280932),\n",
       "   ('boundaries', 0.023188391503969515),\n",
       "   ('districts', 0.02252271268163503)]),\n",
       " (6,\n",
       "  [('tax', 0.10880991433780593),\n",
       "   ('indicators', 0.10751748576162672),\n",
       "   ('property', 0.069308777381756934),\n",
       "   ('sales', 0.037196330343837111),\n",
       "   ('taxes', 0.033786272249013975),\n",
       "   ('government', 0.030322621209323646),\n",
       "   ('local', 0.026614989589206894),\n",
       "   ('healthcare', 0.022395966919868757),\n",
       "   ('strategic', 0.018126820171746917),\n",
       "   ('information', 0.017705183489637347)]),\n",
       " (7,\n",
       "  [('healthy', 0.040536288450239301),\n",
       "   ('maryland', 0.039697837126034828),\n",
       "   ('adult', 0.036783085259813324),\n",
       "   ('metrics', 0.030822305147683672),\n",
       "   ('book', 0.025703656226219635),\n",
       "   ('home', 0.024903559641186471),\n",
       "   ('nursing', 0.023933018471787533),\n",
       "   ('api', 0.023522881054947469),\n",
       "   ('bus', 0.02293761257748279),\n",
       "   ('databook', 0.022415054666852473)]),\n",
       " (8,\n",
       "  [('health', 0.04607631122419701),\n",
       "   ('disease', 0.042454686556044202),\n",
       "   ('county', 0.037094645782666053),\n",
       "   ('ny', 0.027116699955902875),\n",
       "   ('000', 0.026878199339118971),\n",
       "   ('new', 0.026795931778202678),\n",
       "   ('chronic', 0.024624186568081838),\n",
       "   ('per', 0.023635701683009547),\n",
       "   ('rate', 0.023128762038681378),\n",
       "   ('risk', 0.021059789551649891)]),\n",
       " (9,\n",
       "  [('building', 0.069426064860328471),\n",
       "   ('permits', 0.058326880524479693),\n",
       "   ('infrastructure', 0.048682996571987519),\n",
       "   ('permit', 0.047086770500979702),\n",
       "   ('development', 0.039337954440902427),\n",
       "   (\"city's\", 0.032959672036997248),\n",
       "   ('buildings', 0.03154967703844927),\n",
       "   ('city', 0.028393086340869178),\n",
       "   ('housing', 0.02786402608485013),\n",
       "   ('planning', 0.02262791701736672)]),\n",
       " (10,\n",
       "  [('quality', 0.06425249719042396),\n",
       "   ('cook', 0.044591791563257051),\n",
       "   ('environmental', 0.03267110207819339),\n",
       "   ('discharge', 0.031876562849636081),\n",
       "   ('assessment', 0.02902599902046453),\n",
       "   ('inspection', 0.02810715292222192),\n",
       "   ('food', 0.02769900248859801),\n",
       "   ('life', 0.02048286221755222),\n",
       "   ('results', 0.020234439409066621),\n",
       "   ('developed', 0.019548461553473787)]),\n",
       " (11,\n",
       "  [('energy', 0.091246602480905478),\n",
       "   ('government', 0.07742247472195847),\n",
       "   ('politics', 0.068254266616204803),\n",
       "   ('environment', 0.044243459973014689),\n",
       "   ('gas', 0.034684408000976615),\n",
       "   ('election', 0.026590492081611575),\n",
       "   ('electricity', 0.021840843778312986),\n",
       "   ('elections', 0.020643221323303571),\n",
       "   ('consumption', 0.020393713089114602),\n",
       "   ('climate', 0.018841303307979878)]),\n",
       " (12,\n",
       "  [('information', 0.036099481758043159),\n",
       "   ('department', 0.025270524812700885),\n",
       "   ('page', 0.023473425997452571),\n",
       "   ('new', 0.022201738271764963),\n",
       "   ('reporting', 0.021464372101553046),\n",
       "   ('click', 0.020761820019327519),\n",
       "   ('use', 0.018310807950630652),\n",
       "   ('may', 0.017950667591523681),\n",
       "   ('orleans', 0.013922794364798399),\n",
       "   ('based', 0.012359005770715383)]),\n",
       " (13,\n",
       "  [('rates', 0.057348939378130945),\n",
       "   ('per', 0.0498028671914219),\n",
       "   ('year', 0.047485050379768787),\n",
       "   ('calculated', 0.044139516268646178),\n",
       "   ('statewide', 0.036938353654712361),\n",
       "   ('percentage', 0.036455182657467902),\n",
       "   ('average', 0.03230806852210772),\n",
       "   ('rate', 0.031012921104547717),\n",
       "   ('percent', 0.027991000640688043),\n",
       "   ('number', 0.025259184461190555)]),\n",
       " (14,\n",
       "  [('san', 0.083256324265336495),\n",
       "   ('services', 0.063981303571309434),\n",
       "   ('county', 0.05745352356313739),\n",
       "   ('human', 0.039836010292073849),\n",
       "   ('population', 0.030642417829553025),\n",
       "   ('rates', 0.026776647027237355),\n",
       "   ('public', 0.024573785406292827),\n",
       "   ('diego', 0.02449541397292038),\n",
       "   ('emergency', 0.02169821034851048),\n",
       "   ('harbison', 0.02112230044689984)]),\n",
       " (15,\n",
       "  [('hospital', 0.062850696644415047),\n",
       "   ('communities', 0.045842856942120425),\n",
       "   ('library', 0.037526037214752665),\n",
       "   ('hospitals', 0.035981998960511047),\n",
       "   ('contains', 0.027593125776451972),\n",
       "   ('patient', 0.023788168655210388),\n",
       "   ('quality', 0.023167701296608827),\n",
       "   ('info', 0.021979038359709293),\n",
       "   ('medicare', 0.021905736322968276),\n",
       "   ('air', 0.02071179486431695)]),\n",
       " (16,\n",
       "  [('environment', 0.095666481361812342),\n",
       "   ('chart', 0.079280529977737477),\n",
       "   ('month', 0.026519337412139282),\n",
       "   ('training', 0.025385061826662125),\n",
       "   ('release', 0.025149417680599535),\n",
       "   ('weekly', 0.024462146126139599),\n",
       "   ('site', 0.022240585269087381),\n",
       "   ('activities', 0.01868837983101709),\n",
       "   ('modified', 0.017313738908331492),\n",
       "   ('january', 0.016581699915083684)]),\n",
       " (17,\n",
       "  [('animal', 0.025703512353015118),\n",
       "   ('licensing', 0.025618893589230987),\n",
       "   ('services', 0.023255513765651006),\n",
       "   ('agenda', 0.020704541158375969),\n",
       "   ('environment', 0.018687257072300716),\n",
       "   ('ecology', 0.016399384403600777),\n",
       "   ('article', 0.014777649200517703),\n",
       "   ('social', 0.013945292278427807),\n",
       "   ('complex', 0.013934702042136871),\n",
       "   ('conducted', 0.013279597508791069)]),\n",
       " (18,\n",
       "  [('report', 0.11309993335670535),\n",
       "   ('performance', 0.10693239198401772),\n",
       "   ('city', 0.067734745275896041),\n",
       "   ('measures', 0.062061592556929279),\n",
       "   ('finance', 0.051350866859982441),\n",
       "   ('monthly', 0.046945471001827485),\n",
       "   ('management', 0.0455049895142111),\n",
       "   ('administration', 0.039930153090324917),\n",
       "   ('resultsnola', 0.038485593467084182),\n",
       "   ('key', 0.036288364974582578)]),\n",
       " (19,\n",
       "  [('guide', 0.047794623566945832),\n",
       "   ('events', 0.044536378800836898),\n",
       "   ('well', 0.027205940919646992),\n",
       "   ('price', 0.02237309734049241),\n",
       "   ('online', 0.022082874118741651),\n",
       "   ('2)', 0.021740782364228776),\n",
       "   ('open', 0.018255946951179981),\n",
       "   ('revenues', 0.017872296670717464),\n",
       "   ('1)', 0.017824429963492667),\n",
       "   ('portal', 0.017813079949268611)]),\n",
       " (20,\n",
       "  [('dallas', 0.034748998421497132),\n",
       "   ('quarter', 0.025452535201724389),\n",
       "   ('first', 0.024663696937517798),\n",
       "   ('observed', 0.020362491309424538),\n",
       "   ('jail', 0.017805575453660041),\n",
       "   ('st', 0.01545139976095627),\n",
       "   ('300', 0.014901001739766684),\n",
       "   ('services', 0.013986951086546124),\n",
       "   ('shows', 0.013859931319576644),\n",
       "   ('month', 0.013045545674644023)]),\n",
       " (21,\n",
       "  [('tab', 0.027703926397059028),\n",
       "   ('contains', 0.021265951054124353),\n",
       "   ('check', 0.020575259598681624),\n",
       "   ('additional', 0.019926557075709065),\n",
       "   ('date', 0.019675809994619591),\n",
       "   ('year', 0.018864678050128401),\n",
       "   ('county', 0.01884117629509267),\n",
       "   ('information', 0.017810654924052333),\n",
       "   ('years', 0.014985010090415101),\n",
       "   ('details', 0.01363819794121642)]),\n",
       " (22,\n",
       "  [('services', 0.11975029314354596),\n",
       "   ('social', 0.072304774696455965),\n",
       "   ('dfps', 0.043922014276578443),\n",
       "   ('programs', 0.034885560790211781),\n",
       "   ('program', 0.031199977945281048),\n",
       "   ('care', 0.030226424747726681),\n",
       "   ('information', 0.027720049050064734),\n",
       "   ('children', 0.026631709118945232),\n",
       "   ('finance', 0.024459451465222968),\n",
       "   ('state', 0.02389052515457455)]),\n",
       " (23,\n",
       "  [('public', 0.22207268316791848),\n",
       "   ('safety', 0.18159692002033534),\n",
       "   ('police', 0.071458643049135698),\n",
       "   ('department', 0.022338787561749729),\n",
       "   ('user', 0.021760821087791902),\n",
       "   ('city', 0.017534744361574598),\n",
       "   ('crime', 0.017193894617370826),\n",
       "   ('incidents', 0.014504197952397433),\n",
       "   ('incident', 0.013586944817985414),\n",
       "   ('court', 0.011337581486897898)]),\n",
       " (24,\n",
       "  [('new', 0.17502994270000488),\n",
       "   ('york', 0.10739911274604931),\n",
       "   ('state', 0.075382878182894436),\n",
       "   ('lead', 0.025211672433748895),\n",
       "   ('names', 0.021590287928904932),\n",
       "   ('expected', 0.015055803195338209),\n",
       "   ('displays', 0.013479180655113039),\n",
       "   ('number', 0.013172810401006962),\n",
       "   ('contains', 0.011934696371338626),\n",
       "   ('units', 0.010744640447282652)]),\n",
       " (25,\n",
       "  [('state', 0.19013349645700181),\n",
       "   ('department', 0.031393168939200543),\n",
       "   ('colorado', 0.029358296967931567),\n",
       "   ('codes', 0.027654665933774795),\n",
       "   ('race', 0.02302639155114727),\n",
       "   ('federal', 0.022580324723975073),\n",
       "   ('office', 0.020539699438293358),\n",
       "   ('zip', 0.020169157975873608),\n",
       "   ('registry', 0.019496185393976184),\n",
       "   ('contract', 0.018447623799867276)]),\n",
       " (26,\n",
       "  [('illinois', 0.028844827117399118),\n",
       "   ('view', 0.028008239553307262),\n",
       "   ('files', 0.027771473817461329),\n",
       "   ('chicago', 0.02742016174136689),\n",
       "   ('software', 0.02636581825154756),\n",
       "   ('use', 0.02280591623463607),\n",
       "   ('gis', 0.021673466654685902),\n",
       "   ('statistical', 0.020719406955361976),\n",
       "   ('alone', 0.020153481148325457),\n",
       "   ('special', 0.018274185964338312)]),\n",
       " (27,\n",
       "  [('visit', 0.039379847957405643),\n",
       "   ('traffic', 0.036558394238224774),\n",
       "   ('medicaid', 0.031117854081079588),\n",
       "   ('us', 0.029242000674359083),\n",
       "   ('tx', 0.025454758580420352),\n",
       "   ('governance', 0.023412071284364403),\n",
       "   ('counts', 0.023298588662957691),\n",
       "   ('pedestrian', 0.016242298011090193),\n",
       "   ('fair', 0.015854161474508154),\n",
       "   ('infants', 0.013619253993604528)]),\n",
       " (28,\n",
       "  [('plan', 0.048102599522800257),\n",
       "   ('agencies', 0.043761656852759311),\n",
       "   ('plans', 0.038575028938211083),\n",
       "   ('system', 0.030735113727238146),\n",
       "   ('authority', 0.024379305324172613),\n",
       "   ('retirement', 0.023977303394418507),\n",
       "   ('pension', 0.020903683885962129),\n",
       "   ('state', 0.020489279596324399),\n",
       "   ('information', 0.019657152144403982),\n",
       "   ('contracts', 0.019570247925866108)]),\n",
       " (29,\n",
       "  [('community', 0.1114115329269167),\n",
       "   ('economic', 0.090520071910249161),\n",
       "   ('development', 0.078637044500181205),\n",
       "   ('indicator', 0.052348671381409848),\n",
       "   ('parking', 0.032574331401855783),\n",
       "   ('credit', 0.019364606584637921),\n",
       "   ('nys', 0.016567640372408109),\n",
       "   ('city', 0.014920241220154862),\n",
       "   ('tools', 0.01429289052807076),\n",
       "   ('see', 0.014000748161748332)]),\n",
       " (30,\n",
       "  [('jersey', 0.035421864348416814),\n",
       "   ('prevention', 0.030753628637951801),\n",
       "   ('city', 0.02173135191304933),\n",
       "   ('land', 0.01747117470712772),\n",
       "   ('use', 0.017323493000691654),\n",
       "   ('address', 0.015535422260109353),\n",
       "   ('parcel', 0.015372566568602031),\n",
       "   ('survey', 0.013359809308493105),\n",
       "   ('point', 0.012292152537942984),\n",
       "   ('planning', 0.011947640481506042)]),\n",
       " (31,\n",
       "  [('water', 0.090582475969745016),\n",
       "   ('natural', 0.032565121019842429),\n",
       "   ('environment', 0.029408170240893428),\n",
       "   ('resources', 0.021186836899474381),\n",
       "   ('lake', 0.014273050359517326),\n",
       "   ('jobs', 0.01422682466093609),\n",
       "   ('sites', 0.01255379079316553),\n",
       "   ('state', 0.011889305396646102),\n",
       "   ('quantities', 0.011787422898984299),\n",
       "   ('fuel', 0.011337228739557682)]),\n",
       " (32,\n",
       "  [('crime', 0.085224528499059288),\n",
       "   ('filter', 0.031916244433835131),\n",
       "   ('part', 0.03051201484242267),\n",
       "   ('change', 0.02410892251579479),\n",
       "   ('crimes', 0.021561048656334861),\n",
       "   ('violent', 0.019256612218737375),\n",
       "   ('fee', 0.018399066055740088),\n",
       "   ('affordable', 0.018163491914291501),\n",
       "   ('reported', 0.017747290186363809),\n",
       "   ('property', 0.016188079317697539)]),\n",
       " (33,\n",
       "  [('dashboard', 0.058935667816531069),\n",
       "   ('finance', 0.056225568976211435),\n",
       "   ('summary', 0.045126865312294866),\n",
       "   ('ethics', 0.043812125694609474),\n",
       "   ('form', 0.038783430215279427),\n",
       "   ('accounts', 0.032632818977577367),\n",
       "   ('reports', 0.022118261423120265),\n",
       "   ('board', 0.021272359228427226),\n",
       "   ('e', 0.018243452679399006),\n",
       "   ('460', 0.017479683684426865)]),\n",
       " (34,\n",
       "  [('service', 0.077567723438970865),\n",
       "   ('requests', 0.02789766973025163),\n",
       "   ('public', 0.023087237691551764),\n",
       "   ('city', 0.021652290464120375),\n",
       "   ('daily', 0.02135043769872522),\n",
       "   ('311', 0.02076480133246764),\n",
       "   ('request', 0.018090640510719617),\n",
       "   ('tracking', 0.017936331564596678),\n",
       "   ('days', 0.01728117732092567),\n",
       "   ('survey', 0.017095415203307145)]),\n",
       " (35,\n",
       "  [('locations', 0.075217519544207107),\n",
       "   ('iowa', 0.030216586046723323),\n",
       "   ('access', 0.029594340307776534),\n",
       "   ('location', 0.02633775920213503),\n",
       "   ('map', 0.024652800652288396),\n",
       "   ('stations', 0.023194136598173026),\n",
       "   ('station', 0.020506295732808818),\n",
       "   ('easy', 0.016502557042400348),\n",
       "   ('francisco', 0.016224917024241919),\n",
       "   ('points', 0.016193921628155097)]),\n",
       " (36,\n",
       "  [('business', 0.089555767122524291),\n",
       "   ('license', 0.03040374913357528),\n",
       "   ('economy', 0.023666383575045448),\n",
       "   ('street', 0.022744396633882245),\n",
       "   ('list', 0.02118748562735109),\n",
       "   ('activity', 0.021104987534302733),\n",
       "   ('active', 0.019520270705869255),\n",
       "   ('zone', 0.018970507381895672),\n",
       "   ('businesses', 0.018706915919229822),\n",
       "   ('city', 0.018635338057567601)]),\n",
       " (37,\n",
       "  [('housing', 0.13613666831079232),\n",
       "   ('development', 0.092715037212398516),\n",
       "   ('projects', 0.041069073965097333),\n",
       "   ('project', 0.031783653743536637),\n",
       "   ('account', 0.03082745017006551),\n",
       "   ('items', 0.029134415826706804),\n",
       "   ('infrastructure', 0.019543914508581065),\n",
       "   ('washington', 0.017498323693945678),\n",
       "   ('determine', 0.013543310618768945),\n",
       "   ('quarterly', 0.01111503022191253)]),\n",
       " (38,\n",
       "  [('employees', 0.029209068241275206),\n",
       "   ('employment', 0.024659665395701094),\n",
       "   ('employee', 0.022731502052737943),\n",
       "   ('job', 0.020560363657006173),\n",
       "   ('information', 0.018632718694890135),\n",
       "   ('city', 0.017381123398029458),\n",
       "   ('government', 0.017178086706355764),\n",
       "   ('fewer', 0.016704693882009885),\n",
       "   ('cause', 0.015135316177756602),\n",
       "   ('hours', 0.015109567703757786)]),\n",
       " (39,\n",
       "  [('live', 0.02316859536168913),\n",
       "   ('birth', 0.021809546765811816),\n",
       "   ('network', 0.021476451084613917),\n",
       "   ('births', 0.020331499592316959),\n",
       "   ('including', 0.020239845067979632),\n",
       "   ('system', 0.014197276170616157),\n",
       "   ('level', 0.01405289374706103),\n",
       "   ('etc', 0.01330167734331935),\n",
       "   ('equipment', 0.012368400871368597),\n",
       "   ('aged', 0.011980617175282411)]),\n",
       " (40,\n",
       "  [('finance', 0.13347845729945038),\n",
       "   ('budget', 0.06456827743982603),\n",
       "   ('financial', 0.053291090638323275),\n",
       "   ('fiscal', 0.043377625840464569),\n",
       "   ('year', 0.040506274065989251),\n",
       "   ('government', 0.033881108100525741),\n",
       "   ('revenue', 0.029525023082175431),\n",
       "   ('expenditures', 0.025447173571049243),\n",
       "   ('fund', 0.023690337063191232),\n",
       "   ('economy', 0.01820688297723282)]),\n",
       " (41,\n",
       "  [('estimates', 0.079447320056052845),\n",
       "   ('code', 0.076527977513759338),\n",
       "   ('vehicle', 0.041411278252431259),\n",
       "   ('violations', 0.032755668980755714),\n",
       "   ('shown', 0.031660531068105888),\n",
       "   ('current', 0.026445561526935703),\n",
       "   ('unknown', 0.022970330697554275),\n",
       "   ('maintenance', 0.02040127435186978),\n",
       "   ('vehicles', 0.019679807220408234),\n",
       "   ('violation', 0.019673164259098996)]),\n",
       " (42,\n",
       "  [('city', 0.034119234473956851),\n",
       "   ('historic', 0.033076795209977417),\n",
       "   ('plan', 0.021659265204137742),\n",
       "   ('political', 0.021262504545710111),\n",
       "   ('commission', 0.020284394409359599),\n",
       "   ('council', 0.019239426202843146),\n",
       "   ('government', 0.018686066209933884),\n",
       "   ('arts', 0.017900062888970943),\n",
       "   ('master', 0.017894430865467891),\n",
       "   ('standards', 0.016899535711210108)]),\n",
       " (43,\n",
       "  [('education', 0.1621372567676829),\n",
       "   ('school', 0.072583947419091202),\n",
       "   ('students', 0.029580159517734107),\n",
       "   ('schools', 0.028297230645472449),\n",
       "   ('high', 0.02407319747628877),\n",
       "   ('labor', 0.022411272411624189),\n",
       "   ('unemployment', 0.018244612598276113),\n",
       "   ('employment', 0.015195444317505332),\n",
       "   ('student', 0.014400347436259453),\n",
       "   ('year', 0.012802970034699131)]),\n",
       " (44,\n",
       "  [('demographics', 0.10822167569947802),\n",
       "   ('census', 0.097324489237748849),\n",
       "   ('population', 0.064200698323164884),\n",
       "   ('bureau', 0.03015341118012493),\n",
       "   ('age', 0.026936822866379693),\n",
       "   ('community', 0.025071941850688829),\n",
       "   ('survey', 0.022716285865145347),\n",
       "   ('years', 0.022197733784582424),\n",
       "   ('u', 0.021004448564945506),\n",
       "   ('groups', 0.014847448462533207)]),\n",
       " (45,\n",
       "  [('death', 0.046260287354997653),\n",
       "   ('statistics', 0.038865501989061309),\n",
       "   ('deaths', 0.02909596039189917),\n",
       "   ('national', 0.020765738681385024),\n",
       "   ('managed', 0.0206170634473705),\n",
       "   ('cdc', 0.018574731131597806),\n",
       "   ('technology', 0.015418365552010666),\n",
       "   ('office', 0.015352216492570424),\n",
       "   ('graphs', 0.014520991077692499),\n",
       "   ('database', 0.012259793380617442)]),\n",
       " (46,\n",
       "  [('fire', 0.075199236736057468),\n",
       "   ('response', 0.025260492877741524),\n",
       "   ('calls', 0.024058329398616591),\n",
       "   ('emergency', 0.019148090831215724),\n",
       "   ('time', 0.018725158246202665),\n",
       "   ('call', 0.018423678769522282),\n",
       "   ('discharges', 0.01760678646512431),\n",
       "   ('grade', 0.017476343513254222),\n",
       "   ('department', 0.016259640468291415),\n",
       "   ('center', 0.015023758503248318)]),\n",
       " (47,\n",
       "  [('income', 0.074285016650505575),\n",
       "   ('provider', 0.057243956897702217),\n",
       "   ('safe', 0.052703862897327911),\n",
       "   ('waste', 0.046417425897680831),\n",
       "   ('utah', 0.044416265433974374),\n",
       "   ('poverty', 0.026316938016650332),\n",
       "   ('works', 0.023964647486194812),\n",
       "   ('low', 0.018529651128605454),\n",
       "   ('family', 0.017635896621846919),\n",
       "   ('*', 0.017173323285862529)])]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_48_forty.show_topics(num_topics=48, formatted=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 50 topics looks pretty damn good... try 49 and a bunch of passes just to compare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lda_49_sixty = gensim.models.ldamodel.LdaModel(corpus, num_topics=49, id2word = dictionary, \n",
    "                                         passes = 60, random_state=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  [('new', 0.19401806380593659),\n",
       "   ('york', 0.080494504471462794),\n",
       "   ('prevention', 0.035488842816394882),\n",
       "   ('percentage', 0.027175453469101662),\n",
       "   ('inspection', 0.025312274756160352),\n",
       "   ('inspections', 0.024283191255110546),\n",
       "   ('case', 0.01966651872670919),\n",
       "   ('reported', 0.017813933632749392),\n",
       "   ('cases', 0.014989217362823921),\n",
       "   ('department', 0.01423899182353321)]),\n",
       " (1,\n",
       "  [('environment', 0.077563792232817577),\n",
       "   ('recreation', 0.068552759547399977),\n",
       "   ('gas', 0.037045144234832891),\n",
       "   ('parks', 0.035580036085293414),\n",
       "   ('natural', 0.027961643955858952),\n",
       "   ('historic', 0.021556676519912169),\n",
       "   ('energy', 0.021233097162056038),\n",
       "   ('park', 0.019746229255702165),\n",
       "   ('demographics', 0.018809940818879642),\n",
       "   ('greenhouse', 0.016907553791280894)]),\n",
       " (2,\n",
       "  [('fire', 0.1134373467679646),\n",
       "   ('ny', 0.063013405573689465),\n",
       "   ('currently', 0.019268395851742632),\n",
       "   ('important', 0.016565157256828302),\n",
       "   ('forest', 0.013552797076731593),\n",
       "   ('set', 0.012400715671188714),\n",
       "   ('website', 0.011400896478527156),\n",
       "   ('growing', 0.011172646971447431),\n",
       "   ('content', 0.010714321973754957),\n",
       "   ('economy', 0.010655288949195145)]),\n",
       " (3,\n",
       "  [('education', 0.19299388196349218),\n",
       "   ('school', 0.099510065740484321),\n",
       "   ('students', 0.040003396421959381),\n",
       "   ('schools', 0.038629276211793274),\n",
       "   ('high', 0.036966494591377144),\n",
       "   ('governance', 0.020402671388799535),\n",
       "   ('student', 0.019620302298016944),\n",
       "   ('district', 0.018576896377460583),\n",
       "   ('public', 0.017213131681181385),\n",
       "   ('youth', 0.016466889792437035)]),\n",
       " (4,\n",
       "  [('information', 0.052305071337282093),\n",
       "   ('projects', 0.029517310672326182),\n",
       "   ('project', 0.02742680324775042),\n",
       "   ('concerning', 0.022208109737470075),\n",
       "   ('action', 0.014208521199699368),\n",
       "   ('provided', 0.012880743883068696),\n",
       "   ('capital', 0.01235921974275148),\n",
       "   ('additional', 0.012260206043061123),\n",
       "   ('improvement', 0.011169592370938125),\n",
       "   ('please', 0.011024110746650238)]),\n",
       " (5,\n",
       "  [('plan', 0.045699402806273534),\n",
       "   ('mateo', 0.044789252132967715),\n",
       "   ('jersey', 0.042610494356069659),\n",
       "   ('plans', 0.035720570117651645),\n",
       "   ('area', 0.034858671701426605),\n",
       "   ('address', 0.031315853411143665),\n",
       "   ('areas', 0.029490836514807231),\n",
       "   ('quality', 0.025936739891574759),\n",
       "   ('map', 0.022220222776779874),\n",
       "   ('system', 0.01976139581764837)]),\n",
       " (6,\n",
       "  [('communities', 0.040319662347912053),\n",
       "   ('standards', 0.032570163614791529),\n",
       "   ('value', 0.028666509227781765),\n",
       "   ('ratio', 0.026646025666499096),\n",
       "   ('real', 0.026640365174694786),\n",
       "   ('statistical', 0.023819492553538747),\n",
       "   ('discharges', 0.022058066918417163),\n",
       "   ('standard', 0.021946264005214135),\n",
       "   ('national', 0.021228019965819984),\n",
       "   ('analysis', 0.019743615262669992)]),\n",
       " (7,\n",
       "  [('includes', 0.078785851663004072),\n",
       "   ('indicators', 0.072835527521893276),\n",
       "   (\"city's\", 0.057622062270310255),\n",
       "   ('adult', 0.043729786041277606),\n",
       "   ('city', 0.035441378615372603),\n",
       "   ('retirement', 0.030056871405386751),\n",
       "   ('bike', 0.026712108690036176),\n",
       "   ('system', 0.016776185496380873),\n",
       "   ('bicycle', 0.016658754873926717),\n",
       "   ('providence', 0.015576335858751126)]),\n",
       " (8,\n",
       "  [('health', 0.10815280685579683),\n",
       "   ('rates', 0.037365935584385904),\n",
       "   ('disease', 0.036587328734600619),\n",
       "   ('per', 0.034883096037524021),\n",
       "   ('population', 0.034441489808364825),\n",
       "   ('000', 0.023969167862284663),\n",
       "   ('chronic', 0.020962711303446967),\n",
       "   ('public', 0.019174319441518099),\n",
       "   ('statistics', 0.018958988796434779),\n",
       "   ('diego', 0.017308741337287041)]),\n",
       " (9,\n",
       "  [('city', 0.070357570275327561),\n",
       "   ('district', 0.042647312214972428),\n",
       "   ('map', 0.042004990283277269),\n",
       "   ('districts', 0.040106068503540142),\n",
       "   ('boundaries', 0.029343362059680474),\n",
       "   ('gis', 0.02721491447293959),\n",
       "   ('zoning', 0.02282223009349487),\n",
       "   ('planning', 0.021289796953729637),\n",
       "   ('council', 0.019637623916464109),\n",
       "   ('cities', 0.018164324901053259)]),\n",
       " (10,\n",
       "  [('health', 0.34357268739898766),\n",
       "   ('indicators', 0.037357875404334294),\n",
       "   ('information', 0.022501871049583944),\n",
       "   ('centers', 0.018160688412389967),\n",
       "   ('assessment', 0.017314081606385701),\n",
       "   ('home', 0.016244206516083333),\n",
       "   ('community', 0.016052449230469674),\n",
       "   ('level', 0.014501419211735374),\n",
       "   ('fair', 0.012269287265899317),\n",
       "   ('condition', 0.012114987092671137)]),\n",
       " (11,\n",
       "  [('energy', 0.10946160616099405),\n",
       "   ('environment', 0.043977753697640322),\n",
       "   ('electricity', 0.030516439704742146),\n",
       "   ('year', 0.029105849153042145),\n",
       "   ('total', 0.025131873111410425),\n",
       "   ('goal', 0.020924249504734985),\n",
       "   ('residential', 0.020882671323919529),\n",
       "   ('june', 0.01856815661749274),\n",
       "   ('july', 0.017480875231784987),\n",
       "   ('facilities', 0.016779654526647498)]),\n",
       " (12,\n",
       "  [('public', 0.26046640569642981),\n",
       "   ('safety', 0.20559525563076833),\n",
       "   ('police', 0.065734182891543827),\n",
       "   ('crime', 0.029476249271973128),\n",
       "   ('department', 0.02843368064490992),\n",
       "   ('incident', 0.022260676691127083),\n",
       "   ('incidents', 0.021153626547002414),\n",
       "   ('city', 0.013198116390555297),\n",
       "   ('law', 0.0094418239916068524),\n",
       "   ('reports', 0.0079558356611630857)]),\n",
       " (13,\n",
       "  [('housing', 0.19347777803964902),\n",
       "   ('development', 0.14771971411260038),\n",
       "   ('economy', 0.033525772948582694),\n",
       "   ('buildings', 0.029932309627786063),\n",
       "   ('home', 0.015509886498729416),\n",
       "   ('economic', 0.015346637320290794),\n",
       "   ('prosperous', 0.013494780932937711),\n",
       "   ('authority', 0.012175606567228402),\n",
       "   ('infrastructure', 0.011847547773065894),\n",
       "   ('department', 0.011279612962725652)]),\n",
       " (14,\n",
       "  [('county', 0.37832450883181235),\n",
       "   ('zip', 0.038165127907072827),\n",
       "   ('codes', 0.036266447990488546),\n",
       "   ('statewide', 0.029487689200429414),\n",
       "   ('code', 0.028878718058197268),\n",
       "   ('inpatient', 0.023094676322490949),\n",
       "   ('site', 0.023059338057486709),\n",
       "   ('unit', 0.02073926307654481),\n",
       "   ('abuse', 0.014870835417407509),\n",
       "   ('california', 0.012976394824275837)]),\n",
       " (15,\n",
       "  [('hospital', 0.042675682043289823),\n",
       "   ('health', 0.039537078321675294),\n",
       "   ('contains', 0.026171531542715499),\n",
       "   ('hospitals', 0.024459072674749255),\n",
       "   ('medical', 0.021818735685956499),\n",
       "   ('arts', 0.021448030378276171),\n",
       "   ('birth', 0.021348720627450999),\n",
       "   ('api', 0.020076181067591854),\n",
       "   ('township', 0.019987973472142172),\n",
       "   ('births', 0.019819820706380571)]),\n",
       " (16,\n",
       "  [('report', 0.1309219552560153),\n",
       "   ('monthly', 0.075399059823004808),\n",
       "   ('finance', 0.046007109843288867),\n",
       "   ('city', 0.041442894855658316),\n",
       "   ('used', 0.03234426090842895),\n",
       "   ('tab', 0.031943203803660239),\n",
       "   ('annual', 0.025275761011157548),\n",
       "   ('topeka', 0.024755618985466438),\n",
       "   ('training', 0.017288722304270712),\n",
       "   ('weekly', 0.017118942842171997)]),\n",
       " (17,\n",
       "  [('tracking', 0.046472390401901459),\n",
       "   ('agenda', 0.027841674287167075),\n",
       "   ('priority', 0.027117597639052447),\n",
       "   ('ecology', 0.022281079603880687),\n",
       "   ('info', 0.021485583494175543),\n",
       "   ('corporation', 0.020426207631393814),\n",
       "   ('published', 0.018533352882916208),\n",
       "   ('area', 0.018052362900199907),\n",
       "   ('explorer', 0.013644232494249694),\n",
       "   ('points', 0.013565978108942436)]),\n",
       " (18,\n",
       "  [('performance', 0.12619796911864564),\n",
       "   ('michigan', 0.10417005137541095),\n",
       "   ('measures', 0.069938362569113136),\n",
       "   ('resultsnola', 0.047520230770732551),\n",
       "   ('government', 0.046773783797078985),\n",
       "   ('administration', 0.043171003628436684),\n",
       "   ('report', 0.038976118198367576),\n",
       "   ('indicator', 0.038573004385778042),\n",
       "   ('key', 0.03706624917347065),\n",
       "   ('management', 0.036268192091839561)]),\n",
       " (19,\n",
       "  [('court', 0.028262308423449546),\n",
       "   ('board', 0.02614334119610941),\n",
       "   ('document', 0.022338436020090762),\n",
       "   ('officer', 0.021451929349428647),\n",
       "   ('certified', 0.019357010301495819),\n",
       "   ('women', 0.01880077353652642),\n",
       "   ('maps', 0.018485179747532245),\n",
       "   ('office', 0.018194036224456422),\n",
       "   ('technology', 0.01709007627878634),\n",
       "   ('arrests', 0.016776383835491245)]),\n",
       " (20,\n",
       "  [('emergency', 0.091759968423550636),\n",
       "   ('events', 0.055103453693052924),\n",
       "   ('estimates', 0.048856888517411823),\n",
       "   ('calls', 0.043446100434167509),\n",
       "   ('response', 0.039479114556236913),\n",
       "   ('call', 0.027298225210513467),\n",
       "   ('el', 0.02345787429286356),\n",
       "   ('service', 0.021061044981352924),\n",
       "   ('observed', 0.020384429260337518),\n",
       "   ('time', 0.016520919844425327)]),\n",
       " (21,\n",
       "  [('traffic', 0.037832121950726195),\n",
       "   ('time', 0.026819281797699223),\n",
       "   ('month', 0.022132648147401856),\n",
       "   ('counts', 0.020819242441400924),\n",
       "   ('number', 0.019884954128096526),\n",
       "   ('transportation', 0.019395532313999028),\n",
       "   ('vehicle', 0.01924346180732878),\n",
       "   ('average', 0.017557356840885517),\n",
       "   ('period', 0.013365206862085735),\n",
       "   ('release', 0.011970340974015418)]),\n",
       " (22,\n",
       "  [('services', 0.26198292944909751),\n",
       "   ('human', 0.068414033815698119),\n",
       "   ('social', 0.066138737964807931),\n",
       "   ('cases', 0.035991083165306532),\n",
       "   ('support', 0.022541340700012787),\n",
       "   ('animal', 0.019315915147925879),\n",
       "   ('environment', 0.018534754130104917),\n",
       "   ('information', 0.017344742645494786),\n",
       "   ('due', 0.014553910770391891),\n",
       "   ('department', 0.01411833880353088)]),\n",
       " (23,\n",
       "  [('information', 0.033928197730501902),\n",
       "   ('police', 0.025530961936990716),\n",
       "   ('use', 0.024200957189221186),\n",
       "   ('department', 0.021497177230007995),\n",
       "   ('may', 0.018849817452550353),\n",
       "   ('page', 0.017773266158344),\n",
       "   ('orleans', 0.014971862998110767),\n",
       "   ('click', 0.014383351820475412),\n",
       "   ('date', 0.012685259665388577),\n",
       "   ('clear', 0.012499936472938839)]),\n",
       " (24,\n",
       "  [('san', 0.10145612708548912),\n",
       "   ('business', 0.082850931574176834),\n",
       "   ('name', 0.029825146938243778),\n",
       "   ('license', 0.028247588461443183),\n",
       "   ('list', 0.025156199588537938),\n",
       "   ('economy', 0.021823062787004004),\n",
       "   ('businesses', 0.01860944843710069),\n",
       "   ('francisco', 0.017120051989136336),\n",
       "   ('active', 0.016512667519868844),\n",
       "   ('update', 0.01505436994363789)]),\n",
       " (25,\n",
       "  [('tax', 0.11694608192332744),\n",
       "   ('property', 0.091127588840327983),\n",
       "   ('sales', 0.039539303641794227),\n",
       "   ('local', 0.035334004162646794),\n",
       "   ('taxes', 0.034712100363201555),\n",
       "   ('finance', 0.022600697418477478),\n",
       "   ('government', 0.020939276971423525),\n",
       "   ('sale', 0.015210889148540907),\n",
       "   ('properties', 0.014754295305000121),\n",
       "   ('credits', 0.013039459657366985)]),\n",
       " (26,\n",
       "  [('special', 0.053038246740142743),\n",
       "   ('view', 0.050397518445174885),\n",
       "   ('gis', 0.03660416963787301),\n",
       "   ('software', 0.029854609182171594),\n",
       "   ('chicago', 0.02611652228676312),\n",
       "   ('use', 0.022356219602850187),\n",
       "   ('boundaries', 0.020764399898674971),\n",
       "   ('esri', 0.019695511548940611),\n",
       "   ('top', 0.019249989329264281),\n",
       "   ('files', 0.019076301630937792)]),\n",
       " (27,\n",
       "  [('water', 0.079634410479964488),\n",
       "   ('cook', 0.031528072116856397),\n",
       "   ('quality', 0.026134069335503541),\n",
       "   ('resources', 0.022245437041531268),\n",
       "   ('environment', 0.02222168398001154),\n",
       "   ('information', 0.014134212455299272),\n",
       "   ('state', 0.013372745927643998),\n",
       "   ('monitoring', 0.013162833988840243),\n",
       "   ('credit', 0.013057789456318856),\n",
       "   ('environmental', 0.011637380656401285)]),\n",
       " (28,\n",
       "  [('transportation', 0.037029801384655003),\n",
       "   ('waste', 0.036047017246977814),\n",
       "   ('transit', 0.028417424016417905),\n",
       "   ('jobs', 0.024747693533904876),\n",
       "   ('bus', 0.022870650874161683),\n",
       "   ('environment', 0.021916971533381343),\n",
       "   ('station', 0.016735462391944726),\n",
       "   ('recycling', 0.016492783142059542),\n",
       "   ('job', 0.01644456199617585),\n",
       "   ('infrastructure', 0.015640613694203581)]),\n",
       " (29,\n",
       "  [('development', 0.094096826211495316),\n",
       "   ('economic', 0.082316928123772989),\n",
       "   ('community', 0.078796682124209552),\n",
       "   ('counties', 0.045871198811823971),\n",
       "   ('parking', 0.023502376290637377),\n",
       "   ('land', 0.018847904228510348),\n",
       "   ('tab', 0.01800846044245057),\n",
       "   ('nys', 0.016613808731336686),\n",
       "   ('use', 0.01432527765290352),\n",
       "   ('motor', 0.014288601039877924)]),\n",
       " (30,\n",
       "  [('transportation', 0.1276165121039797),\n",
       "   ('street', 0.026933911981754235),\n",
       "   ('city', 0.0269137543065411),\n",
       "   ('streets', 0.019494041554556859),\n",
       "   ('department', 0.013915675419245531),\n",
       "   ('map', 0.013778185088138049),\n",
       "   ('information', 0.013547217598961806),\n",
       "   ('zone', 0.013261060154595095),\n",
       "   ('location', 0.012383873321290175),\n",
       "   ('layer', 0.012201026331358663)]),\n",
       " (31,\n",
       "  [('government', 0.11740317674112909),\n",
       "   ('politics', 0.10765274745622788),\n",
       "   ('infrastructure', 0.051605706066346534),\n",
       "   ('election', 0.041768250283069852),\n",
       "   ('consumption', 0.036015176287927025),\n",
       "   ('results', 0.034918347595297793),\n",
       "   ('elections', 0.032404238078432027),\n",
       "   ('general', 0.013659115133413747),\n",
       "   ('november', 0.012061519598447271),\n",
       "   ('addressing', 0.01161742485919425)]),\n",
       " (32,\n",
       "  [('crime', 0.049074294710586122),\n",
       "   ('employment', 0.046757443176113293),\n",
       "   ('labor', 0.029721263139934572),\n",
       "   ('unemployment', 0.024097891969927211),\n",
       "   ('workforce', 0.015547588236652674),\n",
       "   ('rate', 0.015153150748777973),\n",
       "   ('economy', 0.015127258921971034),\n",
       "   ('part', 0.014052707433973717),\n",
       "   ('force', 0.013843984203384221),\n",
       "   ('violent', 0.013498789549377206)]),\n",
       " (33,\n",
       "  [('summary', 0.053178747951143891),\n",
       "   ('ethics', 0.04891246186288907),\n",
       "   ('finance', 0.046912625788521879),\n",
       "   ('form', 0.043002616923747836),\n",
       "   ('city', 0.03319656019143364),\n",
       "   ('management', 0.0309038920440751),\n",
       "   ('e', 0.029254082517286232),\n",
       "   ('460', 0.01933673642424593),\n",
       "   ('see', 0.01664801975667253),\n",
       "   ('includes', 0.015940505107838499)]),\n",
       " (34,\n",
       "  [('survey', 0.084410415298188463),\n",
       "   ('percent', 0.032064119534147666),\n",
       "   ('year', 0.027588320209221234),\n",
       "   ('education', 0.024835631623466331),\n",
       "   ('colorado', 0.024567165186092595),\n",
       "   ('community', 0.019965011127945336),\n",
       "   ('estimates', 0.019529457514148783),\n",
       "   ('state', 0.019500777201331466),\n",
       "   ('department', 0.018769468947916718),\n",
       "   ('class', 0.016976236465022311)]),\n",
       " (35,\n",
       "  [('healthy', 0.046228856747022019),\n",
       "   ('safe', 0.040975791756714272),\n",
       "   ('goal', 0.036155517415757905),\n",
       "   ('program', 0.033568222224881684),\n",
       "   ('lead', 0.030799562131664862),\n",
       "   ('washington', 0.027211029942391638),\n",
       "   ('network', 0.026458931403368427),\n",
       "   ('community', 0.022701661566674136),\n",
       "   ('well', 0.020392219491125557),\n",
       "   ('primary', 0.020250006127624468)]),\n",
       " (36,\n",
       "  [('state', 0.04637309519869233),\n",
       "   ('cost', 0.039083719115143661),\n",
       "   ('campaign', 0.036139190837354916),\n",
       "   ('commission', 0.032161191044920275),\n",
       "   ('non', 0.026541963732428009),\n",
       "   ('filed', 0.024747508500853572),\n",
       "   ('costs', 0.02329310725231137),\n",
       "   ('using', 0.022864938439181651),\n",
       "   ('term', 0.017408135803283216),\n",
       "   ('significant', 0.01484258140974213)]),\n",
       " (37,\n",
       "  [('financial', 0.11239491282513803),\n",
       "   ('account', 0.042407260603364),\n",
       "   ('items', 0.038389279138996688),\n",
       "   ('provides', 0.031413302114740631),\n",
       "   ('fund', 0.030511180123555318),\n",
       "   ('guide', 0.029349453324988357),\n",
       "   ('general', 0.027968716835027031),\n",
       "   ('operating', 0.024043572773535766),\n",
       "   ('government', 0.023158198034957066),\n",
       "   ('information', 0.021761734736510129)]),\n",
       " (38,\n",
       "  [('rate', 0.023780718304943629),\n",
       "   ('death', 0.022059742727489912),\n",
       "   ('years', 0.022008182069404469),\n",
       "   ('\"about\"', 0.019658596747305863),\n",
       "   ('number', 0.017302595557744507),\n",
       "   ('age', 0.014345548619436689),\n",
       "   ('deaths', 0.013944490097672327),\n",
       "   ('details', 0.013409532862789583),\n",
       "   ('measure', 0.013087294038933774),\n",
       "   ('life', 0.012121081528392358)]),\n",
       " (39,\n",
       "  [('dfps', 0.074505985325052537),\n",
       "   ('us', 0.060570738264009452),\n",
       "   ('information', 0.047289898210830954),\n",
       "   ('agency', 0.041921766581904697),\n",
       "   ('calculated', 0.038232586051622601),\n",
       "   ('visit', 0.036966214333178572),\n",
       "   ('source', 0.031759087694201146),\n",
       "   ('operations', 0.026948391158646431),\n",
       "   ('tx', 0.024048531358500474),\n",
       "   ('metadata', 0.020905768899513233)]),\n",
       " (40,\n",
       "  [('finance', 0.14297029026768104),\n",
       "   ('budget', 0.074854385688774533),\n",
       "   ('year', 0.061436181085755673),\n",
       "   ('fiscal', 0.054733467538925482),\n",
       "   ('revenue', 0.029004912113631227),\n",
       "   ('expenditures', 0.020450368185886227),\n",
       "   ('funds', 0.018627653351522788),\n",
       "   ('government', 0.017664578036939001),\n",
       "   ('agencies', 0.015315297098344539),\n",
       "   ('economy', 0.015006428505204943)]),\n",
       " (41,\n",
       "  [('service', 0.07171628530275391),\n",
       "   ('requests', 0.032894109582375639),\n",
       "   ('311', 0.024706451099968805),\n",
       "   ('public', 0.023121835009440553),\n",
       "   ('works', 0.022129435464488718),\n",
       "   ('code', 0.021793598346364965),\n",
       "   ('request', 0.020021472242281374),\n",
       "   ('violations', 0.019010588175061023),\n",
       "   ('complaints', 0.018188506257192021),\n",
       "   ('cdc', 0.017832277633877208)]),\n",
       " (42,\n",
       "  [('state', 0.1462959082785644),\n",
       "   ('payments', 0.04091580518580782),\n",
       "   ('open', 0.036389069915286829),\n",
       "   ('official', 0.032847174713562043),\n",
       "   ('finance', 0.032063559814549891),\n",
       "   ('created', 0.029279053762198994),\n",
       "   ('accounts', 0.028328689519469801),\n",
       "   ('check', 0.023202766328779881),\n",
       "   ('employee', 0.018990227493440433),\n",
       "   ('contains', 0.01776903125347138)]),\n",
       " (43,\n",
       "  [('facilities', 0.070166096651872609),\n",
       "   ('facility', 0.060828324288657552),\n",
       "   ('discharge', 0.044718634392017592),\n",
       "   ('day', 0.031440848770140147),\n",
       "   ('licensing', 0.02781567233203348),\n",
       "   ('information', 0.026856889157801645),\n",
       "   ('bacteria', 0.023548146020626638),\n",
       "   ('easy', 0.017571253561391761),\n",
       "   ('boating', 0.017257218425063472),\n",
       "   ('equipment', 0.015565041500290117)]),\n",
       " (44,\n",
       "  [('census', 0.096731748460385639),\n",
       "   ('demographics', 0.091234564508041877),\n",
       "   ('population', 0.057021501953206939),\n",
       "   ('strategic', 0.037829574455175985),\n",
       "   ('bureau', 0.024275899405797944),\n",
       "   ('income', 0.023635522257549967),\n",
       "   ('filter', 0.022226743147531699),\n",
       "   ('age', 0.019463326875819656),\n",
       "   ('montgomery', 0.017288006713392628),\n",
       "   ('planning', 0.016796330310602018)]),\n",
       " (45,\n",
       "  [('dashboard', 0.07133667956217446),\n",
       "   ('user', 0.037531775365236254),\n",
       "   ('endorsed', 0.034179451007598415),\n",
       "   ('library', 0.028296829221106719),\n",
       "   ('access', 0.026880611913705534),\n",
       "   ('vermont', 0.018731069435545776),\n",
       "   ('graphs', 0.016726197342162734),\n",
       "   ('among', 0.015945975773127875),\n",
       "   ('book', 0.013875146476062054),\n",
       "   ('infrastructure', 0.013614503920820937)]),\n",
       " (46,\n",
       "  [('permits', 0.08319053403608441),\n",
       "   ('building', 0.079960331123175646),\n",
       "   ('permit', 0.067754214860738446),\n",
       "   ('food', 0.055990761942262456),\n",
       "   ('construction', 0.037263025356690002),\n",
       "   ('issued', 0.026598446349605211),\n",
       "   ('infrastructure', 0.023299101499440228),\n",
       "   ('nutrition', 0.017520967979082577),\n",
       "   ('directory', 0.015782538911134496),\n",
       "   ('city', 0.013876168583576896)]),\n",
       " (47,\n",
       "  [('care', 0.074378634200592569),\n",
       "   ('services', 0.059085409483331144),\n",
       "   ('health', 0.039401424621518515),\n",
       "   ('social', 0.03592106356031495),\n",
       "   ('chart', 0.033380037939970485),\n",
       "   ('state', 0.032179853290101527),\n",
       "   ('programs', 0.030798140187798466),\n",
       "   ('children', 0.028279954997374353),\n",
       "   ('program', 0.019602205178831315),\n",
       "   ('finance', 0.019505240155276018)]),\n",
       " (48,\n",
       "  [('neighborhoods', 0.031611165041509078),\n",
       "   ('employees', 0.03150081845173771),\n",
       "   ('city', 0.030931738300281078),\n",
       "   ('king', 0.024845484493774678),\n",
       "   ('contracts', 0.02421453894476604),\n",
       "   ('demographic', 0.023167302834089732),\n",
       "   ('schedule', 0.022698886802111351),\n",
       "   ('contract', 0.021364393895362942),\n",
       "   ('gender', 0.020714904400949363),\n",
       "   ('salaries', 0.018692133978494369)])]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_49_sixty.show_topics(num_topics=49, formatted=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#lda_50_forty.save('lda_50_forty_clean_data_clusters') #only need to run this once"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lda_52_sixty = gensim.models.ldamodel.LdaModel(corpus, num_topics=52, id2word = dictionary, \n",
    "                                         passes = 60, random_state=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  [('prevention', 0.039228265770020823),\n",
       "   ('hospital', 0.038899194905431915),\n",
       "   ('ny', 0.034882850346069173),\n",
       "   ('healthy', 0.032344049932466652),\n",
       "   ('statewide', 0.029269391777562728),\n",
       "   ('hospitals', 0.022294582283033028),\n",
       "   ('inpatient', 0.021854135158083089),\n",
       "   ('healthcare', 0.018663035226976647),\n",
       "   ('api', 0.018299534915519441),\n",
       "   ('quality-safety-costs', 0.016526700960621438)]),\n",
       " (1,\n",
       "  [('chart', 0.069963659585634108),\n",
       "   ('children', 0.059274004768666677),\n",
       "   ('income', 0.03908399441304488),\n",
       "   ('historic', 0.025093228857131812),\n",
       "   ('home', 0.021852521187430106),\n",
       "   ('assistance', 0.018091177706023875),\n",
       "   ('homes', 0.017940795661987072),\n",
       "   ('families', 0.017918354840225928),\n",
       "   ('low', 0.017857714487521079),\n",
       "   ('pay', 0.017610064270261648)]),\n",
       " (2,\n",
       "  [('population', 0.13151520321707977),\n",
       "   ('county', 0.091960434049722567),\n",
       "   ('demographics', 0.084275005086619365),\n",
       "   ('age', 0.0434468663431666),\n",
       "   ('total', 0.028221558698175012),\n",
       "   ('king', 0.019468810925607257),\n",
       "   ('washington', 0.01927613699149228),\n",
       "   ('montgomery', 0.018261080011821789),\n",
       "   ('number', 0.017900571442482898),\n",
       "   ('years', 0.016960472490539676)]),\n",
       " (3,\n",
       "  [('transportation', 0.1645420154231319),\n",
       "   ('traffic', 0.038249415092180432),\n",
       "   ('street', 0.029708854670228657),\n",
       "   ('parking', 0.026048447964619643),\n",
       "   ('infrastructure', 0.024499661836046696),\n",
       "   ('city', 0.024248488006745492),\n",
       "   ('safe', 0.023908406994518551),\n",
       "   ('vehicle', 0.023135110066475582),\n",
       "   ('streets', 0.021985229238776469),\n",
       "   ('bike', 0.01551025776156169)]),\n",
       " (4,\n",
       "  [('information', 0.045629510720610396),\n",
       "   ('tab', 0.02519267217634192),\n",
       "   ('projects', 0.024289165547602756),\n",
       "   ('concerning', 0.018274567627167695),\n",
       "   ('may', 0.017740368430513121),\n",
       "   ('contains', 0.017113371818170686),\n",
       "   ('project', 0.015854281382776003),\n",
       "   ('provided', 0.013839518701256619),\n",
       "   ('additional', 0.013692382248742174),\n",
       "   ('improvement', 0.012452002727572102)]),\n",
       " (5,\n",
       "  [('locations', 0.069775812645793758),\n",
       "   ('bacteria', 0.02648594877681025),\n",
       "   ('hours', 0.025954898196716671),\n",
       "   ('culture', 0.024828275368617661),\n",
       "   ('county', 0.023656957609001595),\n",
       "   ('levels', 0.02197024161214425),\n",
       "   ('contact', 0.02095100106449152),\n",
       "   ('directory', 0.019849859557347929),\n",
       "   ('e', 0.019300970726415546),\n",
       "   ('contains', 0.018603307582784098)]),\n",
       " (6,\n",
       "  [('property', 0.13136570965022701),\n",
       "   ('consumption', 0.051884459340934608),\n",
       "   ('real', 0.046286074447914313),\n",
       "   ('value', 0.03077786423134931),\n",
       "   ('properties', 0.025283903585934197),\n",
       "   ('significant', 0.021787551505509636),\n",
       "   ('easy', 0.021674540711116137),\n",
       "   ('usage', 0.019934752091226977),\n",
       "   ('assessment', 0.019827225768331354),\n",
       "   ('company', 0.015464303111405545)]),\n",
       " (7,\n",
       "  [('rates', 0.097364682402046362),\n",
       "   ('calculated', 0.069242760650261914),\n",
       "   ('rate', 0.052423706953771118),\n",
       "   ('year', 0.046226566488417858),\n",
       "   ('counts', 0.041696995181128169),\n",
       "   ('average', 0.026100750207484962),\n",
       "   ('seattle', 0.018955865652478202),\n",
       "   ('percent', 0.017770826386179429),\n",
       "   ('mortality', 0.016407853651273428),\n",
       "   ('number', 0.015107387053477006)]),\n",
       " (8,\n",
       "  [('health', 0.094596970812741571),\n",
       "   ('disease', 0.043755686476799373),\n",
       "   ('000', 0.039024957736277545),\n",
       "   ('per', 0.030917812190623686),\n",
       "   ('rate', 0.029034554086777568),\n",
       "   ('rates', 0.026226335228672973),\n",
       "   ('chronic', 0.025069794781680706),\n",
       "   ('risk', 0.022384318262765962),\n",
       "   ('agency', 0.02162102669270825),\n",
       "   ('due', 0.0195463066793998)]),\n",
       " (9,\n",
       "  [('recreation', 0.13128622968124695),\n",
       "   ('parks', 0.064528596736303123),\n",
       "   ('jersey', 0.060593984662716933),\n",
       "   ('park', 0.040013809141157139),\n",
       "   ('center', 0.029877716132019151),\n",
       "   ('neighborhoods', 0.0258798720295484),\n",
       "   ('centers', 0.02214072729834908),\n",
       "   ('facilities', 0.019767883044293812),\n",
       "   ('environment', 0.017226759948461606),\n",
       "   ('access', 0.01570620198722774)]),\n",
       " (10,\n",
       "  [('measures', 0.092380842032599053),\n",
       "   ('quality', 0.077820821260262546),\n",
       "   ('performance', 0.069476579222436344),\n",
       "   ('government', 0.049591497417367757),\n",
       "   ('life', 0.025916014373607278),\n",
       "   ('developed', 0.024796650118479639),\n",
       "   ('fair', 0.016476546976723017),\n",
       "   ('measurement', 0.014967900754065268),\n",
       "   ('definitions', 0.013866800035767382),\n",
       "   ('index', 0.013374835113969707)]),\n",
       " (11,\n",
       "  [('energy', 0.12371084945388265),\n",
       "   ('environment', 0.090690020883510858),\n",
       "   ('electricity', 0.027522591132607205),\n",
       "   ('air', 0.02448100005593962),\n",
       "   ('sustainable', 0.024139471140549176),\n",
       "   ('action', 0.023631696929809619),\n",
       "   ('climate', 0.023112523604460807),\n",
       "   ('city', 0.019150364298185181),\n",
       "   ('clean', 0.01909891884790434),\n",
       "   ('facilities', 0.01847151888251522)]),\n",
       " (12,\n",
       "  [('map', 0.07003231159311829),\n",
       "   ('\"about\"', 0.065737327359522146),\n",
       "   ('shows', 0.044643143524590485),\n",
       "   ('details', 0.037417466989238299),\n",
       "   ('tracking', 0.033865619389321588),\n",
       "   ('click', 0.028565691919877855),\n",
       "   ('view', 0.027447393223963716),\n",
       "   ('additional', 0.026872823680713778),\n",
       "   ('weekly', 0.021964112590207557),\n",
       "   ('information', 0.017908829658562275)]),\n",
       " (13,\n",
       "  [('gas', 0.059972589629464695),\n",
       "   ('natural', 0.038247506812764911),\n",
       "   ('food', 0.035123659997646883),\n",
       "   ('greenhouse', 0.02737167049995624),\n",
       "   ('environment', 0.026272618464459947),\n",
       "   ('emissions', 0.023135082043847805),\n",
       "   ('costs', 0.021398374769158977),\n",
       "   ('resources', 0.020865670871248309),\n",
       "   ('program', 0.020842524225516069),\n",
       "   ('fuel', 0.016081016497170121)]),\n",
       " (14,\n",
       "  [('health', 0.047486595563458037),\n",
       "   ('code', 0.041436774555512541),\n",
       "   ('statistics', 0.038542408744321223),\n",
       "   ('county', 0.035207719557086382),\n",
       "   ('codes', 0.029854335897479736),\n",
       "   ('zip', 0.029139372691221986),\n",
       "   ('public', 0.024872570253527144),\n",
       "   ('diego', 0.024784995832963057),\n",
       "   ('harbison', 0.023033852102136411),\n",
       "   ('crest', 0.021395606263276593)]),\n",
       " (15,\n",
       "  [('budget', 0.14154768271395413),\n",
       "   ('finance', 0.061629847818770116),\n",
       "   ('fund', 0.052866707925857535),\n",
       "   ('year', 0.027361528382526451),\n",
       "   ('operating', 0.026714548673113327),\n",
       "   ('city', 0.026151540354093086),\n",
       "   ('fiscal', 0.023587756171379842),\n",
       "   ('funds', 0.020153544191078145),\n",
       "   ('calendar', 0.018291264587569866),\n",
       "   ('communities', 0.016813255820410825)]),\n",
       " (16,\n",
       "  [('economy', 0.074831970475957757),\n",
       "   ('employment', 0.059024507694450813),\n",
       "   ('labor', 0.037518750233590671),\n",
       "   ('unemployment', 0.030420042880313702),\n",
       "   ('insurance', 0.022811919476438829),\n",
       "   ('credit', 0.02240291501623564),\n",
       "   ('workforce', 0.021053326053081162),\n",
       "   ('economic', 0.017523311551405966),\n",
       "   ('training', 0.016769979428771047),\n",
       "   ('federal', 0.015805104806649688)]),\n",
       " (17,\n",
       "  [('services', 0.27536646810144061),\n",
       "   ('social', 0.095962651687079945),\n",
       "   ('human', 0.053809687721545367),\n",
       "   ('dfps', 0.052589282054521021),\n",
       "   ('information', 0.03466089006854034),\n",
       "   ('programs', 0.033066788534579276),\n",
       "   ('us', 0.030089170965030963),\n",
       "   ('definition', 0.022733622690420504),\n",
       "   ('visit', 0.02151984124994798),\n",
       "   ('state', 0.020155768734965274)]),\n",
       " (18,\n",
       "  [('column', 0.063427308877555669),\n",
       "   ('update', 0.046675236579513182),\n",
       "   ('homeless', 0.041663950666934658),\n",
       "   ('annually', 0.030072376551628751),\n",
       "   ('frequency', 0.02926894249626533),\n",
       "   ('position', 0.021127522302048646),\n",
       "   ('city', 0.01984153972699073),\n",
       "   ('commercial', 0.01846514098874218),\n",
       "   ('daily', 0.016869053694575468),\n",
       "   ('salaries', 0.01558924337154113)]),\n",
       " (19,\n",
       "  [('death', 0.048126995262336554),\n",
       "   ('maryland', 0.04203981078656735),\n",
       "   ('database', 0.028903753013522989),\n",
       "   ('deaths', 0.027176667718008366),\n",
       "   ('office', 0.027062908105413903),\n",
       "   ('department', 0.024200455450014258),\n",
       "   ('new', 0.020639763071301036),\n",
       "   ('cdc', 0.020246707002184408),\n",
       "   ('metadata', 0.018528581835236948),\n",
       "   ('citizens', 0.017503462164537666)]),\n",
       " (20,\n",
       "  [('expected', 0.032218200719388315),\n",
       "   ('dept', 0.03192755749313185),\n",
       "   ('observed', 0.028518686601885064),\n",
       "   ('every', 0.023146698375314433),\n",
       "   ('300', 0.021255493076380451),\n",
       "   ('pacific', 0.021116246104674861),\n",
       "   ('physical', 0.020257410998302562),\n",
       "   ('performed', 0.018484869357599031),\n",
       "   ('contribution', 0.017489873926872384),\n",
       "   ('std', 0.01619190856680653)]),\n",
       " (21,\n",
       "  [('public', 0.071331431077657451),\n",
       "   ('safety', 0.069027661647336119),\n",
       "   ('fire', 0.047750379014825345),\n",
       "   ('month', 0.020581987820630462),\n",
       "   ('calls', 0.017375649507546796),\n",
       "   ('police', 0.017078124466635133),\n",
       "   ('emergency', 0.016522104916252903),\n",
       "   ('response', 0.015789103751618325),\n",
       "   ('discharge', 0.015068830152377888),\n",
       "   ('incident', 0.014034152768794762)]),\n",
       " (22,\n",
       "  [('arts', 0.030623882738573149),\n",
       "   ('grants', 0.027305070352776701),\n",
       "   ('live', 0.026616435603867498),\n",
       "   ('county', 0.02352575927246791),\n",
       "   ('includes', 0.019948024676078455),\n",
       "   ('library', 0.01873744681009169),\n",
       "   ('clearance', 0.01769201979004003),\n",
       "   ('grant', 0.01744185803724117),\n",
       "   ('inventory', 0.017416447546996693),\n",
       "   ('cps', 0.017216111917570089)]),\n",
       " (23,\n",
       "  [('police', 0.049599652100346163),\n",
       "   ('user', 0.032265367100513642),\n",
       "   ('information', 0.029626460569977066),\n",
       "   ('page', 0.024435878993697473),\n",
       "   ('department', 0.023728883932669205),\n",
       "   ('use', 0.023314363268950755),\n",
       "   ('orleans', 0.01831277092169864),\n",
       "   ('injury', 0.017497753573167622),\n",
       "   ('may', 0.016658073483472296),\n",
       "   ('records', 0.01319973742765217)]),\n",
       " (24,\n",
       "  [('official', 0.061094290910533561),\n",
       "   ('account', 0.04976604753946838),\n",
       "   ('accounts', 0.048301458649605035),\n",
       "   ('(openmichigan@michigan', 0.040820714695857956),\n",
       "   ('special', 0.029163998589317921),\n",
       "   ('nursing', 0.02877038841603027),\n",
       "   ('software', 0.026806546446223656),\n",
       "   ('gis', 0.023974220833955701),\n",
       "   ('use', 0.022528606483685758),\n",
       "   ('consumer', 0.022447395060642104)]),\n",
       " (25,\n",
       "  [('tax', 0.11694941744290623),\n",
       "   ('revenue', 0.058281674929582826),\n",
       "   ('sales', 0.038634383980769106),\n",
       "   ('taxes', 0.034713010510365863),\n",
       "   ('local', 0.033205416149379233),\n",
       "   ('colorado', 0.030687046796886448),\n",
       "   ('government', 0.029706917267673681),\n",
       "   ('state', 0.021996118000518896),\n",
       "   ('department', 0.019154178963866255),\n",
       "   ('sale', 0.015211224135670038)]),\n",
       " (26,\n",
       "  [('public', 0.16838176841104285),\n",
       "   ('safety', 0.1399822838525531),\n",
       "   ('crime', 0.063180867294822307),\n",
       "   ('police', 0.035971649940013783),\n",
       "   ('department', 0.01934244932463609),\n",
       "   ('illinois', 0.015211807554571162),\n",
       "   ('race', 0.013365201285222672),\n",
       "   ('reported', 0.011776909210955196),\n",
       "   ('crimes', 0.011550469075927439),\n",
       "   ('criminal', 0.010510578525453793)]),\n",
       " (27,\n",
       "  [('water', 0.096124708891673349),\n",
       "   ('environment', 0.048101960808142975),\n",
       "   ('protection', 0.020628382724667733),\n",
       "   ('tx', 0.020203690273746876),\n",
       "   ('filed', 0.019569864371137932),\n",
       "   ('waste', 0.019271624659872911),\n",
       "   ('site', 0.019200525696858466),\n",
       "   ('quality', 0.018557602041230845),\n",
       "   ('environmental', 0.017395095439341653),\n",
       "   ('monitoring', 0.015888456167874085)]),\n",
       " (28,\n",
       "  [('transportation', 0.049342201509275964),\n",
       "   ('plans', 0.0463408207807137),\n",
       "   ('iowa', 0.03724002146171048),\n",
       "   ('area', 0.031583902009011684),\n",
       "   ('transit', 0.029746652763280607),\n",
       "   ('operations', 0.029650998680989683),\n",
       "   ('region', 0.028630893370822634),\n",
       "   ('bus', 0.023940404924489834),\n",
       "   ('priority', 0.019765257375106392),\n",
       "   ('people', 0.0157596265394866)]),\n",
       " (29,\n",
       "  [('planning', 0.060932885606218332),\n",
       "   ('county', 0.059246046260552521),\n",
       "   ('district', 0.059082234604936598),\n",
       "   ('districts', 0.046091488648481185),\n",
       "   ('boundaries', 0.037211490590221732),\n",
       "   ('city', 0.029524721059023727),\n",
       "   ('areas', 0.027095006552687029),\n",
       "   ('zoning', 0.026634642336752109),\n",
       "   ('gis', 0.024439676328900622),\n",
       "   ('council', 0.023082692493783637)]),\n",
       " (30,\n",
       "  [('mateo', 0.04184248749834401),\n",
       "   ('city', 0.023985079812541651),\n",
       "   ('gis', 0.019805378260364801),\n",
       "   ('land', 0.019219281849626336),\n",
       "   ('county', 0.018870460262658366),\n",
       "   ('parcel', 0.017318355184099247),\n",
       "   ('map', 0.017293542672449272),\n",
       "   ('point', 0.017217088467835123),\n",
       "   ('survey', 0.016483910162221081),\n",
       "   ('feature', 0.016048488156929334)]),\n",
       " (31,\n",
       "  [('politics', 0.11069826014751941),\n",
       "   ('government', 0.098315050573643273),\n",
       "   ('election', 0.038259777750699074),\n",
       "   ('campaign', 0.030948562034191236),\n",
       "   ('elections', 0.029682305103395073),\n",
       "   ('commission', 0.029314992871497855),\n",
       "   ('results', 0.024928316671568231),\n",
       "   ('city', 0.02132785699829063),\n",
       "   ('460', 0.020080548927070931),\n",
       "   ('finance', 0.019735216016406935)]),\n",
       " (32,\n",
       "  [('open', 0.11136160624479899),\n",
       "   ('created', 0.066633337086342601),\n",
       "   ('items', 0.046975156803616497),\n",
       "   ('clear', 0.024971656082104495),\n",
       "   ('volume', 0.017569028429596321),\n",
       "   ('please', 0.01620818951157825),\n",
       "   ('presented', 0.015509271243001166),\n",
       "   ('difficult', 0.014152486549053123),\n",
       "   ('potential', 0.013902418105983163),\n",
       "   ('note', 0.012958764775942001)]),\n",
       " (33,\n",
       "  [('finance', 0.071043372892685447),\n",
       "   ('summary', 0.060625406434642808),\n",
       "   ('dashboard', 0.057306369737017436),\n",
       "   ('ethics', 0.046020336169751791),\n",
       "   ('form', 0.040459920451206687),\n",
       "   ('management', 0.032747889878733547),\n",
       "   ('board', 0.023358112210499077),\n",
       "   ('pension', 0.017133753473461615),\n",
       "   ('includes', 0.016256755260059545),\n",
       "   ('last', 0.016102898579562843)]),\n",
       " (34,\n",
       "  [('education', 0.18150475758690063),\n",
       "   ('school', 0.081857692569206983),\n",
       "   ('schools', 0.0317766682175891),\n",
       "   ('students', 0.029571629399532486),\n",
       "   ('public', 0.017173423581827232),\n",
       "   ('governance', 0.016783317325556272),\n",
       "   ('state', 0.016578579370884614),\n",
       "   ('year', 0.016159604793580536),\n",
       "   ('student', 0.016139734254594727),\n",
       "   ('district', 0.01600362788318669)]),\n",
       " (35,\n",
       "  [('infrastructure', 0.082445242432479721),\n",
       "   ('city', 0.065893376157151143),\n",
       "   ('public', 0.032845769972150357),\n",
       "   ('maps', 0.030519532936435358),\n",
       "   ('works', 0.027935011789728573),\n",
       "   ('dallas', 0.022443140628194077),\n",
       "   ('los', 0.019882704408148669),\n",
       "   ('angeles', 0.019192175434757104),\n",
       "   ('types', 0.015113120712693303),\n",
       "   ('way', 0.013887150827763371)]),\n",
       " (36,\n",
       "  [('census', 0.078804980837745758),\n",
       "   ('indicators', 0.077373830079416436),\n",
       "   ('survey', 0.048035159786913896),\n",
       "   ('includes', 0.044835173852544735),\n",
       "   ('indicator', 0.036924523941196177),\n",
       "   ('community', 0.033221116599808108),\n",
       "   ('demographics', 0.028209305968815383),\n",
       "   ('bureau', 0.023872041750842676),\n",
       "   ('u', 0.021139289477577056),\n",
       "   ('level', 0.019204313061601801)]),\n",
       " (37,\n",
       "  [('financial', 0.095961100691599369),\n",
       "   ('permits', 0.066863273960211606),\n",
       "   ('building', 0.063195390054562622),\n",
       "   ('permit', 0.054456521696982667),\n",
       "   ('expenditures', 0.042924738948058642),\n",
       "   ('guide', 0.025058053425860767),\n",
       "   ('issued', 0.023243432964197546),\n",
       "   ('construction', 0.022884617487689943),\n",
       "   ('information', 0.02279686241791478),\n",
       "   ('filter', 0.021954851645997791)]),\n",
       " (38,\n",
       "  [('plan', 0.043864302305066827),\n",
       "   ('strategic', 0.034170749236072259),\n",
       "   ('master', 0.02434141736730544),\n",
       "   ('community', 0.020477914960891377),\n",
       "   ('fewer', 0.019330699927391008),\n",
       "   ('limitations', 0.018044419984850159),\n",
       "   ('registry', 0.017693548198715405),\n",
       "   ('cause', 0.016906069114867663),\n",
       "   ('years', 0.015944418985748578),\n",
       "   ('information', 0.014420142106983924)]),\n",
       " (39,\n",
       "  [('health', 0.30531189584891172),\n",
       "   ('san', 0.061101772036273934),\n",
       "   ('county', 0.058639270651817503),\n",
       "   ('care', 0.028005505975059769),\n",
       "   ('counties', 0.019925395256915498),\n",
       "   ('system', 0.016377029581779941),\n",
       "   ('facilities', 0.014007603368699854),\n",
       "   ('network', 0.013561457306569633),\n",
       "   ('home', 0.011206164684491395),\n",
       "   ('francisco', 0.010310472003196668)]),\n",
       " (40,\n",
       "  [('finance', 0.16474209816597238),\n",
       "   ('year', 0.062366577351005315),\n",
       "   ('state', 0.053382253954750457),\n",
       "   ('monthly', 0.050696248199469038),\n",
       "   ('fiscal', 0.047622222135699312),\n",
       "   ('payments', 0.035719919718136567),\n",
       "   ('government', 0.02645402325871795),\n",
       "   ('june', 0.022577028747073),\n",
       "   ('july', 0.021675911556054386),\n",
       "   ('report', 0.021590010484289327)]),\n",
       " (41,\n",
       "  [('service', 0.087262714441883243),\n",
       "   ('requests', 0.035031694542933929),\n",
       "   ('inspection', 0.032678935192867728),\n",
       "   ('inspections', 0.031350351826353977),\n",
       "   ('311', 0.026311943104393398),\n",
       "   ('public', 0.025968462717016926),\n",
       "   ('violations', 0.023768792319548215),\n",
       "   ('request', 0.022587674832020798),\n",
       "   ('complaints', 0.019370415310915674),\n",
       "   ('days', 0.018002600321326952)]),\n",
       " (42,\n",
       "  [('information', 0.024602766215276441),\n",
       "   ('list', 0.021580669434979348),\n",
       "   ('code', 0.017461004706817183),\n",
       "   ('required', 0.016538859124753839),\n",
       "   ('political', 0.013997537552533643),\n",
       "   ('utah', 0.013858241707600623),\n",
       "   ('public', 0.013319630833329574),\n",
       "   ('city', 0.013266613262239497),\n",
       "   ('years', 0.012297917571542876),\n",
       "   ('see', 0.012277655731227213)]),\n",
       " (43,\n",
       "  [('state', 0.1756382967463313),\n",
       "   ('new', 0.13340351799805059),\n",
       "   ('michigan', 0.10266572561037926),\n",
       "   ('york', 0.077727077387110127),\n",
       "   ('information', 0.029283874552224455),\n",
       "   ('check', 0.02032209041652535),\n",
       "   ('measurements', 0.019708043153636445),\n",
       "   ('jobs', 0.017040272127524295),\n",
       "   ('licensing', 0.016976894067415114),\n",
       "   ('ny', 0.016777641736857838)]),\n",
       " (44,\n",
       "  [('book', 0.040135367577023996),\n",
       "   ('lead', 0.040111314911710098),\n",
       "   ('stations', 0.035230307653653078),\n",
       "   ('databook', 0.03363611816437783),\n",
       "   ('station', 0.032779991270326547),\n",
       "   ('closed', 0.023280298248918037),\n",
       "   ('ecology', 0.022906441104600394),\n",
       "   ('solar', 0.019267555473289202),\n",
       "   ('power', 0.01904494570812373),\n",
       "   ('residential', 0.016817032555105343)]),\n",
       " (45,\n",
       "  [('report', 0.11604973949315449),\n",
       "   ('performance', 0.072106784996400927),\n",
       "   ('city', 0.064553993348579072),\n",
       "   ('goal', 0.049701354025019934),\n",
       "   ('resultsnola', 0.045459898953553526),\n",
       "   ('administration', 0.044925526194230785),\n",
       "   (\"city's\", 0.03417954310339625),\n",
       "   ('management', 0.033852728584638718),\n",
       "   ('key', 0.033837307690521595),\n",
       "   ('topeka', 0.022145009545635606)]),\n",
       " (46,\n",
       "  [('address', 0.042514297658594498),\n",
       "   ('name', 0.028895265435991142),\n",
       "   ('number', 0.026095926621437948),\n",
       "   ('facility', 0.021195054756114773),\n",
       "   ('patient', 0.018728329073792953),\n",
       "   ('location', 0.017356509556266626),\n",
       "   ('contains', 0.014702734565212696),\n",
       "   ('type', 0.014078242116513321),\n",
       "   ('g', 0.013900974382997767),\n",
       "   ('transfer', 0.013899331867379633)]),\n",
       " (47,\n",
       "  [('care', 0.11036088229652928),\n",
       "   ('finance', 0.049033830573026793),\n",
       "   ('provider', 0.044146456939484949),\n",
       "   ('child', 0.041722110536953544),\n",
       "   ('cost', 0.034993745722234584),\n",
       "   ('plan', 0.030778145306684032),\n",
       "   ('day', 0.028558344716958277),\n",
       "   ('bay', 0.025286386344854205),\n",
       "   ('government', 0.024128574624377789),\n",
       "   ('fee', 0.018205044303512732)]),\n",
       " (48,\n",
       "  [('business', 0.080327130459714152),\n",
       "   ('license', 0.032302648156979613),\n",
       "   ('job', 0.022002624988460773),\n",
       "   ('contracts', 0.021934165455248577),\n",
       "   ('businesses', 0.019770850331225706),\n",
       "   ('vendor', 0.017832975411663454),\n",
       "   ('owned', 0.017528163358468088),\n",
       "   ('contract', 0.017516996738737415),\n",
       "   ('licenses', 0.016969889175799842),\n",
       "   ('certified', 0.016688461456116979)]),\n",
       " (49,\n",
       "  [('development', 0.17602775675090593),\n",
       "   ('housing', 0.15221265764931263),\n",
       "   ('economic', 0.058854109443175158),\n",
       "   ('community', 0.04859194305354804),\n",
       "   ('infrastructure', 0.019040125293198554),\n",
       "   ('managed', 0.018304696035360443),\n",
       "   ('medicaid', 0.017876406351462711),\n",
       "   ('department', 0.014023829088608938),\n",
       "   ('buildings', 0.013775405992175799),\n",
       "   ('economy', 0.012347394313369113)]),\n",
       " (50,\n",
       "  [('program', 0.03251146357097217),\n",
       "   ('employees', 0.028975172275721717),\n",
       "   ('adult', 0.024408505499127525),\n",
       "   ('employee', 0.02429680205275107),\n",
       "   ('annual', 0.020498612612776752),\n",
       "   ('names', 0.018217941353291824),\n",
       "   ('full', 0.017909118396460151),\n",
       "   ('abuse', 0.016209683412924534),\n",
       "   ('nutrition', 0.015767712459609552),\n",
       "   ('beginning', 0.014213770821721556)]),\n",
       " (51,\n",
       "  [('cook', 0.05830855785510166),\n",
       "   ('high', 0.057680528554684904),\n",
       "   ('cases', 0.039417055039367849),\n",
       "   ('youth', 0.03165394178442571),\n",
       "   ('case', 0.027938083160155187),\n",
       "   ('court', 0.02661638705111698),\n",
       "   ('enrollment', 0.018080429121202373),\n",
       "   ('college', 0.01773522389517114),\n",
       "   ('wa', 0.016770584173977882),\n",
       "   ('gis/maps', 0.016618223819135965)])]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_52_sixty.show_topics(num_topics=52, formatted=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lda_52_sixty.save('lda_52_sixty_good_model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion of Results:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a very decent topic analysis. By my count, **we've created 48 genuinely useful \"clusters\" of latent topics.** Since topic analysis is probabilistic, and our data contains \"noise\" (that is, words either far too specific or far too common to add real meaning) to begin with, **we have a few topics that won't help us.**\n",
    "- Topic 32 is just too vague. It probably has something to do with some open government performance metrics, but there are no words to really a distinguish it.  This is not a failure of the algorithm; I don't doubt these words really are occuring together alot in the data. But \"open, created, items\" doesn't tell us much.\n",
    "- Topic 24 clearly has to do with some \"special\" software/gis for open data (probably having to do with nursing) -- it looks like through the openmichigan portal. Again, these words are quite likely appearing together. But the practical application of a topic like this when it comes to determinging what is popular in the real world is limited.\n",
    "- Topic 42 has a similar problem to topics 32 and 42 topics, but is a little more clear for content type. It looks like political information for several years. But we can't really tell more specifics.\n",
    "- Topic 43 is what I'd call a \"parochial\" topic; it's clearly about common information like jobs and licensing from new york and michigan. Again, not useful as its own topic - but my spin is that this helps isolate \"new york\" and \"ny\" to keep them from inflating other topic views.\n",
    "\n",
    "**Other topics are useful, but may mix content. It could be a vagary of the English language, or it might actually reveal new insights:**\n",
    "- Topic 51 is a topic that tells us something, but it appears to be a mix of youth court cases and youth college enrollment. This could be that \"enrollment\" is used in English in the context of college and court-mandated programs. Then again, this could be \"at-risk youth outcome statistics - either court or college (or both).\n",
    "- Topic 37 appears to be about the construction, permits, and financials of building projects; since these are are pretty closely related in real life, however, I'd argue this is a good topic formation.\n",
    "\n",
    "**Many topics contain a proper name, but clearly identify something useful:**\n",
    "- Topic 9 -- \"recreation\", \"parks\", \"jersey\", \"park\", \"centers\", \"neighborhood\", \"centers\", etc. is clearly about parks and recreation facilities. It just contains jersey. Because of how we'll award our views/downloads counts to each topic, \"jersey\" should only skew this slightly. Again, this is just \"error\" (but not really error, according to the model - jersey probably had a lot of parks and recreation datasets) we have to tolerate if we can't exclude proper names as stop words.\n",
    "\n",
    "**Topic 13 has a seemingly weird outlier:**\n",
    "- Topic 13 is clearly about gas & fuel emissions, but also contains \"food\". I'm guessing \"food\" is linked to \"gas\" by the word \"natural\" across many datasets (natural food, natural gas). This is just noise we will have to tolerate.\n",
    "\n",
    "**Many topics are absolutely beautiful. A few examples:**\n",
    "- Topic 11 -- energy, environment, electricity, air, sustainable, action, climate, city, clean, facilities -- identifies a city's environmental initiatives with words I wouldn't have even thought of to group together.\n",
    "- Topic 31 -- politics, government, election, campaign, elections, commissions, results, etc -- leaves no doubt about its content.\n",
    "- Topic 5 -- locations, bacteria, hours, culture, county, levels, contact, directory, e [as in e coli], contains -- is amazing. Words that could be all sorts of different topics that become so clear in context together. This is about local bacteria levels! (Presumably in lakes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Also to note**: I chose 50 topics after extensive trial and error. Fewer and more topics results in worse \"human eye\" evaluation of topics. It appears you need to provide enough topics to draw out parochial and general words into their own topics, without choosing so many topics that you stretch them too thin."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Topic Composition of Documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus_lda = lda_52_sixty[corpus] #this is just a wrapper; calculates on the fly when you call it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(36, 0.060921471085349795), (47, 0.88565972549584659)]\n",
      "[(25, 0.11324786324786323), (30, 0.18654121408011587), (38, 0.22435897435897434), (40, 0.10694707536980026), (50, 0.26847752251589579)]\n",
      "[(0, 0.059954751131221624), (9, 0.1073825278351072), (16, 0.35215278936410371), (31, 0.058460290192871016), (33, 0.12027274148133688), (49, 0.13209364207680635), (51, 0.11877828054298625)]\n"
     ]
    }
   ],
   "source": [
    "#view topic composition of documents\n",
    "for doc in corpus_lda[10:13]:\n",
    "    print(doc) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus_lda_list = list(corpus_lda) #bit of a run time here, as corpus_lda was just a wrapper; this calcs on the fly\n",
    "\n",
    "#we're going to use this list later for a df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Our corpus index numbers and df index numbers DO line up, as we can see below:\n",
    "- corpus[] returns integer ids and frequency for each string\n",
    "- dictionary.token2id shows us the id for each string so we can look'\n",
    "- big_mash.big_mash[] shows that cell's strings in our df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(6, 1)]"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus[500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'point': 0,\n",
       " 'time': 1,\n",
       " 'homelessness': 2,\n",
       " 'ss': 3,\n",
       " 'dchs': 4,\n",
       " 'housing': 5,\n",
       " 'transportation': 6,\n",
       " 'delete': 7,\n",
       " 'office': 8,\n",
       " 'fair': 9,\n",
       " 'well': 10,\n",
       " 'rent': 11,\n",
       " 'relief': 12,\n",
       " 'well-being': 13,\n",
       " 'safety': 14,\n",
       " 'older': 15,\n",
       " 'adults': 16,\n",
       " 'resident': 17,\n",
       " 'survey': 18,\n",
       " 'city': 19,\n",
       " 'strategic': 20,\n",
       " 'plan': 21,\n",
       " 'finance': 22,\n",
       " 'relevant': 23,\n",
       " 'summary': 24,\n",
       " 'reports': 25,\n",
       " 'technical': 26,\n",
       " 'appendices': 27,\n",
       " '<a': 28,\n",
       " 'href=\"https': 29,\n",
       " 'alexandriava': 30,\n",
       " 'gov/performance/default': 31,\n",
       " 'aspx?id=89091\">click': 32,\n",
       " 'here</a>': 33,\n",
       " 'assisted': 34,\n",
       " 'rental': 35,\n",
       " 'education': 36,\n",
       " 'level': 37,\n",
       " 'increase': 38,\n",
       " 'employment': 39,\n",
       " 'workforce': 40,\n",
       " 'development': 41,\n",
       " 'center': 42,\n",
       " 'worfkforce': 43,\n",
       " 'ranges': 44,\n",
       " 'local': 45,\n",
       " 'gross': 46,\n",
       " 'domestic': 47,\n",
       " 'product': 48,\n",
       " 'youth': 49,\n",
       " 'work': 50,\n",
       " 'experience': 51,\n",
       " 'politics': 52,\n",
       " 'force': 53,\n",
       " 'participants': 54,\n",
       " 'behavioral': 55,\n",
       " 'health': 56,\n",
       " 'detox': 57,\n",
       " 'drug': 58,\n",
       " 'use': 59,\n",
       " 'report': 60,\n",
       " 'alaina': 61,\n",
       " 'sov': 62,\n",
       " 'tes': 63,\n",
       " 'emergency': 64,\n",
       " 'services': 65,\n",
       " 'community': 66,\n",
       " 'partners': 67,\n",
       " 'dental': 68,\n",
       " 'care': 69,\n",
       " 'sexual': 70,\n",
       " 'assault': 71,\n",
       " 'violence': 72,\n",
       " 'opa': 73,\n",
       " 'count': 74,\n",
       " 'departments': 75,\n",
       " 'participating': 76,\n",
       " 'led': 77,\n",
       " 'performance': 78,\n",
       " 'tracking': 79,\n",
       " 'fy': 80,\n",
       " 'including': 81,\n",
       " 'cm': 82,\n",
       " 'non': 83,\n",
       " 'status': 84,\n",
       " 'child': 85,\n",
       " 'protection': 86,\n",
       " 'bh': 87,\n",
       " 'opioid': 88,\n",
       " 'treatment': 89,\n",
       " 'sa': 90,\n",
       " 'cy': 91,\n",
       " 'benefits': 92,\n",
       " 'loans': 93,\n",
       " \"al's\": 94,\n",
       " 'pals': 95,\n",
       " 'children': 96,\n",
       " 'substance': 97,\n",
       " 'abuse': 98,\n",
       " 'early': 99,\n",
       " 'childhood': 100,\n",
       " 'school': 101,\n",
       " 'readiness': 102,\n",
       " 'card': 103,\n",
       " 'master': 104,\n",
       " 'home': 105,\n",
       " 'aps': 106,\n",
       " 'environment': 107,\n",
       " 'tes;': 108,\n",
       " 'winter': 109,\n",
       " 'weather': 110,\n",
       " 'social': 111,\n",
       " 'emotional': 112,\n",
       " 'skills': 113,\n",
       " 'dpi': 114,\n",
       " 'populate': 115,\n",
       " 'open': 116,\n",
       " 'retain': 117,\n",
       " 'infrastructure': 118,\n",
       " 'audit': 119,\n",
       " 'stats': 120,\n",
       " 'socrata': 121,\n",
       " 'date': 122,\n",
       " 'water': 123,\n",
       " 'quality': 124,\n",
       " 'homeless': 125,\n",
       " 'shelter': 126,\n",
       " 'alternative': 127,\n",
       " 'dv': 128,\n",
       " 'pavement': 129,\n",
       " 'condition': 130,\n",
       " 'internal': 131,\n",
       " 'psychiatric': 132,\n",
       " 'shape': 133,\n",
       " 'file': 134,\n",
       " 'recyling': 135,\n",
       " 'diversion': 136,\n",
       " 'sheriff;': 137,\n",
       " 'poverty': 138,\n",
       " 'collected': 139,\n",
       " 'small': 140,\n",
       " 'area': 141,\n",
       " 'income': 142,\n",
       " 'estimates': 143,\n",
       " 'u': 144,\n",
       " 'census': 145,\n",
       " 'bureau': 146,\n",
       " 'risky': 147,\n",
       " 'behavior': 148,\n",
       " 'cymp': 149,\n",
       " 'updated': 150,\n",
       " 'june': 151,\n",
       " 'waste': 152,\n",
       " 'collection': 153,\n",
       " 'vacancy': 154,\n",
       " 'economy': 155,\n",
       " 'units': 156,\n",
       " 'rate': 157,\n",
       " 'unemployment': 158,\n",
       " 'jobs': 159,\n",
       " 'bls': 160,\n",
       " 'percentage': 161,\n",
       " 'low': 162,\n",
       " 'moderate': 163,\n",
       " 'households': 164,\n",
       " 'considered': 165,\n",
       " 'cost': 166,\n",
       " 'burdened': 167,\n",
       " 'percent': 168,\n",
       " 'alexandria': 169,\n",
       " 'state': 170,\n",
       " 'virginia': 171,\n",
       " 'united': 172,\n",
       " 'states': 173,\n",
       " 'green': 174,\n",
       " 'buildings': 175,\n",
       " 'demographics': 176,\n",
       " 'ia': 177,\n",
       " 'includes': 178,\n",
       " 'recommendations': 179,\n",
       " 'closed': 180,\n",
       " 'days': 181,\n",
       " 'per': 182,\n",
       " 'fiscal': 183,\n",
       " 'year': 184,\n",
       " 'omb': 185,\n",
       " 'recreation': 186,\n",
       " 'programs': 187,\n",
       " 'participation': 188,\n",
       " 'comes': 189,\n",
       " 'department': 190,\n",
       " 'parks': 191,\n",
       " 'cultural': 192,\n",
       " 'activities': 193,\n",
       " 'needs': 194,\n",
       " 'assessment': 195,\n",
       " 'conducted': 196,\n",
       " 'every': 197,\n",
       " 'starting': 198,\n",
       " 'questions': 199,\n",
       " 'full': 200,\n",
       " 'range': 201,\n",
       " 'usage': 202,\n",
       " 'satisfaction': 203,\n",
       " 'unmet': 204,\n",
       " 'priorities': 205,\n",
       " 'issues': 206,\n",
       " 'tree': 207,\n",
       " 'canopy': 208,\n",
       " 'response': 209,\n",
       " 'times': 210,\n",
       " 'case': 211,\n",
       " 'management': 212,\n",
       " 'clean': 213,\n",
       " 'air': 214,\n",
       " 'sewer': 215,\n",
       " 'flooding': 216,\n",
       " 'employment.': 217,\n",
       " 'number': 218,\n",
       " 'counts': 219,\n",
       " 'wide': 220,\n",
       " 'indicators': 221,\n",
       " 'managed': 222,\n",
       " '\"goals': 223,\n",
       " 'strategies\"': 224,\n",
       " 'regime': 225,\n",
       " 'building': 226,\n",
       " 'summer': 227,\n",
       " 'program': 228,\n",
       " 'wdc': 229,\n",
       " 'space': 230,\n",
       " 'job': 231,\n",
       " 'listings': 232,\n",
       " 'detailed': 233,\n",
       " 'jobboard': 234,\n",
       " 'raw': 235,\n",
       " 'violent': 236,\n",
       " 'crime': 237,\n",
       " 'provided': 238,\n",
       " 'taken': 239,\n",
       " 'market': 240,\n",
       " 'affordable': 241,\n",
       " 'obesity': 242,\n",
       " 'infant': 243,\n",
       " 'mortality': 244,\n",
       " 'activity': 245,\n",
       " 'b.h.': 246,\n",
       " 'stable': 247,\n",
       " 'satisfying': 248,\n",
       " 'lifestyle': 249,\n",
       " 'wait': 250,\n",
       " 'list': 251,\n",
       " 'business': 252,\n",
       " 'mous': 253,\n",
       " 'general': 254,\n",
       " 'goals': 255,\n",
       " 'discharge': 256,\n",
       " 'lakers': 257,\n",
       " 'game': 258,\n",
       " 'outcomes': 259,\n",
       " 'kobe': 260,\n",
       " 'bryant': 261,\n",
       " 'box': 262,\n",
       " 'scores': 263,\n",
       " 'remain': 264,\n",
       " 'oa': 265,\n",
       " 'hawaiian': 266,\n",
       " 'oha': 267,\n",
       " 'students': 268,\n",
       " 'university': 269,\n",
       " 'formal': 270,\n",
       " 'culture': 271,\n",
       " 'participate': 272,\n",
       " 'baseline': 273,\n",
       " 'land': 274,\n",
       " 'individual': 275,\n",
       " 'rights': 276,\n",
       " 'results': 277,\n",
       " '(2010': 278,\n",
       " '2018)': 279,\n",
       " 'hawai‘i': 280,\n",
       " 'residents': 281,\n",
       " 'understand': 282,\n",
       " 'agree': 283,\n",
       " 'viable': 284,\n",
       " 'base': 285,\n",
       " 'necessary': 286,\n",
       " 'new': 287,\n",
       " 'native': 288,\n",
       " 'governing': 289,\n",
       " 'entity': 290,\n",
       " 'aina': 291,\n",
       " 'sustainability': 292,\n",
       " 'environmental': 293,\n",
       " 'increasing': 294,\n",
       " 'ka': 295,\n",
       " 'pae': 296,\n",
       " \"'āina\": 297,\n",
       " 'hawaii': 298,\n",
       " 'create': 299,\n",
       " 'economic': 300,\n",
       " 'value': 301,\n",
       " 'preserve': 302,\n",
       " 'natural': 303,\n",
       " 'resources': 304,\n",
       " 'historic': 305,\n",
       " 'properties': 306,\n",
       " 'and/or': 307,\n",
       " 'provide': 308,\n",
       " 'opportunities': 309,\n",
       " 'hawaiians': 310,\n",
       " 'sustainable': 311,\n",
       " 'balanced': 312,\n",
       " 'manner': 313,\n",
       " '12%': 314,\n",
       " '15%': 315,\n",
       " 'ahupuaʻa': 316,\n",
       " 'landowners': 317,\n",
       " 'based': 318,\n",
       " 'resource': 319,\n",
       " 'place': 320,\n",
       " 'student': 321,\n",
       " 'history': 322,\n",
       " '85%': 323,\n",
       " 'hawaiʻi': 324,\n",
       " 'appreciate': 325,\n",
       " 'reading': 326,\n",
       " 'disease': 327,\n",
       " 'table': 328,\n",
       " 'depicts': 329,\n",
       " 'prevalence': 330,\n",
       " '000': 331,\n",
       " 'selected': 332,\n",
       " 'chronic': 333,\n",
       " 'diseases': 334,\n",
       " 'conditions': 335,\n",
       " 'age': 336,\n",
       " 'adjusted': 337,\n",
       " 'three': 338,\n",
       " 'discussed': 339,\n",
       " 'asthma': 340,\n",
       " 'cardiovascular': 341,\n",
       " '(heart)': 342,\n",
       " 'diabetes': 343,\n",
       " 'hypertension': 344,\n",
       " 'known': 345,\n",
       " 'risk': 346,\n",
       " 'multiple': 347,\n",
       " 'affect': 348,\n",
       " 'many': 349,\n",
       " 'also': 350,\n",
       " 'addressed': 351,\n",
       " 'brief': 352,\n",
       " 'description': 353,\n",
       " 'factors': 354,\n",
       " 'monthly': 355,\n",
       " 'average': 356,\n",
       " 'inmates': 357,\n",
       " 'occc': 358,\n",
       " 'public': 359,\n",
       " 'h': 360,\n",
       " 'power': 361,\n",
       " 'reduction': 362,\n",
       " 'category': 363,\n",
       " 'grade': 364,\n",
       " 'equipment': 365,\n",
       " 'expenditure': 366,\n",
       " '(hsa': 367,\n",
       " 'standards': 368,\n",
       " 'define': 369,\n",
       " 'learning': 370,\n",
       " 'testing': 371,\n",
       " '[hsa': 372,\n",
       " 'proficient': 373,\n",
       " 'tested': 374,\n",
       " 'grades': 375,\n",
       " '(combined)': 376,\n",
       " 'high': 377,\n",
       " 'complex': 378,\n",
       " 'sy': 379,\n",
       " '2011–12': 380,\n",
       " 'hawai‘i]': 381,\n",
       " 'teachers': 382,\n",
       " 'human': 383,\n",
       " 'represents': 384,\n",
       " 'inventory': 385,\n",
       " \"hawaii's\": 386,\n",
       " 'projects': 387,\n",
       " 'owned': 388,\n",
       " 'private': 389,\n",
       " 'profit': 390,\n",
       " 'governmental': 391,\n",
       " 'entities': 392,\n",
       " 'developed': 393,\n",
       " 'funding': 394,\n",
       " 'support': 395,\n",
       " 'federal': 396,\n",
       " 'county': 397,\n",
       " 'energy': 398,\n",
       " 'efficiency': 399,\n",
       " 'training': 400,\n",
       " 'class': 401,\n",
       " 'size': 402,\n",
       " 'reflects': 403,\n",
       " 'sizes': 404,\n",
       " 'budget': 405,\n",
       " 'capital': 406,\n",
       " 'hi': 407,\n",
       " 'race': 408,\n",
       " 'uh': 409,\n",
       " 'degree': 410,\n",
       " 'earned': 411,\n",
       " 'campus': 412,\n",
       " 'overtime': 413,\n",
       " 'regular': 414,\n",
       " 'holiday': 415,\n",
       " 'planned': 416,\n",
       " 'vs': 417,\n",
       " 'actual/projected': 418,\n",
       " 'math': 419,\n",
       " 'mathematics': 420,\n",
       " 'day': 421,\n",
       " 'for:': 422,\n",
       " 'labor': 423,\n",
       " 'statistics': 424,\n",
       " 'population': 425,\n",
       " 'working': 426,\n",
       " 'inspectors': 427,\n",
       " 'consultation': 428,\n",
       " 'branch': 429,\n",
       " 'hiosh': 430,\n",
       " 'displays': 431,\n",
       " 'positions': 432,\n",
       " 'filled': 433,\n",
       " 'required': 434,\n",
       " 'law': 435,\n",
       " 'operating': 436,\n",
       " 'improvement': 437,\n",
       " 'project': 438,\n",
       " '(cip)': 439,\n",
       " 'funds': 440,\n",
       " 'agriculture': 441,\n",
       " 'irrigation': 442,\n",
       " 'medical': 443,\n",
       " 'assistance': 444,\n",
       " '(medicaid)': 445,\n",
       " 'effect': 446,\n",
       " 'since': 447,\n",
       " 'january': 448,\n",
       " '1966': 449,\n",
       " 'august': 450,\n",
       " 'hawaii’s': 451,\n",
       " 'medicaid': 452,\n",
       " 'separated': 453,\n",
       " 'two': 454,\n",
       " 'methods': 455,\n",
       " 'providing': 456,\n",
       " 'major': 457,\n",
       " 'groups': 458,\n",
       " 'recipients': 459,\n",
       " 'generally': 460,\n",
       " 'individuals': 461,\n",
       " 'certified': 462,\n",
       " 'blind': 463,\n",
       " 'disabled': 464,\n",
       " 'coverage': 465,\n",
       " 'fee': 466,\n",
       " 'service': 467,\n",
       " 'providers': 468,\n",
       " 'paid': 469,\n",
       " 'directly': 470,\n",
       " 'called': 471,\n",
       " 'quest': 472,\n",
       " 'beginning': 473,\n",
       " 'february': 474,\n",
       " 'longer': 475,\n",
       " 'covered': 476,\n",
       " 'insurance': 477,\n",
       " 'call': 478,\n",
       " 'expanded': 479,\n",
       " 'access': 480,\n",
       " '(qexa)': 481,\n",
       " 'continue': 482,\n",
       " 'model': 483,\n",
       " 'scr40': 484,\n",
       " 'doh': 485,\n",
       " 'contract': 486,\n",
       " 'execution': 487,\n",
       " 'duration': 488,\n",
       " '(days)': 489,\n",
       " 'distribution': 490,\n",
       " 'years': 491,\n",
       " 'measured': 492,\n",
       " 'award': 493,\n",
       " 'either': 494,\n",
       " 'start': 495,\n",
       " 'notice': 496,\n",
       " 'proceed': 497,\n",
       " 'director’s': 498,\n",
       " 'signature': 499,\n",
       " 'kamehameha': 500,\n",
       " 'schools': 501,\n",
       " 'proficiency': 502,\n",
       " 'race/ethnicity': 503,\n",
       " '2007–08': 504,\n",
       " 'agricultural': 505,\n",
       " 'farmer': 506,\n",
       " 'loan': 507,\n",
       " 'breastfeeding': 508,\n",
       " 'prenatal': 509,\n",
       " '*note': 510,\n",
       " '95%': 511,\n",
       " 'cl': 512,\n",
       " 'refers': 513,\n",
       " 'confi': 514,\n",
       " 'dence': 515,\n",
       " 'interval': 516,\n",
       " 'around': 517,\n",
       " 'estimate': 518,\n",
       " 'employee': 519,\n",
       " 'wages': 520,\n",
       " 'executive': 521,\n",
       " 'employees': 522,\n",
       " 'july': 523,\n",
       " 'development’s': 524,\n",
       " '(dhrd)': 525,\n",
       " 'database': 526,\n",
       " 'information': 527,\n",
       " 'listed': 528,\n",
       " 'accounting': 529,\n",
       " 'attorney': 530,\n",
       " 'tourism': 531,\n",
       " 'commerce': 532,\n",
       " 'consumer': 533,\n",
       " 'affairs': 534,\n",
       " 'defense': 535,\n",
       " 'lands': 536,\n",
       " 'industrial': 537,\n",
       " 'relations': 538,\n",
       " 'taxation': 539,\n",
       " 'libraries': 540,\n",
       " 'system': 541,\n",
       " 'technology': 542,\n",
       " 'innovation': 543,\n",
       " 'decibel': 544,\n",
       " 'levels': 545,\n",
       " 'precise': 546,\n",
       " 'lat/long': 547,\n",
       " 'locations': 548,\n",
       " '\"what': 549,\n",
       " 'important': 550,\n",
       " 'facing': 551,\n",
       " 'today\"': 552,\n",
       " '*': 553,\n",
       " 'ci': 554,\n",
       " 'confidence': 555,\n",
       " 'obese': 556,\n",
       " '(bmi>=30': 557,\n",
       " '0)': 558,\n",
       " 'overweight': 559,\n",
       " '(bmi>=': 560,\n",
       " '9)': 561,\n",
       " 'economics': 562,\n",
       " 'family': 563,\n",
       " 'median': 564,\n",
       " 'equal': 565,\n",
       " '100%': 566,\n",
       " 'greater': 567,\n",
       " 'statewide': 568,\n",
       " '92%': 569,\n",
       " '201': 570,\n",
       " 'tuition': 571,\n",
       " 'pregnancy': 572,\n",
       " 'food': 573,\n",
       " 'bessd': 574,\n",
       " 'dhs': 575,\n",
       " 'tanf': 576,\n",
       " 'temporary': 577,\n",
       " 'aid': 578,\n",
       " 'needy': 579,\n",
       " 'families': 580,\n",
       " '(tanf)': 581,\n",
       " 'cases': 582,\n",
       " 'clients': 583,\n",
       " 'total': 584,\n",
       " 'present': 585,\n",
       " 'farmers': 586,\n",
       " 'owner': 587,\n",
       " 'inspections': 588,\n",
       " 'performed': 589,\n",
       " 'agreement': 590,\n",
       " 'expected': 591,\n",
       " 'perform': 592,\n",
       " '300': 593,\n",
       " 'fy13': 594,\n",
       " '480': 595,\n",
       " 'fy14': 596,\n",
       " '640': 597,\n",
       " 'fy15': 598,\n",
       " 'weight': 599,\n",
       " 'net': 600,\n",
       " 'served': 601,\n",
       " 'sourced': 602,\n",
       " 'section': 603,\n",
       " 'percents': 604,\n",
       " 'given': 605,\n",
       " 'exceeds': 606,\n",
       " 'meets': 607,\n",
       " 'approaches': 608,\n",
       " 'k-12': 609,\n",
       " 'subset': 610,\n",
       " 'sectors': 611,\n",
       " 'experienced': 612,\n",
       " 'noticeable': 613,\n",
       " 'change': 614,\n",
       " 'mid': 615,\n",
       " 'seeker': 616,\n",
       " 'entered': 617,\n",
       " 'one-stop': 618,\n",
       " 'career': 619,\n",
       " 'seekers': 620,\n",
       " 'one': 621,\n",
       " 'stop': 622,\n",
       " 'investment': 623,\n",
       " 'act': 624,\n",
       " 'wegner': 625,\n",
       " 'peyser': 626,\n",
       " 'obtained': 627,\n",
       " 'within': 628,\n",
       " 'first': 629,\n",
       " 'quarter': 630,\n",
       " 'note': 631,\n",
       " '(hhs)': 632,\n",
       " 'identified': 633,\n",
       " 'caucasian': 634,\n",
       " 'filipino': 635,\n",
       " 'japanese': 636,\n",
       " 'other/unknown;': 637,\n",
       " '0–14': 638,\n",
       " '15–17': 639,\n",
       " '18–24': 640,\n",
       " '25–34': 641,\n",
       " '35–44': 642,\n",
       " '45–54': 643,\n",
       " '55–64': 644,\n",
       " '65–74': 645,\n",
       " '75+': 646,\n",
       " 'dept': 647,\n",
       " 'financial': 648,\n",
       " 'include': 649,\n",
       " '(ga)': 650,\n",
       " 'aged': 651,\n",
       " '(aabd)': 652,\n",
       " 'supplemental': 653,\n",
       " 'nutrition': 654,\n",
       " '(snap)': 655,\n",
       " 'renewable': 656,\n",
       " 'portfolio': 657,\n",
       " 'standard': 658,\n",
       " '(rps)': 659,\n",
       " 'electric': 660,\n",
       " 'utility': 661,\n",
       " 'company': 662,\n",
       " 'sells': 663,\n",
       " 'electricity': 664,\n",
       " 'consumption': 665,\n",
       " 'must': 666,\n",
       " 'establish': 667,\n",
       " 'following': 668,\n",
       " 'percentages': 669,\n",
       " '\"renewable': 670,\n",
       " 'electrical': 671,\n",
       " 'energy\"': 672,\n",
       " 'sales': 673,\n",
       " '10%': 674,\n",
       " 'december': 675,\n",
       " '2010;': 676,\n",
       " '2015;': 677,\n",
       " '25%': 678,\n",
       " '2020;': 679,\n",
       " '40%': 680,\n",
       " '2030': 681,\n",
       " '2004.': 682,\n",
       " 'source:': 683,\n",
       " 'pearson': 684,\n",
       " '[as': 685,\n",
       " 'occupied': 686,\n",
       " 'weighted': 687,\n",
       " 'averages': 688,\n",
       " 'brfss': 689,\n",
       " '[2005': 690,\n",
       " '2007]/hawaiian': 691,\n",
       " 'conservation': 692,\n",
       " 'dhstb': 693,\n",
       " 'snap': 694,\n",
       " '(snap': 695,\n",
       " 'formerly': 696,\n",
       " 'stamps)': 697,\n",
       " '%': 698,\n",
       " 'solid': 699,\n",
       " 'sent': 700,\n",
       " 'landfill': 701,\n",
       " \"o'ahu\": 702,\n",
       " 'personal': 703,\n",
       " 'bpm': 704,\n",
       " 'grouped': 705,\n",
       " 'stewardship': 706,\n",
       " 'powering': 707,\n",
       " 'goal': 708,\n",
       " 'definitions': 709,\n",
       " 'kindergarten': 710,\n",
       " 'receive': 711,\n",
       " 'free': 712,\n",
       " 'reduced': 713,\n",
       " 'lunch': 714,\n",
       " 'source': 715,\n",
       " 'systems': 716,\n",
       " 'accountability': 717,\n",
       " 'footnotes': 718,\n",
       " 'charter': 719,\n",
       " 'government': 720,\n",
       " 'vector': 721,\n",
       " 'pool': 722,\n",
       " 'inspection': 723,\n",
       " 'plano': 724,\n",
       " 'mosquito': 725,\n",
       " \"plano's\": 726,\n",
       " 'set': 727,\n",
       " 'contains': 728,\n",
       " '/': 729,\n",
       " 'departmental': 730,\n",
       " 'functions': 731,\n",
       " 'control': 732,\n",
       " '(vc)': 733,\n",
       " '(p)': 734,\n",
       " '(f)': 735,\n",
       " 'muehlenbeck': 736,\n",
       " 'attendance': 737,\n",
       " 'letters': 738,\n",
       " 'contained': 739,\n",
       " 'parentheses': 740,\n",
       " 'top': 741,\n",
       " 'row': 742,\n",
       " 'denote': 743,\n",
       " 'left': 744,\n",
       " 'right': 745,\n",
       " 'stand': 746,\n",
       " '(t)': 747,\n",
       " '(aq)': 748,\n",
       " 'aquatics': 749,\n",
       " '(adath)': 750,\n",
       " 'adult': 751,\n",
       " 'athletics': 752,\n",
       " '(car)': 753,\n",
       " 'carpenter': 754,\n",
       " 'park': 755,\n",
       " '(lib)': 756,\n",
       " 'liberty': 757,\n",
       " '(oak)': 758,\n",
       " 'oak': 759,\n",
       " '(out)': 760,\n",
       " 'outdoor': 761,\n",
       " '(sen)': 762,\n",
       " 'senior': 763,\n",
       " '(tr)': 764,\n",
       " 'therapeutic': 765,\n",
       " '(tom)': 766,\n",
       " 'tom': 767,\n",
       " 'rain': 768,\n",
       " 'temperature': 769,\n",
       " 'amount': 770,\n",
       " 'pumped': 771,\n",
       " 'ntmwd': 772,\n",
       " 'necessarily': 773,\n",
       " 'used': 774,\n",
       " 'tracked': 775,\n",
       " 'works': 776,\n",
       " 'additionally': 777,\n",
       " '(for': 778,\n",
       " 'rainfall)': 779,\n",
       " 'wunderground': 780,\n",
       " 'com': 781,\n",
       " 'national': 782,\n",
       " 'maintenance': 783,\n",
       " 'fix': 784,\n",
       " 'edited': 785,\n",
       " 'original': 786,\n",
       " 'output': 787,\n",
       " 'ways': 788,\n",
       " '1)': 789,\n",
       " 'id': 790,\n",
       " 'numbers': 791,\n",
       " 'addresses': 792,\n",
       " 'removed': 793,\n",
       " 'privacy': 794,\n",
       " 'reasons': 795,\n",
       " '2)': 796,\n",
       " 'months': 797,\n",
       " 'request': 798,\n",
       " 'requests': 799,\n",
       " 'large': 800,\n",
       " 'volume': 801,\n",
       " 'points': 802,\n",
       " 'puts': 803,\n",
       " 'strain': 804,\n",
       " 'viewing': 805,\n",
       " 'via': 806,\n",
       " 'dashboard': 807,\n",
       " 'order': 808,\n",
       " 'smoother': 809,\n",
       " 'user': 810,\n",
       " 'representative': 811,\n",
       " 'currently': 812,\n",
       " 'completions': 813,\n",
       " 'occurring': 814,\n",
       " 'lastly': 815,\n",
       " 'organized': 816,\n",
       " 'follows': 817,\n",
       " 'several': 818,\n",
       " 'columns': 819,\n",
       " 'show': 820,\n",
       " 'progress': 821,\n",
       " 'etc': 822,\n",
       " 'latter': 823,\n",
       " 'breakdown': 824,\n",
       " 'type': 825,\n",
       " 'requested': 826,\n",
       " 'whether': 827,\n",
       " 'violations': 828,\n",
       " 'animal': 829,\n",
       " 'bonds': 830,\n",
       " \"department's\": 831,\n",
       " 'addition': 832,\n",
       " 'supporting': 833,\n",
       " 'current': 834,\n",
       " 'produced': 835,\n",
       " 'derived': 836,\n",
       " 'police': 837,\n",
       " 'approval': 838,\n",
       " 'rating': 839,\n",
       " 'litigation': 840,\n",
       " 'subrogation': 841,\n",
       " 'describes': 842,\n",
       " 'received': 843,\n",
       " 'claims': 844,\n",
       " 'values': 845,\n",
       " 'growth': 846,\n",
       " 'compare': 847,\n",
       " 'stems': 848,\n",
       " 'comprehensive': 849,\n",
       " 'annual': 850,\n",
       " '(2014': 851,\n",
       " '2013)': 852,\n",
       " 'usinflationcalculator': 853,\n",
       " 'fire': 854,\n",
       " '6/1/2014': 855,\n",
       " 'related': 856,\n",
       " 'measurements': 857,\n",
       " 'customer': 858,\n",
       " \"service's\": 859,\n",
       " 'feedback': 860,\n",
       " 'ratings': 861,\n",
       " 'tabulated': 862,\n",
       " 'hr': 863,\n",
       " 'veterans': 864,\n",
       " 'turnover': 865,\n",
       " 'hiring': 866,\n",
       " '(vt)': 867,\n",
       " 'turnovers': 868,\n",
       " 'complaint': 869,\n",
       " 'homes': 870,\n",
       " 'property': 871,\n",
       " 'division': 872,\n",
       " 'neighborhood': 873,\n",
       " 'wastewater': 874,\n",
       " 'overflow': 875,\n",
       " 'sso': 876,\n",
       " 'sanitary': 877,\n",
       " 'overflows': 878,\n",
       " 'similar': 879,\n",
       " 'measures': 880,\n",
       " 'healthcare': 881,\n",
       " 'compensation': 882,\n",
       " 'workers': 883,\n",
       " 'regarding': 884,\n",
       " 'due': 885,\n",
       " 'nature': 886,\n",
       " 'modified': 887,\n",
       " 'respect': 888,\n",
       " 'identities': 889,\n",
       " 'claimants': 890,\n",
       " 'cities': 891,\n",
       " 'counties': 892,\n",
       " 'countries': 893,\n",
       " 'complete': 894,\n",
       " 'offer': 895,\n",
       " 'websites': 896,\n",
       " 'ozone': 897,\n",
       " 'inversion': 898,\n",
       " 'pm2.5': 899,\n",
       " 'dirty': 900,\n",
       " 'dating': 901,\n",
       " 'back': 902,\n",
       " 'seasonally': 903,\n",
       " 'salt': 904,\n",
       " 'lake': 905,\n",
       " 'release': 906,\n",
       " 'cat': 907,\n",
       " 'dog': 908,\n",
       " 'dogs': 909,\n",
       " 'cats': 910,\n",
       " 'returned': 911,\n",
       " 'adopted': 912,\n",
       " 'ambulance': 913,\n",
       " '911': 914,\n",
       " 'calls': 915,\n",
       " 'vecc': 916,\n",
       " 'subcategories': 917,\n",
       " 'seniors': 918,\n",
       " 'unduplicated': 919,\n",
       " 'aging': 920,\n",
       " 'permit': 921,\n",
       " 'remodel': 922,\n",
       " 'period': 923,\n",
       " 'permits': 924,\n",
       " 'month': 925,\n",
       " 'insured': 926,\n",
       " 'uninsured': 927,\n",
       " 'younger': 928,\n",
       " 'index': 929,\n",
       " 'rates': 930,\n",
       " 'standardized': 931,\n",
       " 'pothole': 932,\n",
       " 'potholes': 933,\n",
       " 'repaired': 934,\n",
       " 'hours': 935,\n",
       " 'zoning': 936,\n",
       " 'single': 937,\n",
       " 'dwelling': 938,\n",
       " 'third': 939,\n",
       " 'title': 940,\n",
       " 'end': 941,\n",
       " 'road': 942,\n",
       " 'roads': 943,\n",
       " '\"good\"': 944,\n",
       " '3rd': 945,\n",
       " 'recycling': 946,\n",
       " 'trash': 947,\n",
       " 'obligation': 948,\n",
       " 'debt': 949,\n",
       " 'bond': 950,\n",
       " 'aaa': 951,\n",
       " 'agencies': 952,\n",
       " \"(moody's\": 953,\n",
       " \"poor's\": 954,\n",
       " 'fitch)': 955,\n",
       " 'assigned': 956,\n",
       " 'triple': 957,\n",
       " \"county's\": 958,\n",
       " 'pre-k': 959,\n",
       " 'prek': 960,\n",
       " 'enrollment': 961,\n",
       " 'preschool': 962,\n",
       " 'olds': 963,\n",
       " 'enrolled': 964,\n",
       " 'graduates': 965,\n",
       " 'graduate': 966,\n",
       " 'signal': 967,\n",
       " 'traffic': 968,\n",
       " 'stoplight': 969,\n",
       " 'repair': 970,\n",
       " 'complaints': 971,\n",
       " 'responded': 972,\n",
       " 'utah': 973,\n",
       " 'active': 974,\n",
       " 'adolescent': 975,\n",
       " 'physical': 976,\n",
       " 'adolescents': 977,\n",
       " 'physically': 978,\n",
       " 'minutes': 979,\n",
       " 'least': 980,\n",
       " 'last': 981,\n",
       " 'seven': 982,\n",
       " 'exports': 983,\n",
       " 'impact': 984,\n",
       " 'non-mining': 985,\n",
       " 'mining': 986,\n",
       " 'goods': 987,\n",
       " 'summarized': 988,\n",
       " 'crisis': 989,\n",
       " 'diverted': 990,\n",
       " 'mcot': 991,\n",
       " 'seen': 992,\n",
       " 'mobile': 993,\n",
       " 'outreach': 994,\n",
       " 'teams': 995,\n",
       " '(mcot)': 996,\n",
       " 'er': 997,\n",
       " 'people': 998,\n",
       " 'annually': 999,\n",
       " ...}"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dictionary.token2id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['transportation']"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "big_mash.big_mash[500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating Topics Composition of Specific Dataset:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dataset with only one word to identify it:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['transportation']"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "big_mash.big_mash[500] #random topic with one word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The human eye would say the topic above is about electric vehicle charging stations in Austin, and more broadly green energy and transportation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.5096153846153855\t Topic: 0.165*\"transportation\" + 0.038*\"traffic\" + 0.030*\"street\" + 0.026*\"parking\" + 0.024*\"infrastructure\" + 0.024*\"city\" + 0.024*\"safe\" + 0.023*\"vehicle\" + 0.022*\"streets\" + 0.016*\"bike\" + 0.013*\"vehicles\" + 0.012*\"road\" + 0.012*\"motor\" + 0.010*\"bicycle\" + 0.010*\"routes\" \n",
      "\n"
     ]
    }
   ],
   "source": [
    "#cribbed from here: http://nbviewer.jupyter.org/gist/boskaiolo/cc3e1341f59bfbd02726 \n",
    "for index, score in sorted(lda_52_sixty[corpus[500]], key=lambda tup: -1*tup[1]): #500th document\n",
    "    print(\"Score: {}\\t Topic: {} \\n\".format(score, lda_52_sixty.print_topic(index, 15))) #15 word topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dataset with many words to identify it**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['rating',\n",
       " 'customer',\n",
       " 'animal',\n",
       " 'plano',\n",
       " 'environment',\n",
       " 'social',\n",
       " 'services',\n",
       " 'government',\n",
       " 'Data',\n",
       " 'on',\n",
       " 'the',\n",
       " 'City',\n",
       " 'of',\n",
       " 'Plano',\n",
       " 'Animal',\n",
       " \"Service's\",\n",
       " 'Department',\n",
       " 'This',\n",
       " 'data',\n",
       " 'set',\n",
       " 'contains',\n",
       " 'information',\n",
       " 'on',\n",
       " 'the',\n",
       " \"Department's\",\n",
       " 'customer',\n",
       " 'feedback',\n",
       " 'ratings',\n",
       " 'These',\n",
       " 'scores',\n",
       " 'are',\n",
       " 'tabulated',\n",
       " 'once',\n",
       " 'a',\n",
       " 'quarter']"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "big_mash.big_mash[600]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.248182024694236\t Topic: 0.275*\"services\" + 0.096*\"social\" + 0.054*\"human\" + 0.053*\"dfps\" + 0.035*\"information\" + 0.033*\"programs\" + 0.030*\"us\" + 0.023*\"definition\" + 0.022*\"visit\" + 0.020*\"state\" + 0.015*\"animal\" + 0.014*\"support\" + 0.013*\"environment\" + 0.012*\"agenda\" + 0.010*\"agency\" \n",
      "\n",
      "Score: 0.16746794871794857\t Topic: 0.061*\"official\" + 0.050*\"account\" + 0.048*\"accounts\" + 0.041*\"(openmichigan@michigan\" + 0.029*\"special\" + 0.029*\"nursing\" + 0.027*\"software\" + 0.024*\"gis\" + 0.023*\"use\" + 0.022*\"consumer\" + 0.017*\"esri\" + 0.016*\"required\" + 0.016*\"portal\" + 0.015*\"chicago\" + 0.014*\"via\" \n",
      "\n",
      "Score: 0.14285535723202014\t Topic: 0.131*\"property\" + 0.052*\"consumption\" + 0.046*\"real\" + 0.031*\"value\" + 0.025*\"properties\" + 0.022*\"significant\" + 0.022*\"easy\" + 0.020*\"usage\" + 0.020*\"assessment\" + 0.015*\"company\" + 0.015*\"percentages\" + 0.015*\"city\" + 0.014*\"estate\" + 0.014*\"values\" + 0.012*\"government\" \n",
      "\n",
      "Score: 0.10521562143486674\t Topic: 0.087*\"service\" + 0.035*\"requests\" + 0.033*\"inspection\" + 0.031*\"inspections\" + 0.026*\"311\" + 0.026*\"public\" + 0.024*\"violations\" + 0.023*\"request\" + 0.019*\"complaints\" + 0.018*\"days\" + 0.017*\"certificate\" + 0.017*\"food\" + 0.017*\"unknown\" + 0.015*\"violent\" + 0.014*\"violation\" \n",
      "\n",
      "Score: 0.08261739835533997\t Topic: 0.046*\"information\" + 0.025*\"tab\" + 0.024*\"projects\" + 0.018*\"concerning\" + 0.018*\"may\" + 0.017*\"contains\" + 0.016*\"project\" + 0.014*\"provided\" + 0.014*\"additional\" + 0.012*\"improvement\" + 0.010*\"capital\" + 0.009*\"totals\" + 0.009*\"available\" + 0.009*\"release\" + 0.008*\"due\" \n",
      "\n",
      "Score: 0.07754845818118991\t Topic: 0.092*\"measures\" + 0.078*\"quality\" + 0.069*\"performance\" + 0.050*\"government\" + 0.026*\"life\" + 0.025*\"developed\" + 0.016*\"fair\" + 0.015*\"measurement\" + 0.014*\"definitions\" + 0.013*\"index\" + 0.012*\"key\" + 0.011*\"hospitalizations\" + 0.011*\"current\" + 0.011*\"livable\" + 0.010*\"countystat_measures\" \n",
      "\n",
      "Score: 0.07654180977753632\t Topic: 0.082*\"infrastructure\" + 0.066*\"city\" + 0.033*\"public\" + 0.031*\"maps\" + 0.028*\"works\" + 0.022*\"dallas\" + 0.020*\"los\" + 0.019*\"angeles\" + 0.015*\"types\" + 0.014*\"way\" + 0.013*\"new\" + 0.013*\"well\" + 0.013*\"right\" + 0.012*\"factor\" + 0.012*\"self\" \n",
      "\n",
      "Score: 0.06431497135045122\t Topic: 0.165*\"finance\" + 0.062*\"year\" + 0.053*\"state\" + 0.051*\"monthly\" + 0.048*\"fiscal\" + 0.036*\"payments\" + 0.026*\"government\" + 0.023*\"june\" + 0.022*\"july\" + 0.022*\"report\" + 0.017*\"general\" + 0.015*\"fy\" + 0.014*\"spending\" + 0.014*\"authority\" + 0.013*\"total\" \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for index, score in sorted(lda_52_sixty[corpus[600]], key=lambda tup: -1*tup[1]): #600th document\n",
    "    print(\"Score: {}\\t Topic: {} \\n\".format(score, lda_52_sixty.print_topic(index, 15))) #15 word topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Very interesting (and promising, for our model's purposes):**\n",
    "\n",
    "As the human eye can tell, this dataset is customer ratings about the City of Plano's animal department. That is....highly specific. And yet our model's top topic for it, in terms of composition, is a catch-all human and social services category, which includes an animal tag!\n",
    "\n",
    "\n",
    "** *HOWEVER, we can see some potential issues:* **\n",
    "\n",
    "A property value topic also shows some affinity with this (I bet this topic, which has \"assessment\" in the context of property values, also contains words like \"feedback\" and \"ratings\" deeper in the tag. It's important to remember that these topics go deeper than the first 10 words that we've been displaying. Words that are less likely to appear in the topic \"count\" for less in defining that topic, but they're still there. More simply, a fiscal year topic also shows some affinity, probably due to the dataset having \"government\" repeated several times. **This shows the importance of an affinity cut-off (perhaps .2 or above) or only giving the topic with the most affinity credit when it comes to calculating popularity.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Also To Note:\n",
    "- Our topics are sparse in terms of probability/composition. As mentioned above, they look very good the human eye, and can be useful, but remember that they go on much longer than 10 words, and that your top 10 words only compose like 10-15% of the topic (very approximate)\n",
    "- **A significant weakness** of this model is that all topics formed are the same size; there is no way to make \"clusters\" (topics) of varying density. But in real life, we know we have very specific topics that really only make sense with 3-4 words, whereas broad categories (public safety) can make sense with a ton of words\n",
    "- Normally, another LDA weakness is that it is \"bag of words\"; it doesn't take into account the placement of words in a sentence. However, since we are mainly using tags (\"public safety\", \"health\", etc), that's not much of an issue here!\n",
    "- This model allows for words to be re-used in topics. This is good for our purposes - e.g. \"public safety\" and \"public records\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where are there errors and uncertainty?\n",
    "**Again, not every dataset is tagged thoroughy, accurately, or appropriately.** \n",
    "  - Some cities/portals just give their datasets weird names or use stock descriptions for every single category of open data.\n",
    "    \n",
    "**Proper names obviously skew results somewhat; a proper name doesn't really tell us about the content of a dataset.**\n",
    "  - However, it's just not feasible to remove every proper name as a stopword; at least not without extensive trial and error\n",
    "  - This effect is mitigated by using topic composition for calculating the percentage of views that a topic gets. \"Maryland\" is in a community capital projects group, but a Maryland police department mash row will only match around .05 (very approximate)- so, the erroneous boost in views is neglible.\n",
    "  \n",
    "**The human brain interprets our clusters at the end. It's up to us to make up \"category\" or \"topic\" names. Two LDA topics may be very similar IRL topics (we can see this with taxes/public assistance - probably because that's such a prevalent IRL category).** \n",
    "  - Then again, all of this is labeled by humans. Back to our first point, there is always room for disagreement/debate in what \"subject\" a dataset is about, and how narrow to make subjects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculate Popularity of Each Topic Tag:\n",
    "** We have 27k+ datasets, each with their own \"mash\". We also have 42 topics. Each mash is composed of X% of a handful of topics (usually 3-6). We are going to give each topic proportional credit for a given dataset's popularity. So, if topic 38 composes 30% of a dataset's mash, it get's 30% of that dataset's quantitative value.**\n",
    "- However, since an LDA model is probabilistic, it by nature finds topics to be 5-10% of a \"document\" (our mash). These topics are very marginal at best. So if a topic doesn't account for at least 10% of a document, it doesn't get any points.\n",
    "    \n",
    "** We also need to devise a metric for a dataset's popularity. It will be a weighted combination of that dataset's views and downloads. **\n",
    "- There's no way to make this statistically exact, really. We want to give downloads more credit than just combining their raw totals with views, as a download indicates a dataset is more useful and utilized (what if people are just viewing a dataset, but do little with it?).\n",
    "    \n",
    "*The takeaway from all this is that our final metrics will be best read as proportional, not absolute comparisons.* \n",
    "- We are losing some interpretability; we certainly won't be able to take a proposed dataset and predict the views/downloads it will get. However, we will be able to say that certain categories are more broadly popular than others. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "      <th>attribution</th>\n",
       "      <th>columns_field_name</th>\n",
       "      <th>columns_name</th>\n",
       "      <th>type</th>\n",
       "      <th>categories</th>\n",
       "      <th>domain_category</th>\n",
       "      <th>domain_tags</th>\n",
       "      <th>provenance</th>\n",
       "      <th>download_count</th>\n",
       "      <th>page_views_last_month</th>\n",
       "      <th>page_views_last_week</th>\n",
       "      <th>page_views_total</th>\n",
       "      <th>page_views_total_log</th>\n",
       "      <th>domain</th>\n",
       "      <th>mash</th>\n",
       "      <th>big_mash</th>\n",
       "      <th>topic_comp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Homelessness PIT Transitional Age Youth</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[location_on_the_night_of_the_count_total_pers...</td>\n",
       "      <td>[Location on the night of the count, Total Per...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[point in time, homelessness, ss, dchs]</td>\n",
       "      <td>official</td>\n",
       "      <td>4.000</td>\n",
       "      <td>46.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>319.000</td>\n",
       "      <td>8.322</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[point, in, time, homelessness, ss, dchs]</td>\n",
       "      <td>[point, in, time, homelessness, ss, dchs]</td>\n",
       "      <td>[(49, 0.836538461538)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fair Housing Complaints</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[violations, percent_found_to_be_compliant, si...</td>\n",
       "      <td>[Number of complaints, Percent of sites found ...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[housing]</td>\n",
       "      <td>official</td>\n",
       "      <td>21.000</td>\n",
       "      <td>31.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>278.000</td>\n",
       "      <td>8.124</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[housing]</td>\n",
       "      <td>[housing]</td>\n",
       "      <td>[(49, 0.509615384615)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Parking Complaints Bar Chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[percent_of_valid_parking_meter_problem_servic...</td>\n",
       "      <td>[Percent of valid parking meter problem servic...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[transportation]</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>17.000</td>\n",
       "      <td>17.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>274.000</td>\n",
       "      <td>8.103</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[transportation]</td>\n",
       "      <td>[transportation]</td>\n",
       "      <td>[(3, 0.509615384615)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NVMHI Admissions</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[lipos_admissions_per_100k, nvmhi_admission_pe...</td>\n",
       "      <td>[LIPOS Admissions per 100K, NVMHI Admission pe...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[delete, dchs]</td>\n",
       "      <td>official</td>\n",
       "      <td>5.000</td>\n",
       "      <td>38.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>263.000</td>\n",
       "      <td>8.044</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[delete, dchs]</td>\n",
       "      <td>[delete, dchs]</td>\n",
       "      <td>[(16, 0.339743589744), (26, 0.339743589744)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Property Owners Trainined</td>\n",
       "      <td>[Office, of, Housing, Data]</td>\n",
       "      <td>None</td>\n",
       "      <td>[number_of_property_owners_trained, percent_of...</td>\n",
       "      <td>[Number of property owners trained, Percent of...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[housing]</td>\n",
       "      <td>official</td>\n",
       "      <td>11.000</td>\n",
       "      <td>30.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>246.000</td>\n",
       "      <td>7.948</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[housing]</td>\n",
       "      <td>[housing, Office, of, Housing, Data]</td>\n",
       "      <td>[(19, 0.254807692308), (49, 0.504807692308)]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      name                  description  \\\n",
       "0  Homelessness PIT Transitional Age Youth                           []   \n",
       "1                  Fair Housing Complaints                           []   \n",
       "2             Parking Complaints Bar Chart                           []   \n",
       "3                         NVMHI Admissions                           []   \n",
       "4                Property Owners Trainined  [Office, of, Housing, Data]   \n",
       "\n",
       "  attribution                                 columns_field_name  \\\n",
       "0        None  [location_on_the_night_of_the_count_total_pers...   \n",
       "1        None  [violations, percent_found_to_be_compliant, si...   \n",
       "2        None  [percent_of_valid_parking_meter_problem_servic...   \n",
       "3        None  [lipos_admissions_per_100k, nvmhi_admission_pe...   \n",
       "4        None  [number_of_property_owners_trained, percent_of...   \n",
       "\n",
       "                                        columns_name   type        categories  \\\n",
       "0  [Location on the night of the count, Total Per...  chart                []   \n",
       "1  [Number of complaints, Percent of sites found ...  chart                []   \n",
       "2  [Percent of valid parking meter problem servic...  chart  [transportation]   \n",
       "3  [LIPOS Admissions per 100K, NVMHI Admission pe...  chart                []   \n",
       "4  [Number of property owners trained, Percent of...  chart                []   \n",
       "\n",
       "  domain_category                              domain_tags provenance  \\\n",
       "0                  [point in time, homelessness, ss, dchs]   official   \n",
       "1                                                [housing]   official   \n",
       "2                                                       []   official   \n",
       "3                                           [delete, dchs]   official   \n",
       "4                                                [housing]   official   \n",
       "\n",
       "   download_count  page_views_last_month  page_views_last_week  \\\n",
       "0           4.000                 46.000                 5.000   \n",
       "1          21.000                 31.000                 1.000   \n",
       "2          17.000                 17.000                 4.000   \n",
       "3           5.000                 38.000                 5.000   \n",
       "4          11.000                 30.000                 4.000   \n",
       "\n",
       "   page_views_total  page_views_total_log                      domain  \\\n",
       "0           319.000                 8.322  dashboard.alexandriava.gov   \n",
       "1           278.000                 8.124  dashboard.alexandriava.gov   \n",
       "2           274.000                 8.103  dashboard.alexandriava.gov   \n",
       "3           263.000                 8.044  dashboard.alexandriava.gov   \n",
       "4           246.000                 7.948  dashboard.alexandriava.gov   \n",
       "\n",
       "                                        mash  \\\n",
       "0  [point, in, time, homelessness, ss, dchs]   \n",
       "1                                  [housing]   \n",
       "2                           [transportation]   \n",
       "3                             [delete, dchs]   \n",
       "4                                  [housing]   \n",
       "\n",
       "                                    big_mash  \\\n",
       "0  [point, in, time, homelessness, ss, dchs]   \n",
       "1                                  [housing]   \n",
       "2                           [transportation]   \n",
       "3                             [delete, dchs]   \n",
       "4       [housing, Office, of, Housing, Data]   \n",
       "\n",
       "                                     topic_comp  \n",
       "0                        [(49, 0.836538461538)]  \n",
       "1                        [(49, 0.509615384615)]  \n",
       "2                         [(3, 0.509615384615)]  \n",
       "3  [(16, 0.339743589744), (26, 0.339743589744)]  \n",
       "4  [(19, 0.254807692308), (49, 0.504807692308)]  "
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats = big_mash.copy()\n",
    "stats = stats.assign(topic_comp = corpus_lda_list)\n",
    "stats.head() #the topic_comp column are actual Python lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One way of scoring - only when a topic composes more than .2 of a doc does it get \"credit\" for its share of that doc's views."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def calculate_topic_pop(df):\n",
    "    results_dict = {}\n",
    "    \n",
    "    for row_num in df.index:\n",
    "        for tup in df.topic_comp[row_num]:\n",
    "            if not tup[0] in results_dict:\n",
    "                if tup[1] >= 0.1:\n",
    "                    results_dict[tup[0]] = (tup[1] * (df.iloc[row_num].download_count + \n",
    "                                             df.iloc[row_num].page_views_total_log))\n",
    "                else:\n",
    "                    pass\n",
    "            if tup[0] in results_dict:\n",
    "                if tup[1] >= 0.1:\n",
    "                    results_dict[tup[0]] += (tup[1] * (df.iloc[row_num].download_count + \n",
    "                                             df.iloc[row_num].page_views_total_log))\n",
    "    return results_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ok, let's make sure this function works and is calculating accurately:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "small_tester = stats.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d = calculate_topic_pop(small_tester)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{3: 25.586043343189221,\n",
       " 16: 8.8634985682820346,\n",
       " 19: 9.6563794545575803,\n",
       " 26: 8.8634985682820346,\n",
       " 49: 45.022915362457141}"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Topic 49 leads the pack in our metrics. We can see from our LDA model that it is the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('development', 0.17602775675090593),\n",
       " ('housing', 0.15221265764931263),\n",
       " ('economic', 0.058854109443175158),\n",
       " ('community', 0.04859194305354804),\n",
       " ('infrastructure', 0.019040125293198554),\n",
       " ('managed', 0.018304696035360443),\n",
       " ('medicaid', 0.017876406351462711),\n",
       " ('department', 0.014023829088608938),\n",
       " ('buildings', 0.013775405992175799),\n",
       " ('economy', 0.012347394313369113)]"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_52_sixty.show_topic(49)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "      <th>attribution</th>\n",
       "      <th>columns_field_name</th>\n",
       "      <th>columns_name</th>\n",
       "      <th>type</th>\n",
       "      <th>categories</th>\n",
       "      <th>domain_category</th>\n",
       "      <th>domain_tags</th>\n",
       "      <th>provenance</th>\n",
       "      <th>download_count</th>\n",
       "      <th>page_views_last_month</th>\n",
       "      <th>page_views_last_week</th>\n",
       "      <th>page_views_total</th>\n",
       "      <th>page_views_total_log</th>\n",
       "      <th>domain</th>\n",
       "      <th>mash</th>\n",
       "      <th>big_mash</th>\n",
       "      <th>topic_comp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Homelessness PIT Transitional Age Youth</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[location_on_the_night_of_the_count_total_pers...</td>\n",
       "      <td>[Location on the night of the count, Total Per...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[point in time, homelessness, ss, dchs]</td>\n",
       "      <td>official</td>\n",
       "      <td>4.000</td>\n",
       "      <td>46.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>319.000</td>\n",
       "      <td>8.322</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[point, in, time, homelessness, ss, dchs]</td>\n",
       "      <td>[point, in, time, homelessness, ss, dchs]</td>\n",
       "      <td>[(49, 0.836538461538)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fair Housing Complaints</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[violations, percent_found_to_be_compliant, si...</td>\n",
       "      <td>[Number of complaints, Percent of sites found ...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[housing]</td>\n",
       "      <td>official</td>\n",
       "      <td>21.000</td>\n",
       "      <td>31.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>278.000</td>\n",
       "      <td>8.124</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[housing]</td>\n",
       "      <td>[housing]</td>\n",
       "      <td>[(49, 0.509615384615)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Parking Complaints Bar Chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[percent_of_valid_parking_meter_problem_servic...</td>\n",
       "      <td>[Percent of valid parking meter problem servic...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[transportation]</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>17.000</td>\n",
       "      <td>17.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>274.000</td>\n",
       "      <td>8.103</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[transportation]</td>\n",
       "      <td>[transportation]</td>\n",
       "      <td>[(3, 0.509615384615)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NVMHI Admissions</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>[lipos_admissions_per_100k, nvmhi_admission_pe...</td>\n",
       "      <td>[LIPOS Admissions per 100K, NVMHI Admission pe...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[delete, dchs]</td>\n",
       "      <td>official</td>\n",
       "      <td>5.000</td>\n",
       "      <td>38.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>263.000</td>\n",
       "      <td>8.044</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[delete, dchs]</td>\n",
       "      <td>[delete, dchs]</td>\n",
       "      <td>[(16, 0.339743589744), (26, 0.339743589744)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Property Owners Trainined</td>\n",
       "      <td>[Office, of, Housing, Data]</td>\n",
       "      <td>None</td>\n",
       "      <td>[number_of_property_owners_trained, percent_of...</td>\n",
       "      <td>[Number of property owners trained, Percent of...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td></td>\n",
       "      <td>[housing]</td>\n",
       "      <td>official</td>\n",
       "      <td>11.000</td>\n",
       "      <td>30.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>246.000</td>\n",
       "      <td>7.948</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>[housing]</td>\n",
       "      <td>[housing, Office, of, Housing, Data]</td>\n",
       "      <td>[(19, 0.254807692308), (49, 0.504807692308)]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      name                  description  \\\n",
       "0  Homelessness PIT Transitional Age Youth                           []   \n",
       "1                  Fair Housing Complaints                           []   \n",
       "2             Parking Complaints Bar Chart                           []   \n",
       "3                         NVMHI Admissions                           []   \n",
       "4                Property Owners Trainined  [Office, of, Housing, Data]   \n",
       "\n",
       "  attribution                                 columns_field_name  \\\n",
       "0        None  [location_on_the_night_of_the_count_total_pers...   \n",
       "1        None  [violations, percent_found_to_be_compliant, si...   \n",
       "2        None  [percent_of_valid_parking_meter_problem_servic...   \n",
       "3        None  [lipos_admissions_per_100k, nvmhi_admission_pe...   \n",
       "4        None  [number_of_property_owners_trained, percent_of...   \n",
       "\n",
       "                                        columns_name   type        categories  \\\n",
       "0  [Location on the night of the count, Total Per...  chart                []   \n",
       "1  [Number of complaints, Percent of sites found ...  chart                []   \n",
       "2  [Percent of valid parking meter problem servic...  chart  [transportation]   \n",
       "3  [LIPOS Admissions per 100K, NVMHI Admission pe...  chart                []   \n",
       "4  [Number of property owners trained, Percent of...  chart                []   \n",
       "\n",
       "  domain_category                              domain_tags provenance  \\\n",
       "0                  [point in time, homelessness, ss, dchs]   official   \n",
       "1                                                [housing]   official   \n",
       "2                                                       []   official   \n",
       "3                                           [delete, dchs]   official   \n",
       "4                                                [housing]   official   \n",
       "\n",
       "   download_count  page_views_last_month  page_views_last_week  \\\n",
       "0           4.000                 46.000                 5.000   \n",
       "1          21.000                 31.000                 1.000   \n",
       "2          17.000                 17.000                 4.000   \n",
       "3           5.000                 38.000                 5.000   \n",
       "4          11.000                 30.000                 4.000   \n",
       "\n",
       "   page_views_total  page_views_total_log                      domain  \\\n",
       "0           319.000                 8.322  dashboard.alexandriava.gov   \n",
       "1           278.000                 8.124  dashboard.alexandriava.gov   \n",
       "2           274.000                 8.103  dashboard.alexandriava.gov   \n",
       "3           263.000                 8.044  dashboard.alexandriava.gov   \n",
       "4           246.000                 7.948  dashboard.alexandriava.gov   \n",
       "\n",
       "                                        mash  \\\n",
       "0  [point, in, time, homelessness, ss, dchs]   \n",
       "1                                  [housing]   \n",
       "2                           [transportation]   \n",
       "3                             [delete, dchs]   \n",
       "4                                  [housing]   \n",
       "\n",
       "                                    big_mash  \\\n",
       "0  [point, in, time, homelessness, ss, dchs]   \n",
       "1                                  [housing]   \n",
       "2                           [transportation]   \n",
       "3                             [delete, dchs]   \n",
       "4       [housing, Office, of, Housing, Data]   \n",
       "\n",
       "                                     topic_comp  \n",
       "0                        [(49, 0.836538461538)]  \n",
       "1                        [(49, 0.509615384615)]  \n",
       "2                         [(3, 0.509615384615)]  \n",
       "3  [(16, 0.339743589744), (26, 0.339743589744)]  \n",
       "4  [(19, 0.254807692308), (49, 0.504807692308)]  "
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "small_tester"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The human eye indicates that this sample is mainly about housing/property and community development. Two of our datasets are explicitly about housing, and a third is closely related (youth homelessness). Most of these datasets have similar log_views and downloads. **This is very much an eyeball test an an extremely limited slice of our df, but it looks right so far**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculate Stats for Full 27k Row DF:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "to_count = stats.copy()\n",
    "to_count = to_count.fillna(0) #have to fill NaNs for downloads, log views etc with 0 or we get all NaNs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "topic_stats = calculate_topic_pop(to_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#topic_stats #uncomment to view all stats in a dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pop this into a dataframe and add our topics' top 10 words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "topic_pop = pd.DataFrame.from_dict(topic_stats, orient='index')\n",
    "topic_pop = topic_pop.rename(index=str, columns={0:\"Adjusted_Popularity\"}) #rename column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_tuple_lists = list(lda_52_sixty.show_topics(num_topics=52, formatted=False))\n",
    "topics_list = []\n",
    "\n",
    "for tup in topic_tuple_lists:\n",
    "    topics_list.append(tup[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Adjusted_Popularity</th>\n",
       "      <th>Topic</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>92505937.889</td>\n",
       "      <td>[(health, 0.305311895849), (san, 0.06110177203...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>866205.286</td>\n",
       "      <td>[(water, 0.0961247088917), (environment, 0.048...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>599549.333</td>\n",
       "      <td>[(financial, 0.0959611006916), (permits, 0.066...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>559885.634</td>\n",
       "      <td>[(chart, 0.0699636595856), (children, 0.059274...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>471289.932</td>\n",
       "      <td>[(health, 0.0945969708127), (disease, 0.043755...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>406099.043</td>\n",
       "      <td>[(energy, 0.123710849454), (environment, 0.090...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>355623.123</td>\n",
       "      <td>[(finance, 0.164742098166), (year, 0.062366577...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>346728.373</td>\n",
       "      <td>[(transportation, 0.0493422015093), (plans, 0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>344660.582</td>\n",
       "      <td>[(transportation, 0.164542015423), (traffic, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>340707.468</td>\n",
       "      <td>[(state, 0.175638296746), (new, 0.133403517998...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Adjusted_Popularity                                              Topic\n",
       "28         92505937.889  [(health, 0.305311895849), (san, 0.06110177203...\n",
       "41           866205.286  [(water, 0.0961247088917), (environment, 0.048...\n",
       "23           599549.333  [(financial, 0.0959611006916), (permits, 0.066...\n",
       "3            559885.634  [(chart, 0.0699636595856), (children, 0.059274...\n",
       "21           471289.932  [(health, 0.0945969708127), (disease, 0.043755...\n",
       "39           406099.043  [(energy, 0.123710849454), (environment, 0.090...\n",
       "31           355623.123  [(finance, 0.164742098166), (year, 0.062366577...\n",
       "48           346728.373  [(transportation, 0.0493422015093), (plans, 0....\n",
       "26           344660.582  [(transportation, 0.164542015423), (traffic, 0...\n",
       "43           340707.468  [(state, 0.175638296746), (new, 0.133403517998..."
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_pop = topic_pop.assign(Topic = topics_list)\n",
    "topic_pop = topic_pop.sort_values(by='Adjusted_Popularity', ascending=False)\n",
    "\n",
    "topic_pop.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "topics = topic_pop.copy()\n",
    "topics[[\"topic1\", \"topic2\", \"topic3\", \"topic4\", \n",
    "           \"topic5\", \"topic6\", \"topic7\", \"topic8\", \"topic9\", \"topic10\"]] = topics.Topic.apply(pd.Series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Adjusted_Popularity</th>\n",
       "      <th>Topic</th>\n",
       "      <th>topic1</th>\n",
       "      <th>topic2</th>\n",
       "      <th>topic3</th>\n",
       "      <th>topic4</th>\n",
       "      <th>topic5</th>\n",
       "      <th>topic6</th>\n",
       "      <th>topic7</th>\n",
       "      <th>topic8</th>\n",
       "      <th>topic9</th>\n",
       "      <th>topic10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>92505937.889</td>\n",
       "      <td>[(health, 0.305311895849), (san, 0.06110177203...</td>\n",
       "      <td>(health, 0.305311895849)</td>\n",
       "      <td>(san, 0.0611017720363)</td>\n",
       "      <td>(county, 0.0586392706518)</td>\n",
       "      <td>(care, 0.0280055059751)</td>\n",
       "      <td>(counties, 0.0199253952569)</td>\n",
       "      <td>(system, 0.0163770295818)</td>\n",
       "      <td>(facilities, 0.0140076033687)</td>\n",
       "      <td>(network, 0.0135614573066)</td>\n",
       "      <td>(home, 0.0112061646845)</td>\n",
       "      <td>(francisco, 0.0103104720032)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>866205.286</td>\n",
       "      <td>[(water, 0.0961247088917), (environment, 0.048...</td>\n",
       "      <td>(water, 0.0961247088917)</td>\n",
       "      <td>(environment, 0.0481019608081)</td>\n",
       "      <td>(protection, 0.0206283827247)</td>\n",
       "      <td>(tx, 0.0202036902737)</td>\n",
       "      <td>(filed, 0.0195698643711)</td>\n",
       "      <td>(waste, 0.0192716246599)</td>\n",
       "      <td>(site, 0.0192005256969)</td>\n",
       "      <td>(quality, 0.0185576020412)</td>\n",
       "      <td>(environmental, 0.0173950954393)</td>\n",
       "      <td>(monitoring, 0.0158884561679)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>599549.333</td>\n",
       "      <td>[(financial, 0.0959611006916), (permits, 0.066...</td>\n",
       "      <td>(financial, 0.0959611006916)</td>\n",
       "      <td>(permits, 0.0668632739602)</td>\n",
       "      <td>(building, 0.0631953900546)</td>\n",
       "      <td>(permit, 0.054456521697)</td>\n",
       "      <td>(expenditures, 0.0429247389481)</td>\n",
       "      <td>(guide, 0.0250580534259)</td>\n",
       "      <td>(issued, 0.0232434329642)</td>\n",
       "      <td>(construction, 0.0228846174877)</td>\n",
       "      <td>(information, 0.0227968624179)</td>\n",
       "      <td>(filter, 0.021954851646)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>559885.634</td>\n",
       "      <td>[(chart, 0.0699636595856), (children, 0.059274...</td>\n",
       "      <td>(chart, 0.0699636595856)</td>\n",
       "      <td>(children, 0.0592740047687)</td>\n",
       "      <td>(income, 0.039083994413)</td>\n",
       "      <td>(historic, 0.0250932288571)</td>\n",
       "      <td>(home, 0.0218525211874)</td>\n",
       "      <td>(assistance, 0.018091177706)</td>\n",
       "      <td>(homes, 0.017940795662)</td>\n",
       "      <td>(families, 0.0179183548402)</td>\n",
       "      <td>(low, 0.0178577144875)</td>\n",
       "      <td>(pay, 0.0176100642703)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>471289.932</td>\n",
       "      <td>[(health, 0.0945969708127), (disease, 0.043755...</td>\n",
       "      <td>(health, 0.0945969708127)</td>\n",
       "      <td>(disease, 0.0437556864768)</td>\n",
       "      <td>(000, 0.0390249577363)</td>\n",
       "      <td>(per, 0.0309178121906)</td>\n",
       "      <td>(rate, 0.0290345540868)</td>\n",
       "      <td>(rates, 0.0262263352287)</td>\n",
       "      <td>(chronic, 0.0250697947817)</td>\n",
       "      <td>(risk, 0.0223843182628)</td>\n",
       "      <td>(agency, 0.0216210266927)</td>\n",
       "      <td>(due, 0.0195463066794)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>406099.043</td>\n",
       "      <td>[(energy, 0.123710849454), (environment, 0.090...</td>\n",
       "      <td>(energy, 0.123710849454)</td>\n",
       "      <td>(environment, 0.0906900208835)</td>\n",
       "      <td>(electricity, 0.0275225911326)</td>\n",
       "      <td>(air, 0.0244810000559)</td>\n",
       "      <td>(sustainable, 0.0241394711405)</td>\n",
       "      <td>(action, 0.0236316969298)</td>\n",
       "      <td>(climate, 0.0231125236045)</td>\n",
       "      <td>(city, 0.0191503642982)</td>\n",
       "      <td>(clean, 0.0190989188479)</td>\n",
       "      <td>(facilities, 0.0184715188825)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>355623.123</td>\n",
       "      <td>[(finance, 0.164742098166), (year, 0.062366577...</td>\n",
       "      <td>(finance, 0.164742098166)</td>\n",
       "      <td>(year, 0.062366577351)</td>\n",
       "      <td>(state, 0.0533822539548)</td>\n",
       "      <td>(monthly, 0.0506962481995)</td>\n",
       "      <td>(fiscal, 0.0476222221357)</td>\n",
       "      <td>(payments, 0.0357199197181)</td>\n",
       "      <td>(government, 0.0264540232587)</td>\n",
       "      <td>(june, 0.0225770287471)</td>\n",
       "      <td>(july, 0.0216759115561)</td>\n",
       "      <td>(report, 0.0215900104843)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>346728.373</td>\n",
       "      <td>[(transportation, 0.0493422015093), (plans, 0....</td>\n",
       "      <td>(transportation, 0.0493422015093)</td>\n",
       "      <td>(plans, 0.0463408207807)</td>\n",
       "      <td>(iowa, 0.0372400214617)</td>\n",
       "      <td>(area, 0.031583902009)</td>\n",
       "      <td>(transit, 0.0297466527633)</td>\n",
       "      <td>(operations, 0.029650998681)</td>\n",
       "      <td>(region, 0.0286308933708)</td>\n",
       "      <td>(bus, 0.0239404049245)</td>\n",
       "      <td>(priority, 0.0197652573751)</td>\n",
       "      <td>(people, 0.0157596265395)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>344660.582</td>\n",
       "      <td>[(transportation, 0.164542015423), (traffic, 0...</td>\n",
       "      <td>(transportation, 0.164542015423)</td>\n",
       "      <td>(traffic, 0.0382494150922)</td>\n",
       "      <td>(street, 0.0297088546702)</td>\n",
       "      <td>(parking, 0.0260484479646)</td>\n",
       "      <td>(infrastructure, 0.024499661836)</td>\n",
       "      <td>(city, 0.0242484880067)</td>\n",
       "      <td>(safe, 0.0239084069945)</td>\n",
       "      <td>(vehicle, 0.0231351100665)</td>\n",
       "      <td>(streets, 0.0219852292388)</td>\n",
       "      <td>(bike, 0.0155102577616)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>340707.468</td>\n",
       "      <td>[(state, 0.175638296746), (new, 0.133403517998...</td>\n",
       "      <td>(state, 0.175638296746)</td>\n",
       "      <td>(new, 0.133403517998)</td>\n",
       "      <td>(michigan, 0.10266572561)</td>\n",
       "      <td>(york, 0.0777270773871)</td>\n",
       "      <td>(information, 0.0292838745522)</td>\n",
       "      <td>(check, 0.0203220904165)</td>\n",
       "      <td>(measurements, 0.0197080431536)</td>\n",
       "      <td>(jobs, 0.0170402721275)</td>\n",
       "      <td>(licensing, 0.0169768940674)</td>\n",
       "      <td>(ny, 0.0167776417369)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Adjusted_Popularity                                              Topic  \\\n",
       "28         92505937.889  [(health, 0.305311895849), (san, 0.06110177203...   \n",
       "41           866205.286  [(water, 0.0961247088917), (environment, 0.048...   \n",
       "23           599549.333  [(financial, 0.0959611006916), (permits, 0.066...   \n",
       "3            559885.634  [(chart, 0.0699636595856), (children, 0.059274...   \n",
       "21           471289.932  [(health, 0.0945969708127), (disease, 0.043755...   \n",
       "39           406099.043  [(energy, 0.123710849454), (environment, 0.090...   \n",
       "31           355623.123  [(finance, 0.164742098166), (year, 0.062366577...   \n",
       "48           346728.373  [(transportation, 0.0493422015093), (plans, 0....   \n",
       "26           344660.582  [(transportation, 0.164542015423), (traffic, 0...   \n",
       "43           340707.468  [(state, 0.175638296746), (new, 0.133403517998...   \n",
       "\n",
       "                               topic1                          topic2  \\\n",
       "28           (health, 0.305311895849)          (san, 0.0611017720363)   \n",
       "41           (water, 0.0961247088917)  (environment, 0.0481019608081)   \n",
       "23       (financial, 0.0959611006916)      (permits, 0.0668632739602)   \n",
       "3            (chart, 0.0699636595856)     (children, 0.0592740047687)   \n",
       "21          (health, 0.0945969708127)      (disease, 0.0437556864768)   \n",
       "39           (energy, 0.123710849454)  (environment, 0.0906900208835)   \n",
       "31          (finance, 0.164742098166)          (year, 0.062366577351)   \n",
       "48  (transportation, 0.0493422015093)        (plans, 0.0463408207807)   \n",
       "26   (transportation, 0.164542015423)      (traffic, 0.0382494150922)   \n",
       "43            (state, 0.175638296746)           (new, 0.133403517998)   \n",
       "\n",
       "                            topic3                       topic4  \\\n",
       "28       (county, 0.0586392706518)      (care, 0.0280055059751)   \n",
       "41   (protection, 0.0206283827247)        (tx, 0.0202036902737)   \n",
       "23     (building, 0.0631953900546)     (permit, 0.054456521697)   \n",
       "3         (income, 0.039083994413)  (historic, 0.0250932288571)   \n",
       "21          (000, 0.0390249577363)       (per, 0.0309178121906)   \n",
       "39  (electricity, 0.0275225911326)       (air, 0.0244810000559)   \n",
       "31        (state, 0.0533822539548)   (monthly, 0.0506962481995)   \n",
       "48         (iowa, 0.0372400214617)       (area, 0.031583902009)   \n",
       "26       (street, 0.0297088546702)   (parking, 0.0260484479646)   \n",
       "43       (michigan, 0.10266572561)      (york, 0.0777270773871)   \n",
       "\n",
       "                              topic5                        topic6  \\\n",
       "28       (counties, 0.0199253952569)     (system, 0.0163770295818)   \n",
       "41          (filed, 0.0195698643711)      (waste, 0.0192716246599)   \n",
       "23   (expenditures, 0.0429247389481)      (guide, 0.0250580534259)   \n",
       "3            (home, 0.0218525211874)  (assistance, 0.018091177706)   \n",
       "21           (rate, 0.0290345540868)      (rates, 0.0262263352287)   \n",
       "39    (sustainable, 0.0241394711405)     (action, 0.0236316969298)   \n",
       "31         (fiscal, 0.0476222221357)   (payments, 0.0357199197181)   \n",
       "48        (transit, 0.0297466527633)  (operations, 0.029650998681)   \n",
       "26  (infrastructure, 0.024499661836)       (city, 0.0242484880067)   \n",
       "43    (information, 0.0292838745522)      (check, 0.0203220904165)   \n",
       "\n",
       "                             topic7                           topic8  \\\n",
       "28    (facilities, 0.0140076033687)       (network, 0.0135614573066)   \n",
       "41          (site, 0.0192005256969)       (quality, 0.0185576020412)   \n",
       "23        (issued, 0.0232434329642)  (construction, 0.0228846174877)   \n",
       "3           (homes, 0.017940795662)      (families, 0.0179183548402)   \n",
       "21       (chronic, 0.0250697947817)          (risk, 0.0223843182628)   \n",
       "39       (climate, 0.0231125236045)          (city, 0.0191503642982)   \n",
       "31    (government, 0.0264540232587)          (june, 0.0225770287471)   \n",
       "48        (region, 0.0286308933708)           (bus, 0.0239404049245)   \n",
       "26          (safe, 0.0239084069945)       (vehicle, 0.0231351100665)   \n",
       "43  (measurements, 0.0197080431536)          (jobs, 0.0170402721275)   \n",
       "\n",
       "                              topic9                        topic10  \n",
       "28           (home, 0.0112061646845)   (francisco, 0.0103104720032)  \n",
       "41  (environmental, 0.0173950954393)  (monitoring, 0.0158884561679)  \n",
       "23    (information, 0.0227968624179)       (filter, 0.021954851646)  \n",
       "3             (low, 0.0178577144875)         (pay, 0.0176100642703)  \n",
       "21         (agency, 0.0216210266927)         (due, 0.0195463066794)  \n",
       "39          (clean, 0.0190989188479)  (facilities, 0.0184715188825)  \n",
       "31           (july, 0.0216759115561)      (report, 0.0215900104843)  \n",
       "48       (priority, 0.0197652573751)      (people, 0.0157596265395)  \n",
       "26        (streets, 0.0219852292388)        (bike, 0.0155102577616)  \n",
       "43      (licensing, 0.0169768940674)          (ny, 0.0167776417369)  "
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just the most superficial and speculative of conclusions looking at this but... it seems like datasets about health care facilities are astoundingly popular. Environment and water quality and permitting are a distant second. There is ANOTHER health dataset -- clearly about public health and disease rates -- in the top 10. And we can clearly tell that two types of transportation data sets -- one that's more planning-oriented, another that seems more focused on traffic, parking, and bikes -- also show up prominently."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recalculate based on different metrics - either full credit, or only credit for above a .1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ANOVA test of statistical significance:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Different Metrics for Popularity\n",
    "- Take into account views last week/month"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Display titles of strongest document matches for each \"cluster\" (topic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}