Ad Impression Fraud Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detecting Fraud in Ad Impression Data\n",
    "\n",
    "I have a file with one day of impression logs for selected IP addresses. The nine fields\n",
    "are, in order:\n",
    "\n",
    "* Timestamp\n",
    "* IP address\n",
    "* Detected browser type\n",
    "* User agent string\n",
    "* Host (URL)\n",
    "* Whether the impression was in view (1.0 = yes, 0.0 = no)\n",
    "* Number of plugins installed\n",
    "* Browser window position and size (x, y, width, height)\n",
    "* Network latency\n",
    "\n",
    "My task is to identify hosts which are receiving a substantial amount of fraudulent\n",
    "traffic. As part of this, I may also wish to identify IP addresses home to machines that\n",
    "are part of botnets, but this is not required. The definition of \"substantial\" is up to me --\n",
    "this may be a ranked list of all hosts, or a list of hosts reaching a certain threshold, or I\n",
    "may choose not to quantify the amount of fraud and simply classify hosts as likely to be\n",
    "experiencing high fraud or not. To get started, I have a list of hosts which are known\n",
    "to receive substantial amounts of fraudulent traffic:\n",
    "\n",
    "* featureball.com\n",
    "* uvido.com\n",
    "* sprungliving.com\n",
    "* sweetboxgames.com\n",
    "* mammabay.co.uk\n",
    "* workingfathertv.com\n",
    "* worsthorrorgame.com\n",
    "* hourlyparent.com\n",
    "* ulterior-movies.com\n",
    "* myhomesdesign.com\n",
    "* indoorlife.tv\n",
    "* bumclub.info\n",
    "* psychoworld.tv\n",
    "* hunp.us\n",
    "* rlinevideos.com\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numpy 1.10.4\n",
      "pandas 0.18.1\n",
      "seaborn 0.7.0\n",
      "Seconds to load csv:  0.547770252733\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Timestamp</th>\n",
       "      <th>IPadd</th>\n",
       "      <th>Browser</th>\n",
       "      <th>UserAgent</th>\n",
       "      <th>Host</th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Wpossize</th>\n",
       "      <th>Latency</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>393.414.443.469</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...</td>\n",
       "      <td>http://www.domain.com.au</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0,0,1280,629)</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>393.414.443.469</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...</td>\n",
       "      <td>http://www.domain.com.au</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0,0,1280,629)</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>http://www.mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>http://www.mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>http://www.mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Timestamp            IPadd        Browser  \\\n",
       "0  2014-08-25 00:00:00  393.414.443.469  Safari/Webkit   \n",
       "1  2014-08-25 00:00:00  393.414.443.469  Safari/Webkit   \n",
       "2  2014-08-25 00:00:00  325.441.386.395        Unknown   \n",
       "3  2014-08-25 00:00:00  325.441.386.395        Unknown   \n",
       "4  2014-08-25 00:00:00  325.441.386.395        Unknown   \n",
       "\n",
       "                                           UserAgent  \\\n",
       "0  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...   \n",
       "1  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...   \n",
       "2  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...   \n",
       "3  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...   \n",
       "4  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...   \n",
       "\n",
       "                         Host  Inview  Nplugins        Wpossize  Latency  \n",
       "0    http://www.domain.com.au     0.0       NaN  (0,0,1280,629)      0.0  \n",
       "1    http://www.domain.com.au     0.0       NaN  (0,0,1280,629)      0.0  \n",
       "2  http://www.mangareader.net     NaN       NaN             NaN      NaN  \n",
       "3  http://www.mangareader.net     NaN       NaN             NaN      NaN  \n",
       "4  http://www.mangareader.net     NaN       NaN             NaN      NaN  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "import math\n",
    "import timeit\n",
    "import sys\n",
    "sys.path.insert(0, './useragent-0.1.1')\n",
    "import useragent\n",
    "import numpy as np\n",
    "print 'numpy', np.__version__\n",
    "import pandas as pd\n",
    "print 'pandas', pd.__version__\n",
    "import seaborn as sns\n",
    "print 'seaborn', sns.__version__\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('ggplot')\n",
    "%matplotlib inline\n",
    "\n",
    "pd.options.display.max_columns = 25\n",
    "\n",
    "tic = timeit.default_timer()\n",
    "imp_df = pd.io.parsers.read_csv(\"ad_impression_data_set.tsv\",sep='\\t',\n",
    "                                names=['Timestamp','IPadd','Browser','UserAgent','Host','Inview','Nplugins','Wpossize','Latency'],\n",
    "                               header=None)\n",
    "toc = timeit.default_timer()\n",
    "print 'Seconds to load csv: ', toc - tic\n",
    "\n",
    "imp_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              Inview       Nplugins       Latency\n",
      "count  211811.000000  162304.000000  2.002830e+05\n",
      "mean        0.431998      10.139442  3.452002e+02\n",
      "std         0.495355       7.637172  1.329282e+04\n",
      "min         0.000000       1.000000  0.000000e+00\n",
      "25%              NaN            NaN           NaN\n",
      "50%              NaN            NaN           NaN\n",
      "75%              NaN            NaN           NaN\n",
      "max         1.000000      61.000000  5.866142e+06\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda2\\lib\\site-packages\\numpy\\lib\\function_base.py:3403: RuntimeWarning: Invalid value encountered in median\n",
      "  RuntimeWarning)\n"
     ]
    }
   ],
   "source": [
    "print imp_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 235083 entries, 0 to 235082\n",
      "Data columns (total 9 columns):\n",
      "Timestamp    235083 non-null object\n",
      "IPadd        235083 non-null object\n",
      "Browser      235083 non-null object\n",
      "UserAgent    235083 non-null object\n",
      "Host         235083 non-null object\n",
      "Inview       211811 non-null float64\n",
      "Nplugins     162304 non-null float64\n",
      "Wpossize     118451 non-null object\n",
      "Latency      200283 non-null float64\n",
      "dtypes: float64(3), object(6)\n",
      "memory usage: 16.1+ MB\n",
      "None\n",
      "0      http://www.domain.com.au\n",
      "1      http://www.domain.com.au\n",
      "2    http://www.mangareader.net\n",
      "3    http://www.mangareader.net\n",
      "4    http://www.mangareader.net\n",
      "Name: Host, dtype: object\n",
      "\n",
      "Number of ERROR hosts to be removed: 138 \n",
      "\n",
      "Seconds to clean hosts:  2.22865650226\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 234945 entries, 0 to 235082\n",
      "Data columns (total 9 columns):\n",
      "Timestamp    234945 non-null object\n",
      "IPadd        234945 non-null object\n",
      "Browser      234945 non-null object\n",
      "UserAgent    234945 non-null object\n",
      "Host         234945 non-null object\n",
      "Inview       211810 non-null float64\n",
      "Nplugins     162303 non-null float64\n",
      "Wpossize     118450 non-null object\n",
      "Latency      200279 non-null float64\n",
      "dtypes: float64(3), object(6)\n",
      "memory usage: 17.9+ MB\n",
      "None\n",
      "0       domain.comau\n",
      "1       domain.comau\n",
      "2    mangareader.net\n",
      "3    mangareader.net\n",
      "4    mangareader.net\n",
      "Name: Host, dtype: object\n"
     ]
    }
   ],
   "source": [
    "def clean_hosts(data=None):\n",
    "    ''' A bunch of regular expression magic to filter host names to the top level'''\n",
    "    if data is None:\n",
    "\t\traise ValueError(\"Input 'data' to clean_data is None\")\n",
    "    # deal with co.uk and com.au, convert to .couk and .comau, respectively (and other variants)\n",
    "    data[\"Host\"]=data[\"Host\"].apply(lambda x: re.sub('(\\.co)\\.([a-z][a-z])$',r'\\1\\2',x))\n",
    "    data[\"Host\"]=data[\"Host\"].apply(lambda x: re.sub('(\\.com)\\.([a-z][a-z])$',r'\\1\\2',x))\n",
    "    # select just the top level and suffix\n",
    "    data[\"Host\"]=data[\"Host\"].apply(lambda x: '.'.join(str(x).split('.')[-2:]))\n",
    "    # account for cases like http://mlb.com (where there is only one '.')\n",
    "    data[\"Host\"]=data[\"Host\"].apply(lambda x: str(x).split('//')[-1])\n",
    "    \n",
    "    #Remove ERROR Hosts\n",
    "    print '\\nNumber of ERROR hosts to be removed:', sum(data[\"Host\"]==\"ERROR\"), '\\n'\n",
    "    data=data[data[\"Host\"]!=\"ERROR\"]\n",
    "    return data\n",
    "\n",
    "print imp_df.info()\n",
    "print imp_df['Host'].head(5)\n",
    "\n",
    "tic = timeit.default_timer()\n",
    "imp_df = clean_hosts(imp_df)\n",
    "toc = timeit.default_timer()\n",
    "print 'Seconds to clean hosts: ', toc - tic\n",
    "\n",
    "print imp_df.info()\n",
    "print imp_df['Host'].head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Host counts:\n",
      "uvido.com              692\n",
      "featureball.com        560\n",
      "sprungliving.com       228\n",
      "mammabay.couk          123\n",
      "workingfathertv.com    120\n",
      "worsthorrorgame.com     94\n",
      "hourlyparent.com        85\n",
      "ulterior-movies.com     83\n",
      "myhomesdesign.com       69\n",
      "indoorlife.tv           60\n",
      "bumclub.info            59\n",
      "hunp.us                 58\n",
      "psychoworld.tv          57\n",
      "rlinevideos.com         55\n",
      "Name: Host, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "fraudtraffic=['featureball.com',\n",
    "'uvido.com',\n",
    "'sprungliving.com',\n",
    "'sweetboxgames.com',\n",
    "'mammabay.couk', #had to modify .co.uk to .couk because of how I cleaned above\n",
    "'workingfathertv.com',\n",
    "'worsthorrorgame.com',\n",
    "'hourlyparent.com',\n",
    "'ulterior-movies.com',\n",
    "'myhomesdesign.com',\n",
    "'indoorlife.tv',\n",
    "'bumclub.info',\n",
    "'psychoworld.tv',\n",
    "'hunp.us',\n",
    "'rlinevideos.com']\n",
    "\n",
    "print \"Host counts:\\n\", imp_df[imp_df[\"Host\"].isin(fraudtraffic)][\"Host\"].value_counts()#.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yikes, we're missing one host, 'sweetboxgames.com'. By searching through the file, it appears there was a typo and the host name should be 'sweetxboxgames.com' (the 'x' was missing after 'sweet'). I will modify my `fraudtraffic` list to remedy this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Host counts:\n",
      "uvido.com              692\n",
      "featureball.com        560\n",
      "sprungliving.com       228\n",
      "sweetxboxgames.com     150\n",
      "mammabay.couk          123\n",
      "workingfathertv.com    120\n",
      "worsthorrorgame.com     94\n",
      "hourlyparent.com        85\n",
      "ulterior-movies.com     83\n",
      "myhomesdesign.com       69\n",
      "indoorlife.tv           60\n",
      "bumclub.info            59\n",
      "hunp.us                 58\n",
      "psychoworld.tv          57\n",
      "rlinevideos.com         55\n",
      "Name: Host, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "fraudtraffic=['featureball.com',\n",
    "'uvido.com',\n",
    "'sprungliving.com',\n",
    "'sweetxboxgames.com', #fixed typo\n",
    "'mammabay.couk', #had to modify because of how I cleaned above\n",
    "'workingfathertv.com',\n",
    "'worsthorrorgame.com',\n",
    "'hourlyparent.com',\n",
    "'ulterior-movies.com',\n",
    "'myhomesdesign.com',\n",
    "'indoorlife.tv',\n",
    "'bumclub.info',\n",
    "'psychoworld.tv',\n",
    "'hunp.us',\n",
    "'rlinevideos.com']\n",
    "\n",
    "print \"Host counts:\\n\", imp_df[imp_df[\"Host\"].isin(fraudtraffic)][\"Host\"].value_counts()#.describe()\n",
    "\n",
    "#add the fraud column to the data frame\n",
    "imp_df['Fraud']=0\n",
    "imp_df.loc[imp_df[\"Host\"].isin(fraudtraffic),'Fraud']=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#print imp_df.Browser.value_counts()\n",
    "#print imp_df.Browser.head(5)\n",
    "#print 'Chrome', imp_df.UserAgent.iloc[5]\n",
    "#print 'Safari', imp_df.UserAgent.iloc[0]\n",
    "#print 'IE', imp_df.UserAgent.iloc[20]\n",
    "#print 'Firefox', imp_df.UserAgent.iloc[38]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Seconds to creat UA_Browser_Ver:  50.932527039\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Timestamp</th>\n",
       "      <th>IPadd</th>\n",
       "      <th>Browser</th>\n",
       "      <th>UserAgent</th>\n",
       "      <th>Host</th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Wpossize</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Fraud</th>\n",
       "      <th>UA_Browser_Ver</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>393.414.443.469</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...</td>\n",
       "      <td>domain.comau</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0,0,1280,629)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Safari_5.1.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>393.414.443.469</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...</td>\n",
       "      <td>domain.comau</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0,0,1280,629)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Safari_5.1.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Timestamp            IPadd        Browser  \\\n",
       "0  2014-08-25 00:00:00  393.414.443.469  Safari/Webkit   \n",
       "1  2014-08-25 00:00:00  393.414.443.469  Safari/Webkit   \n",
       "2  2014-08-25 00:00:00  325.441.386.395        Unknown   \n",
       "3  2014-08-25 00:00:00  325.441.386.395        Unknown   \n",
       "4  2014-08-25 00:00:00  325.441.386.395        Unknown   \n",
       "\n",
       "                                           UserAgent             Host  Inview  \\\n",
       "0  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...     domain.comau     0.0   \n",
       "1  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...     domain.comau     0.0   \n",
       "2  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "3  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "4  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "\n",
       "   Nplugins        Wpossize  Latency  Fraud    UA_Browser_Ver  \n",
       "0       NaN  (0,0,1280,629)      0.0      0      Safari_5.1.9  \n",
       "1       NaN  (0,0,1280,629)      0.0      0      Safari_5.1.9  \n",
       "2       NaN             NaN      NaN      0  Chrome_36.0.1985  \n",
       "3       NaN             NaN      NaN      0  Chrome_36.0.1985  \n",
       "4       NaN             NaN      NaN      0  Chrome_36.0.1985  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#import useragent\n",
    "def lookup_user_agent(ua_string):\n",
    "    ua=useragent.detect(ua_string)\n",
    "    if ua.browser.family != 'Other':\n",
    "        return ua.browser.family+'_'+ua.browser.version\n",
    "    else:\n",
    "        return 'Other'\n",
    "    \n",
    "tic = timeit.default_timer()\n",
    "imp_df['UA_Browser_Ver']=imp_df[\"UserAgent\"].map(lambda x: lookup_user_agent(x))\n",
    "toc = timeit.default_timer()\n",
    "print 'Seconds to creat UA_Browser_Ver: ', toc - tic\n",
    "imp_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UA_Browser from UA_Browser_Ver:\n",
      "Chrome                          84221\n",
      "Other                           33927\n",
      "Firefox                         32919\n",
      "IE                              32796\n",
      "Safari                          20655\n",
      "Mobile Safari                   17846\n",
      "Chrome Mobile                    6217\n",
      "Android                          4439\n",
      "Chrome Mobile iOS                 805\n",
      "IE Mobile                         231\n",
      "Chrome Frame                      214\n",
      "Firefox Mobile                    126\n",
      "Opera                              89\n",
      "Silk                               73\n",
      "Maxthon                            68\n",
      "Chromium                           63\n",
      "Iceweasel                          54\n",
      "SeaMonkey                          52\n",
      "WebKit Nightly                     39\n",
      "Blackberry WebKit                  21\n",
      "Nintendo 3DS                       19\n",
      "PlayStation                        17\n",
      "Nokia Services (WAP) Browser       17\n",
      "Opera Mobile                       10\n",
      "Nokia OSS Browser                   3\n",
      "RockMelt                            3\n",
      "Pale Moon (Firefox Variant)         2\n",
      "OmniWeb                             2\n",
      "Netscape                            2\n",
      "Baidu Explorer                      2\n",
      "NokiaBrowser                        2\n",
      "Flock                               2\n",
      "Iron                                2\n",
      "Firefox Alpha                       1\n",
      "SEMC-Browser                        1\n",
      "Firefox Beta                        1\n",
      "Sleipnir                            1\n",
      "Opera Tablet                        1\n",
      "Avant                               1\n",
      "Sogou Explorer                      1\n",
      "Name: UA_Browser_Ver, dtype: int64\n",
      "Browser:\n",
      "Chrome               76595\n",
      "Internet Explorer    63874\n",
      "Safari/Webkit        44093\n",
      "Firefox              28697\n",
      "Unknown              21599\n",
      "Opera                   87\n",
      "Name: Browser, dtype: int64\n",
      "Seconds to creat UA_Browser:  0.122408642999\n",
      "UA_Browser:\n",
      "Chrome               91457\n",
      "Other                38811\n",
      "Safari/Webkit        38501\n",
      "Firefox              33049\n",
      "Internet Explorer    33027\n",
      "Opera                  100\n",
      "Name: UA_Browser, dtype: int64\n",
      "Browser after filling Unknowns\n",
      "Chrome               85164\n",
      "Internet Explorer    66723\n",
      "Safari/Webkit        48239\n",
      "Firefox              31412\n",
      "Other                 3311\n",
      "Opera                   96\n",
      "Name: Browser, dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Timestamp</th>\n",
       "      <th>IPadd</th>\n",
       "      <th>Browser</th>\n",
       "      <th>UserAgent</th>\n",
       "      <th>Host</th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Wpossize</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Fraud</th>\n",
       "      <th>UA_Browser_Ver</th>\n",
       "      <th>UA_Browser</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>393.414.443.469</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...</td>\n",
       "      <td>domain.comau</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0,0,1280,629)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Safari_5.1.9</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>393.414.443.469</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...</td>\n",
       "      <td>domain.comau</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0,0,1280,629)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Safari_5.1.9</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "      <td>Chrome</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "      <td>Chrome</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "      <td>Chrome</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Timestamp            IPadd        Browser  \\\n",
       "0  2014-08-25 00:00:00  393.414.443.469  Safari/Webkit   \n",
       "1  2014-08-25 00:00:00  393.414.443.469  Safari/Webkit   \n",
       "2  2014-08-25 00:00:00  325.441.386.395         Chrome   \n",
       "3  2014-08-25 00:00:00  325.441.386.395         Chrome   \n",
       "4  2014-08-25 00:00:00  325.441.386.395         Chrome   \n",
       "\n",
       "                                           UserAgent             Host  Inview  \\\n",
       "0  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...     domain.comau     0.0   \n",
       "1  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...     domain.comau     0.0   \n",
       "2  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "3  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "4  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "\n",
       "   Nplugins        Wpossize  Latency  Fraud    UA_Browser_Ver     UA_Browser  \n",
       "0       NaN  (0,0,1280,629)      0.0      0      Safari_5.1.9  Safari/Webkit  \n",
       "1       NaN  (0,0,1280,629)      0.0      0      Safari_5.1.9  Safari/Webkit  \n",
       "2       NaN             NaN      NaN      0  Chrome_36.0.1985         Chrome  \n",
       "3       NaN             NaN      NaN      0  Chrome_36.0.1985         Chrome  \n",
       "4       NaN             NaN      NaN      0  Chrome_36.0.1985         Chrome  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_ua_browser(browver):\n",
    "    if 'Chrome' in browver:\n",
    "        return 'Chrome'\n",
    "    elif 'Other' in browver:\n",
    "        return 'Other'\n",
    "    elif 'Firefox' in browver:\n",
    "        return 'Firefox'\n",
    "    elif 'IE' in browver:\n",
    "        return 'Internet Explorer'\n",
    "    elif 'Safari' in browver:\n",
    "        return 'Safari/Webkit'\n",
    "    elif 'Opera' in browver:\n",
    "        return 'Opera'\n",
    "    else:\n",
    "        return 'Other'\n",
    "    \n",
    "print 'UA_Browser from UA_Browser_Ver:\\n', imp_df[\"UA_Browser_Ver\"].map(lambda x: x.split('_')[0]).fillna('X').value_counts()\n",
    "print 'Browser:\\n', imp_df[\"Browser\"].value_counts()\n",
    "tic = timeit.default_timer()\n",
    "#Add column for browser from user agent string\n",
    "imp_df['UA_Browser']=imp_df[\"UA_Browser_Ver\"].map(lambda x: get_ua_browser(x))\n",
    "toc = timeit.default_timer()\n",
    "print 'Seconds to creat UA_Browser: ', toc - tic\n",
    "print 'UA_Browser:\\n', imp_df[\"UA_Browser\"].value_counts()\n",
    "imp_df.loc[imp_df['Browser']=='Unknown','Browser'] = imp_df.loc[imp_df['Browser']=='Unknown','UA_Browser']\n",
    "print 'Browser after filling Unknowns\\n', imp_df[\"Browser\"].value_counts()\n",
    "imp_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Seconds to creat Tsecs:  0.460189447826\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Timestamp</th>\n",
       "      <th>IPadd</th>\n",
       "      <th>Browser</th>\n",
       "      <th>UserAgent</th>\n",
       "      <th>Host</th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Wpossize</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Fraud</th>\n",
       "      <th>UA_Browser_Ver</th>\n",
       "      <th>UA_Browser</th>\n",
       "      <th>Tsecs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>393.414.443.469</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...</td>\n",
       "      <td>domain.comau</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0,0,1280,629)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Safari_5.1.9</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>393.414.443.469</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...</td>\n",
       "      <td>domain.comau</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0,0,1280,629)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Safari_5.1.9</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Timestamp            IPadd        Browser  \\\n",
       "0  2014-08-25 00:00:00  393.414.443.469  Safari/Webkit   \n",
       "1  2014-08-25 00:00:00  393.414.443.469  Safari/Webkit   \n",
       "2  2014-08-25 00:00:00  325.441.386.395         Chrome   \n",
       "3  2014-08-25 00:00:00  325.441.386.395         Chrome   \n",
       "4  2014-08-25 00:00:00  325.441.386.395         Chrome   \n",
       "\n",
       "                                           UserAgent             Host  Inview  \\\n",
       "0  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...     domain.comau     0.0   \n",
       "1  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...     domain.comau     0.0   \n",
       "2  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "3  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "4  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "\n",
       "   Nplugins        Wpossize  Latency  Fraud    UA_Browser_Ver     UA_Browser  \\\n",
       "0       NaN  (0,0,1280,629)      0.0      0      Safari_5.1.9  Safari/Webkit   \n",
       "1       NaN  (0,0,1280,629)      0.0      0      Safari_5.1.9  Safari/Webkit   \n",
       "2       NaN             NaN      NaN      0  Chrome_36.0.1985         Chrome   \n",
       "3       NaN             NaN      NaN      0  Chrome_36.0.1985         Chrome   \n",
       "4       NaN             NaN      NaN      0  Chrome_36.0.1985         Chrome   \n",
       "\n",
       "   Tsecs  \n",
       "0      0  \n",
       "1      0  \n",
       "2      0  \n",
       "3      0  \n",
       "4      0  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def timestamp_to_secs(x):\n",
    "    return int(x[11:13])*3600  + int(x[-5:-3])*60 + int(x[-2:])\n",
    "\n",
    "tic = timeit.default_timer()\n",
    "#Add column that converts timestamp to time of day, in seconds\n",
    "imp_df[\"Tsecs\"]=imp_df[\"Timestamp\"].map(lambda x: timestamp_to_secs(x))\n",
    "toc = timeit.default_timer()\n",
    "print 'Seconds to creat Tsecs: ', toc - tic\n",
    "imp_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Seconds to creat Warea:  0.285744084644\n",
      "Seconds to creat WXpos WYPos Wwidth Wheight:  1.6378743159\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Timestamp</th>\n",
       "      <th>IPadd</th>\n",
       "      <th>Browser</th>\n",
       "      <th>UserAgent</th>\n",
       "      <th>Host</th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Wpossize</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Fraud</th>\n",
       "      <th>UA_Browser_Ver</th>\n",
       "      <th>UA_Browser</th>\n",
       "      <th>Tsecs</th>\n",
       "      <th>Warea</th>\n",
       "      <th>WXpos</th>\n",
       "      <th>WYpos</th>\n",
       "      <th>Wwidth</th>\n",
       "      <th>Wheight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>393.414.443.469</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...</td>\n",
       "      <td>domain.comau</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0,0,1280,629)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Safari_5.1.9</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>0</td>\n",
       "      <td>805120.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1280.0</td>\n",
       "      <td>629.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>393.414.443.469</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...</td>\n",
       "      <td>domain.comau</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0,0,1280,629)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Safari_5.1.9</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>0</td>\n",
       "      <td>805120.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1280.0</td>\n",
       "      <td>629.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Timestamp            IPadd        Browser  \\\n",
       "0  2014-08-25 00:00:00  393.414.443.469  Safari/Webkit   \n",
       "1  2014-08-25 00:00:00  393.414.443.469  Safari/Webkit   \n",
       "2  2014-08-25 00:00:00  325.441.386.395         Chrome   \n",
       "3  2014-08-25 00:00:00  325.441.386.395         Chrome   \n",
       "4  2014-08-25 00:00:00  325.441.386.395         Chrome   \n",
       "\n",
       "                                           UserAgent             Host  Inview  \\\n",
       "0  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...     domain.comau     0.0   \n",
       "1  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...     domain.comau     0.0   \n",
       "2  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "3  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "4  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "\n",
       "   Nplugins        Wpossize  Latency  Fraud    UA_Browser_Ver     UA_Browser  \\\n",
       "0       NaN  (0,0,1280,629)      0.0      0      Safari_5.1.9  Safari/Webkit   \n",
       "1       NaN  (0,0,1280,629)      0.0      0      Safari_5.1.9  Safari/Webkit   \n",
       "2       NaN             NaN      NaN      0  Chrome_36.0.1985         Chrome   \n",
       "3       NaN             NaN      NaN      0  Chrome_36.0.1985         Chrome   \n",
       "4       NaN             NaN      NaN      0  Chrome_36.0.1985         Chrome   \n",
       "\n",
       "   Tsecs     Warea  WXpos  WYpos  Wwidth  Wheight  \n",
       "0      0  805120.0    0.0    0.0  1280.0    629.0  \n",
       "1      0  805120.0    0.0    0.0  1280.0    629.0  \n",
       "2      0       NaN    NaN    NaN     NaN      NaN  \n",
       "3      0       NaN    NaN    NaN     NaN      NaN  \n",
       "4      0       NaN    NaN    NaN     NaN      NaN  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def window_area(x):\n",
    "    #Multiplies the width and height of the window\n",
    "    val=np.NAN\n",
    "    if isinstance(x,str):\n",
    "        [xp,yp,w,h]=x[1:-1].split(',')\n",
    "        val=int(w)*int(h)\n",
    "    return val\n",
    "\n",
    "import timeit\n",
    "tic = timeit.default_timer()\n",
    "#Add column that contains browser window area\n",
    "imp_df[\"Warea\"] = imp_df[\"Wpossize\"].map(lambda x: window_area(x))     \n",
    "toc = timeit.default_timer()\n",
    "print 'Seconds to creat Warea: ', toc - tic\n",
    "\n",
    "\n",
    "def parse_wpos(w,n):\n",
    "    #Multiplies the width and height of the window\n",
    "    val=np.NAN\n",
    "    if isinstance(w,str):\n",
    "        val=[int(x) for x in w[1:-1].split(',')][n]\n",
    "    return val\n",
    "\n",
    "import timeit\n",
    "tic = timeit.default_timer()\n",
    "#Add column that contains browser window area\n",
    "#imp_df[\"Warea\"] = imp_df[\"Wpossize\"].map(lambda x: window_area(x))     \n",
    "#print parse_wpos(imp_df.Wpossize.iloc[0],0)\n",
    "imp_df['WXpos']=imp_df[\"Wpossize\"].map(lambda x: parse_wpos(x,0))\n",
    "imp_df['WYpos']=imp_df[\"Wpossize\"].map(lambda x: parse_wpos(x,1))\n",
    "imp_df['Wwidth']=imp_df[\"Wpossize\"].map(lambda x: parse_wpos(x,2))\n",
    "imp_df['Wheight']=imp_df[\"Wpossize\"].map(lambda x: parse_wpos(x,3))\n",
    "toc = timeit.default_timer()\n",
    "print 'Seconds to creat WXpos WYPos Wwidth Wheight: ', toc - tic\n",
    "imp_df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now reduce the data to hosts that have at least 9 visits. Fraudulent traffic should be significant and it's hard to measure in such a small sample. If your botnet is hitting a website 8 times or less a day, you win. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "adsafeprotected.com            33097\n",
      "answers.com                     5544\n",
      "yahoo.com                       4300\n",
      "ebay.couk                       3736\n",
      "ebay.com                        3249\n",
      "pandora.com                     2954\n",
      "juno.com                        2899\n",
      "msn.com                         2703\n",
      "aol.com                         2658\n",
      "amazon.com                      2442\n",
      "mynet.com                       2338\n",
      "domain.comau                    2319\n",
      "cars.com                        2274\n",
      "stuffyoushouldknow.com          1676\n",
      "animeget.com                    1622\n",
      "huffingtonpost.com              1384\n",
      "dailymail.couk                  1366\n",
      "startribune.com                 1202\n",
      "pch.com                         1130\n",
      "usmagazine.com                  1120\n",
      "narutoget.com                   1092\n",
      "ebay.de                         1034\n",
      "people.com                       967\n",
      "theblaze.com                     897\n",
      "go.com                           873\n",
      "ratemyprofessors.com             837\n",
      "tagged.com                       832\n",
      "likes.com                        813\n",
      "foxnews.com                      810\n",
      "cnn.com                          800\n",
      "                               ...  \n",
      "maxim.com                          1\n",
      "archanaskitchen.com                1\n",
      "mxptint.net?;ord=9508408           1\n",
      "calciomercato.it                   1\n",
      "smallbrightbusinessidea.com        1\n",
      "mult-games.ru                      1\n",
      "mydoctorgames.com                  1\n",
      "yatedo.com                         1\n",
      "geekdimensions.com                 1\n",
      "payscale.com                       1\n",
      "bennettknows.com                   1\n",
      "momendeavors.com                   1\n",
      "readersdigest.couk                 1\n",
      "nicoclub.com                       1\n",
      "zixzag.com                         1\n",
      "idgameland.com                     1\n",
      "nakkheeran.in                      1\n",
      "vaildaily.com                      1\n",
      "ottopagine.net                     1\n",
      "credithelpcards.com                1\n",
      "gifanimations.com                  1\n",
      "prisontalk.com                     1\n",
      "ktuu.com                           1\n",
      "weemove.com                        1\n",
      "punto-informatico.it               1\n",
      "cpmregie.com                       1\n",
      "funradio.fr                        1\n",
      "rtlxl.nl                           1\n",
      "forsport.tv                        1\n",
      "mazdaforum.com                     1\n",
      "Name: Host, dtype: int64\n",
      "count     9058.000000\n",
      "mean        25.937845\n",
      "std        373.533524\n",
      "min          1.000000\n",
      "25%          1.000000\n",
      "50%          3.000000\n",
      "75%          9.000000\n",
      "max      33097.000000\n",
      "Name: Host, dtype: float64\n",
      "Number of hosts to keep:  2398\n"
     ]
    }
   ],
   "source": [
    "host_counts = imp_df[\"Host\"].value_counts()\n",
    "print host_counts\n",
    "print host_counts.describe()\n",
    "sighost_counts = host_counts[host_counts>8]\n",
    "keepHosts = sighost_counts.index.map(lambda x: str(x))\n",
    "#print keepHosts\n",
    "print 'Number of hosts to keep: ', len(keepHosts)\n",
    "redhost_imp_df = imp_df[imp_df[\"Host\"].isin(keepHosts)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Timestamp</th>\n",
       "      <th>IPadd</th>\n",
       "      <th>UserAgent</th>\n",
       "      <th>Host</th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Wpossize</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Fraud</th>\n",
       "      <th>UA_Browser_Ver</th>\n",
       "      <th>UA_Browser</th>\n",
       "      <th>Tsecs</th>\n",
       "      <th>Warea</th>\n",
       "      <th>WXpos</th>\n",
       "      <th>WYpos</th>\n",
       "      <th>Wwidth</th>\n",
       "      <th>Wheight</th>\n",
       "      <th>Browser_Chrome</th>\n",
       "      <th>Browser_Firefox</th>\n",
       "      <th>Browser_Internet Explorer</th>\n",
       "      <th>Browser_Opera</th>\n",
       "      <th>Browser_Other</th>\n",
       "      <th>Browser_Safari/Webkit</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>393.414.443.469</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...</td>\n",
       "      <td>domain.comau</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0,0,1280,629)</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Safari_5.1.9</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>0</td>\n",
       "      <td>805120</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1280</td>\n",
       "      <td>629</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>393.414.443.469</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...</td>\n",
       "      <td>domain.comau</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0,0,1280,629)</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Safari_5.1.9</td>\n",
       "      <td>Safari/Webkit</td>\n",
       "      <td>0</td>\n",
       "      <td>805120</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1280</td>\n",
       "      <td>629</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2014-08-25 00:00:00</td>\n",
       "      <td>325.441.386.395</td>\n",
       "      <td>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...</td>\n",
       "      <td>mangareader.net</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>Chrome_36.0.1985</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Timestamp            IPadd  \\\n",
       "0  2014-08-25 00:00:00  393.414.443.469   \n",
       "1  2014-08-25 00:00:00  393.414.443.469   \n",
       "2  2014-08-25 00:00:00  325.441.386.395   \n",
       "3  2014-08-25 00:00:00  325.441.386.395   \n",
       "4  2014-08-25 00:00:00  325.441.386.395   \n",
       "\n",
       "                                           UserAgent             Host  Inview  \\\n",
       "0  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...     domain.comau       0   \n",
       "1  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...     domain.comau       0   \n",
       "2  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "3  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "4  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  mangareader.net     NaN   \n",
       "\n",
       "   Nplugins        Wpossize  Latency  Fraud    UA_Browser_Ver     UA_Browser  \\\n",
       "0       NaN  (0,0,1280,629)        0      0      Safari_5.1.9  Safari/Webkit   \n",
       "1       NaN  (0,0,1280,629)        0      0      Safari_5.1.9  Safari/Webkit   \n",
       "2       NaN             NaN      NaN      0  Chrome_36.0.1985         Chrome   \n",
       "3       NaN             NaN      NaN      0  Chrome_36.0.1985         Chrome   \n",
       "4       NaN             NaN      NaN      0  Chrome_36.0.1985         Chrome   \n",
       "\n",
       "   Tsecs   Warea  WXpos  WYpos  Wwidth  Wheight  Browser_Chrome  \\\n",
       "0      0  805120      0      0    1280      629               0   \n",
       "1      0  805120      0      0    1280      629               0   \n",
       "2      0     NaN    NaN    NaN     NaN      NaN               1   \n",
       "3      0     NaN    NaN    NaN     NaN      NaN               1   \n",
       "4      0     NaN    NaN    NaN     NaN      NaN               1   \n",
       "\n",
       "   Browser_Firefox  Browser_Internet Explorer  Browser_Opera  Browser_Other  \\\n",
       "0                0                          0              0              0   \n",
       "1                0                          0              0              0   \n",
       "2                0                          0              0              0   \n",
       "3                0                          0              0              0   \n",
       "4                0                          0              0              0   \n",
       "\n",
       "   Browser_Safari/Webkit  \n",
       "0                      1  \n",
       "1                      1  \n",
       "2                      0  \n",
       "3                      0  \n",
       "4                      0  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dum_imp_df = pd.get_dummies(redhost_imp_df,columns=['Browser'])\n",
    "dum_imp_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1.0, 10000.0)\n",
      "(1.0, 10000.0)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf4AAAH3CAYAAABaYD3tAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4XNWd5/+PapFsSbZlleRFwsZ2C28JgqYxMSaAM+4s\nEydxhnSOA+7pJx2Ck5B5CMPPmBCW4CaENWTp6UwCwdPTaZLO6YTGnQk9cToMCSHsS0PAxhgcGyyD\nrbItW4vlKpV+f1zVqirpSlWqUtV9v54nD7fOPffeI91Y37rnnvM9VYODgwIAAN7gK3UDAABA8RD4\nAQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8AAA8h8AMA4CGBYl3IGNMu6QFJd1trvztUdreklZJikq60\n1j4zVD5H0nOSTrHWxorVRgAAKl1RnviNMbWSviFpe0rZBZLarLWrJH1W0ndSDrlK0m+K0TYAALyk\nWF39JyStlfROStkaSQ9KkrV2p6QGY0y9MeYSST8bOgYAABRQUQK/tTZmrT2ZUTxH0qGUz4eGylZK\n+pCkMyV9qhjtAwDAK4r2jt8FnyRZa6+QJGPMqZL+qaQtAgCgwpQy8HfIecKPa5F0IP7BWvsZl+dh\nlSEAgBdVjeegUgT+eEO3S7pJ0r3GmLMk7bfW9oznhB0dHQVqGoqtpaWF+1fGuH/li3tX3lpaWsZ9\nbFECvzHmPZJ+IKlZUtQY8zlJF0p6zhjzmKQBSV8sRlsAAPCyogR+a+2Tkk7PsuvaYlwfAAA4yNwH\nAICHEPgBAPAQAj8AAB5C4AcAwEMI/AAAeAiBHwAADyHwAwDgIQR+AAA8hMAPAICHEPgBAPAQAj8A\nAB5C4AcAwEMI/AAAeAiBHwAADyHwAwDgIQR+AAA8hMAPAICHEPgBAPAQAj8AAB5C4AcAwEMI/AAA\neAiBHwAADyHwAwDgIQR+AAA8hMAPAICHEPgBAPAQAj8AAB5C4AcAwEMI/AAAeAiBHwAADyHwAwDg\nIQR+AAA8hMAPAICHEPgBAPAQAj8AAB5C4AcAwEMI/AAAeAiBHwAADyHwAwDgIQR+AAA8hMAPAICH\nEPgBAPAQAj8AAB5C4AcAwEMI/AAAeAiBHwAADyHwAwDgIQR+AAA8hMAPAICHEPgBAPAQAj8AAB5C\n4AcAwEMI/AAAeAiBHwAADyHwAwDgIQR+AAA8hMAPAICHEPgBAPAQAj8AAB5C4AcAwEMI/AAAeAiB\nHwAADyHwAwDgIQR+AAA8hMAPAICHEPgBAPAQAj8AAB5C4AcAwEMI/AAAeAiBHwAADyHwAwDgIQR+\nAAA8hMAPAICHEPgBAPAQAj8AAB5C4AcAwEMI/AAAeAiBHwAADyHwAwDgIQR+AAA8hMAPAICHEPgB\nAPAQAj8AAB5C4AcAwEMI/AAAeAiBHwAADyHwAwDgIQR+AAA8hMAPAICHEPgBAPAQAj8AAB5C4AcA\nwEMI/AAAeAiBHwAADyHwAwDgIQR+AAA8hMAPAICHEPgBAPAQAj8AAB5C4AcAwEMI/AAAeAiBHwAA\nDyHwAwDgIYFiXcgY0y7pAUl3W2u/O1R2t6SVkmKSvmStfdYYs0rS5yUFJd1prX2uWG0EAKDSFeWJ\n3xhTK+kbkranlF0gqc1au0rSZyX97dCurqHPd0taXYz2AQDgFcXq6j8haa2kd1LK1kh6UJKstTsl\nNRhj6q21Lw/tu1XSvxSpfQAAeEJRAr+1NmatPZlRPEfSoZTPhyTNMcacba39N0nrJV1VjPYBAOAV\nRXvH70L8S0jIGPN9SbWS/rGE7QEAoOKUMvB3yHnqj2uRdMBau1vSL8dyopaWlkK2C0XG/Stv3L/y\nxb3zplIE/qqh/26XdJOke40xZ0nab63tGc8JOzo6CtQ0FFtLSwv3r4xx/8oX96685fOlrSiB3xjz\nHkk/kNQsKWqM+ZykCyU9Z4x5TNKApC8Woy0AAHhZUQK/tfZJSadn2XVtMa4PAAAcZO4DAMBDCPwA\nAHgIgR8AAA8h8AMA4CEEfgAAPITADwCAhxD4AQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8AAA8h8AMA\n4CEEfgAAPITADwCAhxD4AQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8AAA8h8AMA4CEEfgAAPITADwCA\nhxD4AQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8AAA8h8AMA4CEEfgAAPITADwCAhxD4AQDwEAI/AAAe\nQuAHAMBDCPwAAHgIgR8AAA8h8AMA4CEEfgAAPITADwCAhxD4AQDwEAI/AAAeQuAHAMBDCPwAAHgI\ngR8AAA8h8AMA4CEEfgAAPITADwCAhxD4AQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8AAA8h8AMA4CEE\nfgAAPITADwCAhxD4AQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8AAA8h8AMA4CEEfgAAPITADwCAhxD4\nAQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8AAA8h8AMA4CEEfgAAPITADwCAhxD4AQDwEAI/AAAeQuAH\nAMBDCPwAAHgIgR8AAA8h8AMA4CEEfgAAPITADwCAhxD4AQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8A\nAA8h8AMA4CEjBn5jTJUx5pPFagwAAJhYIwZ+a+2gpOuMMcEitQcAAEyggIs6/y7pKWPMryUdTt1h\nrf36hLQKAABMCDeBPyTphaH/hlLKByekRQAAYMKMGvittX9djIYAAICJN2rgN8bUSLpZ0ickBay1\npxpjrpa0zVq7a6IbCAAACsfNdL6tkurlBP4TQ2W7JH1/ohoFAAAmhpvAf6619nJr7QuSBiTJWrtN\n0qwJbRkAACg4N4G/3xgzO7XAGNMkBvcBAFB23AT+b0p6wRjzLUlNxpg7JD0xVA4AAMrIqIHfWnuP\nJCOpX9K/SOqR9Elr7X0T3DYAAFBgbkb132mtvVrSoxnl1lprJqxlAACg4HIGfmPM+ZIukPRXxpgj\nGbsbJH1oIhsGAAAKb6Qn/nckRSXVSDotY19E0oaJahQAAJgYIwX+c621txtjeqy1/6NoLQIAABNm\npMB/izFmn6T/zxjzrKSqzArW2t9PWMsAAEDBjRT4/0HSvZJaJf0oy/5BSYsmolEAAGBi5Az81tqv\nSPqKMeYfrbV/mbnfGFM7oS0DAAAF52Z1vr80xvglzZUz739Q0jRJ/0c88QMAUFbczONfL2ehnikp\nxf2SHhzLhYwx7ZIekHS3tfa7Q2V3S1opKSbpS9baZ40xKyV9VpJf0nestc+P5ToAACA3Nyl7b5Yz\nZz8oZ1W+qZJukvTPbi8y9FrgG5K2p5RdIKnNWrtKTqD/26Fd3ZIul/QtSee7vQYAABidm8AftdY+\naq2NSaqy1p601t4h6foxXOeEpLVycgPErdFQr4G1dqekBmNMvbX2D3JyB3xBzgBDAABQIKN29UsK\nG2OulXSnpE5jzAclPStp9siHJQ19aThpTFqG3zmSnkn53ClpjjHmoKTbJV1rrT3q9hoAAGB0bgL/\nZyXdJec9/C2SfiqpVtIdBW5LPE/ANXIGD95gjHnUWvsvox3Y0tJS4KagmLh/5Y37V764d97kZlT/\nq5I+OvTx34wxMyXVWWu78rx2h5yn/rgWSQestdeN+UQdHXk2BaXS0tLC/Stj3L/yxb0rb/l8aXPz\nxJ/GWhuVlE/Qjz/Zb5czSPBeY8xZkvZba3vyOC8AABjFmAP/eBhj3iPpB5KaJUWNMZ+TdKGk54wx\nj0kakPTFYrQFAAAvK0rgt9Y+Ken0LLuuLcb1AQCAw00Cnwty7BqUdFTSLmttf0FbBQAAJoSbJ/7v\nSFosZ87/QTnd9TFJ+yU1SJIx5i+ttdtzngEAAEwKbhL4PCCnS77eWjtfUr2kzZLusdbOkvQpOdP9\nAADAJOfmiX+DtXZJ/IO1dkDS3xljXpJ0l7X2YWNMzYS1EAAAFIybwF9ljNko6Z+stceMMVMkGTlp\ndWWM2Szp8AS2EQAAFIibrv5PSbpM0lFjTFRSr5yu/s8O7T9b0qcnpHUAAKCg3GTue07SiqEn/ZmS\njllre4aWz5W11ox4AgAAMGm4msdvjFklaZGGegiMMfWStsgZ4Q8AAMqEm3n8d8rpyn9Z0p9J+g9J\np0m6cUJbBgAACs7NO/6LJLVZa1dLesta+15Jfylp7kQ2DAAAFJ6bwB9JWYnPJ0nW2l9J+viEtQoA\nAEwIN+/4/8MY83/lBPpXjTG3SHpeQ1n7AABA+XDzxP9pSb+y1p6QdJWc6XtfkXTlBLYLAABMADfT\n+fokfWNoe7ekD0qSMebPJ7ZpAACg0Nw88efyUMFaAQAAiiKfwF9VsFYAAICiyCfwDxasFQAAoChy\nvuM3xrQUsyEAAGDijTS47y05T/W5uvR54gcAoMzkDPzW2nxeA6CMRSLSgQN+SdLcuQMKBkvcIBfK\nsc0AUAquFumBtxw44NfRo8nvffPnD5SwNe6UY5sBoBQI/CiJ+BN6f7/k84kndAAoEgI/hpk7dyDr\ndiHFn9CDQSkS8ef9hF6MNgNAJSDw56kS3y0Hg+XXVV6ObQaAUiDw54l3y+MTfypvbJR8Pn5nAFAs\njNxHScSf0Bcu5P0+ABQTT/x54t0yAKCcVETgj0RK99TIu2UAQDmpiK7++OA6AAAwsop44kdlqMQZ\nEgAw2VRE4OfdemXIZ4ZEob408OUDQKWriMDPH2cUalol0zMBVLqKCPxwbzI/0ZZyhkT899LR4VN1\ntRTgXwaAClURf956e6Xa2lK3ojxM5ifafGZI5PulIf57qa6WTp6UGhpiikalffv8k+4LEgDkoyIC\n/1NPBbV6daTUzagYxegVKPQiPYWaVhkISE1NMUmatF+QACAfFTGdb9euivj+UhRz5w6ooSGmhoZY\nzifj+NPv0aO+CZsqGb/G4cPJLxmRiPOEvW+fX5Eif49z83sBgEpQERGzoSFW6iaUjcmccKiUryEy\nfy9kZARQqSoi8J9xRrTUTago4w16Y3lFMNkX6ZnMX5AAIB8VEfgXLeIPdCGNN+iN5Yk9fo2WFqmj\nwynL/MIxmWcgAEC5qojAT0CoDJlfOPbtc/9Fgi8JAOBORQT+SlLOAayU78Un8zRFAJhMKiLwl3J1\nvkIH6nIOYIV+L84AOwAovIoI/AcO+EsWIMs5UE92Y/kiwZcEAHCnIgJ/JSGAjQ+j8AHAnYoI/E1N\npfuDX+hATQADAEykigj8nZ2l6+onUAMAyklFBH5UpnKb4VBu7QXgTRUR+Ev5LrzQf+wnOniUU3Aq\n5cDJ8fyeGOgJoBxUROAvZfAq9B/7iQ4eBCd3+D0BqFQVEfhRmZqaBtTR4QTftrbJFXiz9QgwIwNA\nOaiIwF/KBD4zZgzo+eedX+Pq1fmvJTvRwWMig2mu7vHeXunFF50P7e0R1da6O76z05+o29np19y5\nAxP+miLehmhUqq+PKRDIfh+y9Qgw0BNAOaiIwF/KBD47dgQVjfoS2ytX5hf8Jzp4ZAbTQl4rV/f4\niy8GtX9/8v9quX5Hmce7PX8hpV6joSFGIAdQcSoi8J84UbprR6NSOFwlSZo9O//zTYbBd6VqQ1+f\n9NJLzv8lzz77ZNqqi6lP+9nk2+b48R0dPlVXS4FR/mXQrQ+gXFVE4D9wwKfFi0vzx7exMab+/qrE\ndr4m+qnWTcAabxtynbu9PZJ1O9OhQz719PgS20uWDKRde6S25/t7ix9fXS2dPCk1NcVGDOh06wMo\nVxUR+EfqHp5ox4751NISS2xLkzsYTGTAynXu2trc3fupAgEpFIoltt2ev5ACASfoE9QBVKqKCPzL\nlkVLdu1Zs2I6diyW2M7XZOhCLkYb4l3r/f2Sz+cE9WXLInrnHedL3LJlYxsrkW+bRzu+2K8/JsMr\nHwCVqSIC/8KFpXs6mzdvIPF0Wim5+ovRhnjXejAoRSLOIMOuLr8WLnS+PHV1+TVjhvs25Nvm0Y4v\n9rx+8ggAmCgVEfhL+TRU6CA53ie9ri7pkUdqJEmrV/drxozit6GYJmMbCzXAUHIGjQLARKiIwF9J\nxvuk98gjNXrjjWSkWbeuv+htGIt470hjo+TzDaSVZW6Xqo2pJnJQZLbj6+tjamiIjXg9ABiPigj8\npUzgU+gnzyNHpH/9V+fJ/WMf69P8+e6OGxiQurqqEtuTXbynpKVF6uhIL4vL/N1Kyc99fdLbbyeD\nZKFlu6/F7G4PBOjeBzAxKiLwlzKBz549fj39dLUkacWKk3lPK3z88Wq9/nogsX3GGX2ujmtri+rl\nl4OJ7VzcfFEpVarczAx/nZ3DE/rEPx87NrFtGc/T+0QPMASAQqiIwF9KO3YEdPCgP7Gdb+Dv7vYl\nBgt2d7ufptjb69Py5QOJ7VzTCt0EtInM7jeSzAx/8WmS2QQC0pw5uaf+lcJEDzAEgEKYJH8y8zOW\n0d+Fv3ZMJ08mt/P13vee0Ouv+xPbbjU2xvT668ntXKLRie0izyZbrv5s0/kyjfQEfOqpA9qxI97D\nUfjBlfFc/ZLzO9u3zz9pBhECQD4qIvAXIkf+eLW0xHTKKQOJ7XxFoz796Z9GE9tuEwIFAlJzc2Ge\ngAvd5ZwtV3+26XyZGf6yPQHHP+/bNzG9Epm5+iUxrQ5ARamIwF9KU6ZIp58eTWzny8leN5jYHstx\nbrq+3dQrVZez2wx/AIDxq4jA39ZWumBR6KfjtraIXn7ZuS0XXuj+53I7IK8Q7c01QDBXebZc/fH2\n9vRILS0TNxDO7awLN8vxMuAOQCWoiMC/e3dQTU2lCf6FfjrevTuohobkttufy+2AvEK0N9cAwVzl\n2Z7k4+2tqxtfV73bn8Pt6PyRluOlex9AJamIwB+fB14K2Qau5ePoUemRR5zzrV3rPn3biRPJJW1X\nrDiZs14pMt51dkrbtk2VJK1b16empuz1Cv27lIo7mHEyZhMEgEwVEfgffniKLrqoNE/82Qau5eOP\nfwwkpvH98Y8BSe7Od/CgTz09VYntXNMKC5HxLlc3e67tbdumateu4NC2dOmlfVkz9xX6dzkWhXgF\nQn59AOWgIgJ/b2/prn3ihLR7t/OUFwrln2D95Enp8GFfYrvQ+vqSPQNnnz2+C0QiSownaGpKPtnm\n6n4fGJC6u9OzCmbL3Dfa7zL1iXrGjOR0vvgMgGxP2/HBjPEekYMHfVl7E5hDD8ArKiLw+9znuZn0\nqqokaTBl251Zs2Lau3cwsZ3LoUM+9fT4EttLlow92I31yfw97zmpd97xJ7ZzSf15s/3sqU/Uzz8f\nGJru6GhpiWV92o4/vT//fED9/T7t3+9z1ebxIPMegHJQEYG/lO9SRwtWY1VbKy1dmhwUV2jRqHTk\nSFViezz6+5NP5k1No59k5szkokHxgYvZ1NQkZyTU1Ax/Z576vv7kSenIEWd79uz088QT7sSPmz/f\nmUEQD/ojyec9Pb0GAMpBRQT+s86agD5xlwYHs2+P17p1fdq2Lbntltt3/IX4ojLWn9ntk3DmtL/M\nd+apGhtjOnnS6dlobo6lnTcaHZ50J9uUwmx4Tw+g0lVE4J81q3TXnjIl+ZRaiAQ+TU3O4Lexcpv4\nJ/OpejzG+jO7fRIeLYFPavKh3t5k4qSpU9OvEX/aH8u5AcArKiLwn3PO+Neez9eyZRG9844vsZ2v\n8XY1n3pqRL/7nbNK4KpVudvh5sm3q0t65BHnW8Hq1f2aMcPdOfKdzpZ5fGZPQW+v865ecu753r3J\nXP2pUwFPPTWiPXuceuedF0nr9h+tTfFrRqPk6AdQmSoi8O/dG9TcuaV5muvq8mvhwlhiO98Fg8bb\n1fzUUzXy+6sS2/F36pncPPk+8kiN3ngjGekyz5XrHPl2k2c7PvUcO3YEEwP6nnqqJvF77+z0D73D\nd/7v/PLLgcRYgsceS9Zz06Z4z8G+fXT5A6hMFRH433mndNd+7TXpW9+qlyRdeeUxzZ+f3/kOHZL+\n6Z+c/vNPfarX9fn6+qQdO+JPtrkDu5un8oEBqasrffpdqlyJdnJNFXTbE5CZbCfzuJFEo1I4HB+0\nkBx40Ncn/eY3Tk/ImjXuVzsEgEpVEYH/8cdrtG5daZ74f/CDeh04EEhsr1nTldf5/v3fp6ijI5DY\n/rM/63F9rJvBem6eypcujSaenpcuHT5qP9d0vlxTBcfbE5B5XOprhWXLIurqSn4pOHFCiemMp59+\nUr29znF79vgUDjvbO3cGtHy5u2szNQ9ApaqIwF/K968+n+T3Dya28+X3S1OnDia23Zo6NTkNcOrU\n/NpQXy9deOHJxLZbzgDD8S8NPNrKgZmvGFJfq6SuktjQkPyS8M47NZoxY+y/T6bmAahUFZH65oIL\nukt27Suv7FJ9fUz19TFdeWV+T/uStHZtj/r6BtXXN6i1a90/7S9a1K9HHw3o0UcDWrQo92DHqqoB\n/fM/1+if/7lGVVXZA1tTkzNYrrfX2c7U1hbR0aPOugKpKyPmKp8xY0B79vi0Z48vEawjEWfg3J49\nzna262Y7Lq63V3riiaCeeCKo3l7nqbyhIaaGhvSpfatX92vRoohOOSWiurqYnngiqM7O9GNT2/PG\nG87/9u3zJ9o1GcTbN9naBaD8VETg/5//c4SsMBOso6NGH/nISX3kIyfV0THO+XEpfv/7qTrllEGd\ncsqgfv9794/u9903TX19PvX1+XTffdNy1vvJT+p04EBABw4E9JOf1GWtE185r7bW2c4UX0GwocHZ\nHq08PigvGvUl0uzGu/EPH06+x8+8brbj4uKvG/bvD+jFF4OJJ/T589PHEMyY4QxOnDcvpq4up/62\nbVPTjk1tz+7dAe3eHdDRo75EuyaDePsmW7sAlJ+KCPwAAMCdigj8n/rU0ZJdu729X3v3Vmnv3iq1\nt+efT6C9vU/PPBPQM88E1N7uPpHPunXHdehQlQ4dqtK6dcdz1lu/vkdz50Y1d25U69dnf5UQDA5o\n+/Zqbd9erWDQfVd/e3tEra1RtbZG0wbiZasf78bfvTv5rr62dkBPPx3Q008HVFs7MOy41O79trbs\n15LSXwN0dTld5I2NMc2e7dRft65v2LHx1wy1tTEtWBAd9sogrlRd7rleZQDAWFXE4L6///uZMuZI\nSa798MNTVVNTldgeT9a9VPfcM13RaFVie8WKY66O+9GPpquuLrn9vvdlP25w0K9PfrI/sS0NDyIP\nPTRVhw75E9uZP1O8Sz++3dTkRMBc8/uz1Y9340cizquAlSsjeuyxGh075lz3scdqNHt2LO04Sa4W\nB0qddfDOO77EPP7582OJAXvxNsfFXzNIzkDBXAP7SpXSl8GGAAqlIp74AQCAOxUR+D/2sdI87UvS\n4sV9+s1vgvrNb4JavDi/p31JMuaYurqctLnGuHval6QrrjimUGhAodCArrgi93EDAwPaunWKtm6d\nooFs2XkkrVrVp7feqtJbb1Vp1arhP1Ourv6uLmnbthpt21ajrpQJDk4KXWd0/qmnOvWXLYsoEIgp\nGEymOj7vvH5Nnz6g6dMHdN55/Yk6gUBMy5ZF0j63tUXSutxzvQZYvbpfDQ3OMc88ExjWtrhcXemZ\nXfvxevX1sURKX0bZAygnFdHV/73vhfT5z4dLcu1vfrNB/f2+xPb55+c33sDa6Ync+NZO1/vf7y74\n9/QEtWFDf2Jbyh6Ntm6tT3SDb91ar5tvHj4V8he/qNPUqVWJ7WXL0scC5Orqz5XqN1s64Xiq4+bm\nZKrj3l6/Vqxw5uL39jpd/qnpkFM/794dTFu2ODVlrzR8vv+2bTWJ3P6pbYvL1ZWeK40wKX0BlKuK\neOIHAADu+G+66aZStyFfN82adVB/8ielufjy5Sf0wgvVmjkzpi1bjmr27PzOt2xZv3buDCoUimnz\n5mMKhdwd19gYU19flaZPj6m9PZIzm+Fpp53U3r0BNTXFtHFj97CV9yRp4cKIDhzwq6kppg0bejQt\nIy1ArmvNnTugrq4qzZwZ0+rV/Ykle7OV19YOKhqtUnNzvaZPPya/P1k2Zcqg5s4d0LRpuT8vXBhV\nLJbc19w88s+fq22jyWxTPPtfrnKvmTZtmo4fzz2LBJMX9668TXP+MG8Zz7EV0dUfHZ5Ovmi6u5UY\nAd9dgASC9fVK5LgfS7rc3l5nVTrJee+e2g2eKhBQIoVtrrS6J05Ib73lS2y7deyY9MwzTsQ966zk\ncr7xJDqpIhGne/7kSamlxelqT112d8YMJxFPR4fTjqam9MQ82brmW1qSq/BlLsWbrQ2Z7cm2kFCu\nVwDZyvNdlhgAiqEiuvo3b3b5WDxB1377bZ/efttXkHbcf3+ddu4MaufOoO6/P3tmvWy2bZuqXbuC\n2rUrqG3bcmf8c3P+b397ml59NahXXw3q298engUwM2te3D331CfOfc89I39riZ9j377kSn/xMQJv\nvBHUI4/UDLvOSNnrUve9+GJwzFnuCpEZj+x6AMpBRQR+AADgTkUE/g9+sDQj+iXpfe8Lq7c3pt7e\nmN73vvzb8a539ejJJ/168km/3vUu94v0uJ1W+MEP9ujwYenwYWc7m/Xrj6u7u0rd3VVav374O8Bs\n0/Mk6ZJLulVdHVN1dUyXXJJ875Et2108y9/8+cmV9DKn82VeJzOzX6rUBX1aW5PHBYMDwxbkyWa0\nhYncILsegHJQEYH/pz8tXVf/D38YkvNr9A1t5+euuxoVifgUifh0112Nro+LTyvs7/fpm9/MvWjR\nL39Zp8ZGqbHR2c7moYfqNWdOTHPmxPTQQ8O77OPT8/z+Kj31VHJhol27anT22QM6++wB7dqVLM/W\nBR7P8nfBBUqMR4hP51uxIqreXv+w68Qz+x075tdjj6UviJS6oM/27VMT2w89NHxBnmxGW5jIjVwL\nBQHAZFIRgR8AALhTEdP5AoGDOvPM0ly8qalPv/3tVPn9g7r55rDa2/M73/LlfXrqqSmaPn1Qt912\nWK2tbo9zN61wtKl60uhT/sYybU8aeepb6pSizHqtrennW7Ag95S81CmGq1b1Kxodvj3SNEem540P\nU8LKF/euvHl+Ot/bb5fu2jNmSFOnJrfzFQhINTUjT7fL5sQJqatr9Cl4b78tPfxwtSRpzZoezZ07\nvE53t/QgLAR/AAAgAElEQVTGG7mnKOaattfbK732mtPoc85Jlnd2StY6UXrDBuea8alv/f2Sz+d0\nk3d1Sb/6ldO2dev61NSUPgWvs1Pq7HR+xsw0ucFgcjpfbW1yOx7oo1Hprbf8mjIlfapd5hS8SER6\n9llnZ3t77mmRo03dS93f1DSQeH3AND8ApVY1ODhY6jbka7Cqql/795dmgN/y5SEdP+4Eo2nTYnrl\nlfza8eEPz9S+fU7wnD8/qocecrcOwZo1jdq/3wkura0D+vWvD4/7/Bs2zEishtfWFtH996cnt7/h\nBmfaniQtXRpJpP296666tPJNm3pylsdT3jY3NysSeUfz5w/ovvucKYmStHhxZNiqgCPtT02h6yyv\n65Tv2eO86w+Hq1RXN6jTT3eW3I3PwU89rqEhlpb6t7U1mnMFwMzjMuf052pPtrrlrKWlRR0dHaVu\nBsaBe1feWlpaJKlqPMfyjh8AAA+piHf8R44c1HvfW5qLz5/vvOOvqRnU7beHtWRJfudbvvyEnn22\nRjNmxHTLLUeydsVnP87d2AA351++vF9/+ENQjY0xXX99l5qb0/fnGgOQa/xAtvJsKXvnz4+qs9On\nUCimdev6hnWzj7Q/9R19ajrfpUsj6u+vUigU0+LFUdXXj5x6d7TUv9mul21MQK72VNr4Ad4Tly/u\nXXnz/Dv+118v3bVfekmJZV5fekn66EfzO19nZzJNbWen++Pq66U5c0ZP9XvihBKvJnKNBYhGpb6+\nqsR2plxpf92MHxhJba30rndFE9u9vcmsfvH37fH9krMMb3xfb29yfMCHP9yXds6VKyNp79x7e53p\nf/Fj588fSOyPRqVZs2IKBJJphF98MahoVGpujiW+CAQCI7+vj0/ti0SkN9/06+BBn2bNimWvPAak\nBQaQL97x56m1NT6PX5JiebfjzDNDOnLEOd/MmTG98IK78432Xj7u/POdFMOSNGdOTI8+Ovz8F13U\noD17nHMtXBjRAw+kLzWc611+rvEDbt/xP/FEMO39uqScn48eVWJp4NbWqF5+OZB4/9/cPKAPfOCk\npOQ79dR37vH3/vFjV66MJPan/m5S3/mHwz7V1cXU3BxL2z/a+/p9+/zauTOg48d9mjYtpqVLo3m9\n4x9tbEGx8Z64fHHvyls+7/iL9sRvjGmX9ICku6213x0qu1vSSkkxSVdaa58xxsyR9G1Jv7TWbi1W\n+wAA8IKiBH5jTK2kb0janlJ2gaQ2a+0qY8xSSVslrZLzJeD7kha4Pf/ataVL2fvFL4b1d38XSmzn\n6847w7r66lBi263rr+/S1742I7Gdyx13hBOLCd1xR/bzb9lyVF/9akNiO9OGDT2JBX42bEim/b3l\nliO67rqZie2R6sdT2jY2Sj6fsx1P3Zu5ne3zhRdGEj0c7e0RtbVFtG2bs+/DH+5TJJLsDk/9rySt\nXh1J6+pP3V9f7zzRx7vy4+l7Z8/O3tU/mrlzBxSNKtHVn28q39TjSQsMYDyK0tVvjPHJ+ZJxraRD\n1trvGmO2SNobf6o3xrwi6Rxrbbcx5kJJf+LyiX+Q7qryRXdjeeP+lS/uXXmb9NP5rLUxa+3JjOI5\nkg6lfO4cKotz/QOtXZtH4/K0aZPznr+1NaRNm/I/3403Js93443uj9u6VTr11JBOPTWkrSN8Xfr5\nz53cA8uXh/Tzn2ev8/jjzliA888P6fHHh+8/cMB5b3/XXXU6cCBZ/tpr0mWXTddll03Xa68lyzs7\nnTn49903NTFgMX6Om25S4hy9vUpbUCfbcXGZdXPt6+py3ovv2uXX736XXrZvn1+9vekLCGVbUCgb\nt/UAYLKZTPP4qyTJGPOfJP03Z9Osc3PgCy+UbpGeH/84uUiPs52f++5Lns/ZdmfLllBiYZotW3If\nd801TsKh48d9uuaa7PU2b3YGAL79ti/xWiDV/fc7g/V27gwmuvAl6Y47pmvXrqB27QrqjjumJ8q3\nbZuaKN+2bWraOf7wByXO8eKLwbQFdbIdF5dZN9e+Rx6p0dGjPj39dLVeeSWYVnb0qE8vvhhMW0Ao\n24JC2bitBwCTTSmn83Uo/Qm/RdIBa+1uSQ+P7VQ18W6Pkit0O9yer6oqfTvXcX5/sq7fn71edbWT\nRje+nVln+vRkJrrp06WWFmdswZQpyRS5U6bUqKXFmVfY0JCcYtjQILW0zMw4xwy1tMxQU5MST+9N\nTdmPi8usm9rE1H3BoNTc7Bzv90szZybLJKmnR6ob+u7S2Jg8Jv4516+/v99dPS+YLP/2MHbcO28q\nReCPh6jtkm6SdK8x5ixJ+6217hegT3HmmR0q1auqiy9W4kn/4ovDebfj0kuVeNK/9FL357vxRiWe\n9G+8MfdxX/+6Ek/6X/969npf+5oST/pf+9rwOh/9qHTsWN3Qdk9i/xVXKPGkf8UVxxLlF14oHT06\ndWi7Tx0dyXNMnz5DH/2oc/9aWqTOTieatrRE1NAw/Li4zLq59i1bFlFXl1+nneYMsAsEkmVO3WQe\n/fggw/jAQJ9vIOfv0edzV6/S8Z64fHHvyls+X9qKNbjvPZJ+IKlZUlTSYUkXStos6QJJA5K+aK19\naRynZ3BfGeOPT3nj/pUv7l15y2dwX0Uk8Jk7t0PPPluai19+ubRtm/N0vG5dWN/9bn7n27QpvQfh\nrrvcHfezn0nXXuscd+utYX3iE9nrPfig9OUvO/Vuuy2sj398eJ2f/zzZK3D77eFh2Qhfeklp0/1O\nP90p37FDaVMKly1zyl97LdkTsHnzMZ12WrJsypR6XXFFh047TdqzR/r2t508v1/60nE1NkqPPFIj\nSVq9ul/Hjkn33OP0/X/mM93y+7NnsEvN+NfWlj7tr7Y2e/a7zDLJ+XzihNNTIDkZ/eKr+8X3j3b9\nkVb4k8o/Ex/Bo3xx78qb5wN/JWXuG+/5Fi8OqafHOa6uLqZdu7Ift3RpSN3dTr36+ph27hxeb7QV\nB3Nl9suVPfCyy6anrap3773HEmXBYI0WLuzWvfce05VXTtOrrzr1liyJ6H3vO6k33nA+L1oU0TPP\nBBMZAFtbo/rMZ5ycw5kZ7FIzAGZm+EvN0pd6bGaZc6xPL70UUE9P1dDvNbm6X3z/aNcfaYU/afJl\n4hsrgkf54t6Vt0k/nQ8AAEwOFbE63/e/f1Cf+1xpLv7663169dWpkga1bl0475wCBw706Q9/cM53\n8cVhfeAD7o5raXFWCayuHtStt4a1fHn2eqeckqx3221hLV06vM5oKw4uX35CL7xQrZkzY9qy5ahm\nz46XZ1/Vb9myfu3cGVQoFNPmzccUCiXL5s6t1lVXdSoUkpYsOandu51V/770peNatmxAXV1Vmjkz\nptWr+/XudydXBfzCF7pVW5t9xbvGxuQKe6tW9SsaTV9tL9vKepll06Y5n2fPHlBt7aCam9NX94vv\nH+36I63wJ42+yt9kxwpv5Yt7V97yWZ2vIrr66a4qX3Q3ljfuX/ni3pU3uvoBAIArFRH4W1tLd+01\na5Ipdtesyf98Gzcmz7dxo/vjPv3p5HGf/nTuet/5TrLed76Tvc73vifNmxfSvHkhfe97w/c//bS0\nZk2j1qxp1NNPJ8vHksp3xw5nMOCHPuRsS86o/iuvnKYrr5ymPXukN9+UbrihXjfcUK8330xPxfvS\nS+nnzJamd98+vzo7pW3barRtW426htYuSk2329WVO/WvG6TuBVBuKqKrn1H97o9zU2/evJBiMaeO\nzxfTm2+m11mzplH79/uHzjegX//6sCQn6MdH3S9dGtGmTU4+pmyj+uMzAAKBai1Y0KP77+8aNqp/\n2rTBtPOtXdufGC1///01Cof9iXNeemlfYl8gENPChc7I+6efDujYMafeokURrVvXnzaSfs8eJ82x\n87OMPAI/m3IflZ8vuovLF/euvNHVDwAAXKmQwF+ap31JWro0LCkmKTa0nZ+1a5Pnc7bdef/7k8c5\n29ldc02ynrM93HXXheXzxeTzxXTddcPr3HbbYbW2Dqi1dUC33XY4Ub5hQ4+WLo1o6dKINmxIZl/e\nvPmYFi+OaPHiiDZvPibJSfDT1hbRkiXOtuQk7VmyJKIlSyL60peOa+PG7sT5Nm7sVnt7RK2tUbW2\nRrVly9G0c6buW726Xw0NMTU0xLRuXZ8WLYpo0aKIVq/ul+QkyonvX726P3Fce/vY++pTzxVP7AMA\nk1lFdPXTXVW+6G4sb9y/8sW9K2909QMAAFcqIvCXclR/a2tylHwh2nHGGcnznXGG++Pczga4/PJk\nvcsvz15n61bp1FNDOvXUkLZuHb7fWqmtLaS2tpCsHb08c3S+lJwZcMYZSswMyJwV0NWVPiI/9fOe\nPdJ9903VffdNVWdn+qj+zs7hI/zfeMP5X+ZI/tQZAJFIcpR+av1IJP38uUb/Z7Z3vJgpAGAiVURX\nP6P6Czuq/9RTQ4mR7oFATHv3ptdpawupr8/ZP3VqTLt3h0csv+GG+rTR+Tff3J2YGeD3BzVnzgn9\n+teHh80KOO20aFqufkmJz3v3VqmmxunlWrw4one9K5o1P398hP/bbzvtmjMnljaSP3UGQGoO/tT6\nDQ0xdXT4Rs2/v21bTVp7163rz3oPRlNOMwXoLi5f3LvyRlc/AABwJVDqBhRG6Ub1O9cOFawdTU1h\ndXaGEtturV0b1i9+EUps57JuXThtGeFsvvrVsLZsCSW2M33962F95SuhxPZo5Rs3dieW0924sVuS\nMzPgy19uVHV1UH/zN87MgA0benT//XWJ7dTlbOMj8uM+/vF+Pfzw1KGfoy+t7oUXJpfiXbYsoq4u\nv+rrnaf5QEBavTqiHTvS90tKG5WfWn/u3AE1NSX35Rr9n9rGzPaORWo7mCkAoNAqoquf7qryRXdj\neeP+lS/uXXmjqx8AALhC4AcAwEMI/AAAeEhFDO5rbZX27y/dtVMH9+XbjvGe78wzpUOHnOOam8N6\n4YXs9YyRHnvMqXfeeeG0+fZxmzdL99/v1NmwIaw77kjf/73vSbfc4uy/7rqwPv95p/xnP5OuvdYp\nv/XWsD7xCaf88celzZud8jvuCOvcc505+/ffX6fp06WPflSaO3d4vQULlDbYLxpVYpDgJZd0a9eu\nGknOQLpjx7LvO+ecfu3dG1R3t3TkiE/V1dJ55/Wrt9cZ0NfUNKDOTr+iUaet0ah08KBv6PcY09Sp\nzgC7oDMWUL290osvBhWNSrNmxTRlSnIA3ptv+nXwoE+zZsU0b17ymEhEOnAgOYAwXg4ApVARg/uY\nx1/Yefyj1cm1et/ixSH19DjldXUx7drllJ9/fihtXvyjj4YTc/Zra2s1f36XNm3qGVZv3boTafP6\nu7qqEp+rq2M6+2wn4C5aFNEzzwSz7hsYGNTChTE991xAfX1VmjcvpunTB7RihRPpe3ul2lolrnvo\nkE89PVWSqlRXF9Ppp0fT5tI/8URQ+/cHFA5Xqa5uMLFfknbuDOj4cZ+mTYtp6dJo4phympc/VgwQ\nK1/cu/LG4D4AAOBKRXT1V9I8/vGer7k5nNbVn8t554XTuvqz2bAhnNbVn+m668JpXf1xt94aTuvq\nj7vjjnBaF75z3p6Urv6erPXG0tV/1ln9I3b1z5gRzdrV39bmdPXH5+0vWJC9qz8uPod/9uzhXf3x\n1wSzZsVyzsVnXj6AUquIrn66q8oX3Y3ljftXvrh35Y2ufgAA4AqBHwAADyHwAwDgIRUxuI95/O6P\nO+886Y9/dOotWBDWY48Nr/Pxj0tPP+3UWbEirAcfTN+/aZP04x87+y++OKy77nLKf/xj6brrnPJb\nbgnr4oud8n/7t/T5+f/5P0sPPih9+csh+XzS17/uXPNXv5Kuvtqpd+edzuC+O+6YLknavPmYTpyQ\nvvpVZ73dL3zhqH772+TCP6GQM7/e+V1EtH27s4DPBRf06be/napIRFqwIKqGBqmtbfgiPvF5/H19\n0quvBuT3O4MGZ8zI/nscaW5+6r54noBs9SYCOQMAjKYiBvcxj7+48/hz7V+0KKT+fqe8piamN95w\nyk8/PaSuLqd8xoyYXnoprKVLQ+ru9qmqKqi6un7t3BnWmWeGdOSIU2/mzJhWrIho1y4nci1eHFE4\n7NOePc7nwcGYTjvNGYm/dGlEa9f2a/9+53vsCy/4dfKkc57Dh6XGRme0fX19TKtXR3T0qNTgfH9Q\nIBDTwoWxxDz+V1/1q7PTr4aGQS1aFNG6ddlX2Rtpbn7qvniegGz1JkKxcwYwQKx8ce/KG4P7AACA\nKxXR1c88fvfHLVgQTuvqz2bFinBaV3+miy8Op3X1x91ySzitqz8u2zz+224Lp3T1O2V33hnOq6s/\n7pxz+oZ19S9cmOzqv/DC4V398Xn8ra3RtK7+XEaam5/6OZ4nIFu9iUDOAACjqYiufrqryhfdjeWN\n+1e+uHflja5+AADgCoEfAAAPIfADAOAhBH4AADyEwA8AgIcQ+AEA8JCKmMdPyl73x7mp9573SG+9\n5dQ55ZSwnnwyff/GjdIvfuHsX7s2rHvuccq/+U3prruc8k2bwvrv/z13eTy9r88n3XyzdPHF0s9/\nLl1zjVPv9tudefzxeftbthzVlCnJef1/9VfHtH17ch6/JN1zj/P5kku6tWtXjSRp+fJ+bdtWp0hE\nam8/qVAoOXdfSqbU7eyUfvWrKZKk97//RCKzXzTqZP0LBKT29kgiC99IRkubm7lfGv55zx6/duwI\nKBSK6ayz3F0XANyoiHn8pOydHCl7x1KeTO8bVE1Nv954I6zly0M6ftypN21aTEuXDiRS9C5cGFEo\nFEuk8O3vl+bNS6bslaSdO5191dUxnX22E0BfecWnwUGfDh+u0syZMX34wycTaXqlZErdH/5wijo6\nnO/BLS1Rvf/9JyVJhw751NNTpVBoUK2tUa1cGcn6e001WtrczP2Shn1+9NFqHTzoV11dTO3tEVfX\nLQXmgpcv7l15Yx4/AABwpSK6+knZO5bjRq93yinhtK7+TGvXhtO6+uM2bQqndemPVB5P7+t09Ttl\nt98eLnhX/0c+MnJXfzyl7sUX92bt6l+wIL2r343R0ua6Sfe7YsXJRFe/2+sCgBsV0dVPd1X5orux\nvHH/yhf3rrzR1Q8AAFwh8AMA4CEEfgAAPITADwCAhxD4AQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8A\nAA+piJS9rM5X2NX5RquTa/W+XKv2XXWV9JOfOOXr14d1993SvfdKN98cUlWVdP310mWXSY8+Kl19\ntVPvzjvDCoelL3/Z+XzbbWGFQsn9V18d1gsvJFP2Hj2aTO979dVHdfCgk7L3nHP6tXdvUOGw9B//\nUa3qaumDH+zRL39ZJ0lat65Hr7xSo4EBacmSqKqqpJ07A4pGpVAoppoaadasmKZMcdLpRiLSiy86\niwHlWq1vtNX5AKCUKiJlL6vzld/qfPPnhzQw4KzO5/f3a9++sFauDOnQIadec3NMR49K3d3O5/r6\nmBoalNjv98d0xhnJ1flefjmQWMmvvj6miy7qlyQNDAxq4cKYHnqoWkeO+NXYGNPhw1Jjo9OaqqqY\nli+PqaurSqGQc75w2Kfjx6tUUzOoBQsGVFc3qNNPj6qhIaaODp/27w8M/VzZV+sbbXW+SkLa1/LF\nvStvpOwFAACuVERXP6vzjeU4N/VGrpNr9b5cq/atXx9O6+qXpBtuCKd09Ttld94ZnrCu/osuihak\nq7+pKfn0nmvVvNFW5wOAUqqIrn66q8oX3Y3ljftXvrh35Y2ufgAA4AqBHwAADyHwAwDgIQR+AAA8\nhMAPAICHEPgBAPAQAj8AAB5C4AcAwEMI/AAAeAiBHwAADyHwAwDgIQR+AAA8hMAPAICHEPgBAPAQ\nAj8AAB4SKHUDCqG1Vdq/v3TXlkJDn8J5t2O853N7nJt6o9XJtf+ss6R33nHKZ88O67nnnPK/+Avp\n8ced8nPPDeunP5Wuukr6yU+csvXrpbvvlr73PemWW5yy664Lq7pa2rLF+fzVr4a1cKF09dXO54su\nCusf/9HZvv32sE45RbruupmSpEsvPaIHHpghSbr++i4tWybt2CF97WtO2ec/36Unn6yTJG3Y0KO5\nc6U335TuuadeJ05IS5dGNHOmtHRpVIGAdPCg8/141qyYBgelQ4d8CgSk9vaIgkHpwAG/olHnZw0E\npKamAXV2+iVJc+cOKBh09kUiTt3M8rjeXunFF4OKRp1rBTL+dQYCw4+LRKQ33/Tr4EGfZs2Kad68\n4ecFgFRVg4ODpW5Dvgarqvq1f3+4JBdvbQ0p2XESy7sd4z2f2+Pc1ButTq79YylPlgUlOfdv3ryQ\nYjGnns8Xk88nRaPO50AgpuZmKRx2Pp88GZPP52xPmxbTggUx7dsXSOybObNKktTWFtH993dpw4YZ\n2r3biYhVVYN697sHJDlBftOmHt1wQ7127gzq0CGf6usHdfbZEYVCMTU2xtTT45yrrs75t9LT41Mo\nFFNra1QtLTEdPerT2287bZkzJ6beXqm21vlpGxpimj/fuda+fX4dPeobVh73xBNB7d8fUDhcpbq6\nQTU3x9L2z5kTG3bcvn1+7dwZ0PHjPk2bFtPSpdFh551ILS0t6ujoKNr1UDjcu/LW0tIiSVXjOZau\nfgAAPKQiuvql0jztJ68dStku1fncHuem3mh1su+fPTuc1tUfd+654bSufklavz6c0tXvlF13XXhC\nuvrj/83V1S9JGzd265576rVgwdi7+iWpvt55Og8EpLa29K7+uFzbce3tkaHf3chd/anmzh1QNKpE\nV3+28wJAqoro6qe7qnzR3VjeuH/li3tX3ujqBwAArhD4AQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8A\nAA8h8AMA4CEEfgAAPITADwCAhxD4AQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8AAA8h8AMA4CEEfgAA\nPITADwCAhxD4AQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8AAA8h8AMA4CEEfgAAPITADwCAhxD4AQDw\nkECxLmSMaZf0gKS7rbXfHSq7W9JKSTFJX7LWPmuMWSHpc5KqJN1krX2zWG0EAKDSFeWJ3xhTK+kb\nkranlF0gqc1au0rSZyX97dCuz0v6gqSvSbqsGO0DAMAritXVf0LSWknvpJStkfSgJFlrd0pqMMbU\nSwpaayOSDkiaVaT2AQDgCUXp6rfWxiSdNMakFs+R9EzK50NDZT3GmBpJp0ja5+b8ra3S/v0FauwY\ntbZKUmjoUzjvdoz3fG6Pc1NvtDq59ucqP/NM6dAhp7y5OawXXpD+4i+kxx93ys49V/rpT6Xrr5f+\n1/9yyv76r8OaNUu6/Xbn8zXXhDV/vvTlLzufP/7xsB54wNm+9dawGhulq692Pv/VX4X1ox852+vX\nh/W//3dIsZj0gQ/0atkyadWqPv3iF3WSpHXrevTKKzU6ckR6662AqqqkefOiqq+XliyJSpJefdX5\nZ7JkSVTBoHTwoPN9edasmCIRZ7/fL61e3a8ZM6SuLumRR2okSeec06+9e4OSpPb2iGprs9+XSEQ6\ncMAvSWpqGlBnp1/RaHqdQECaO3dAwWD2c2SeZ7S6ALypaO/4XYj3Pnxf0ncl+SV9xd2hIUnhCWmU\nu2v7Urbzbcd4z+f2ODf1RquTa3/2cifo+1K2w0NB3ylztsNDQd8pi38BiH++/faQpk2Turudzz/8\nYbLutdeGNH26FA4n606Z4mzfdVdIgYBPsZj0wAP1+uQn+/Xww9WaOrVKkvTqq9O1fHlMv/tdUCdO\nOGVTplTr3e+Oav9+v6QqhcM+DQ5K+/f71dg4qJ4ep97evYM6fNincNinGTMGJUnr1vXrkUdq9MYb\nTsR97bWAFi6MJX5zK1dGsvy+nWB99KjT5o4On2prpbffTu+QmzPHOc/8+QNZz5F5ntHqAvCmUgb+\nDjlP+HEtkg5Ya3skXTqWEw0O1gwdXnyDg5kl+bVjvOdze5ybeqPVybV/LOXpZTVZyibKxD8Cf+EL\nYz+mpUD/9y3UecZ2zdL820P+uHfeVIrAXzX03+2SbpJ0rzHmLEn7h4L+eM8HAABGUTVYhMcsY8x7\nJP1AUrOkqKTDki6UtFnSBZIGJH3RWvvShDcGAAAPK0rgBwAAkwOZ+wAA8BACPwAAHkLgBwDAQwj8\nAAB4yGRK4DMiFvkpXy7u3ZXW2meMMXMkfVvSL621W0vWYKQZw7+9lXLW3fBL+o619vlStRlJY7h/\nq+SslRKUdKe19rlStRkOt387h8rnSHpO0ilD2XJzKosnfhb5KV8u7913hnbF5GRuxCQxxn973ZIu\nl/QtSecXuanIYoz3r2vo892SVhe3pcg0xr+dknSVpN+4OXdZBH6xyE85c33vrLUH5eR0wOQxlvv3\nBzlpGL8g6R+K3VBkNZb79/LQvlsl/UuxG4phXN87Y8wlkn42dMyoyiLwW2tj1tqTGcVz5CzsEzfu\nRX4wcVzeu06lp28mG+MkMZb7Z4yZLukOSddaa48Wq43IbSx/O40xZ1tr/03SejlPjyihMca9lZI+\nJOlMSZ8a7dxlEfhdylzk53pJf1+y1mAsqiTJGPOfJP03Z9OsK22TMAbxL2rXSJom6QZjzH8pYXsw\nNomVtYwx35fzqub/lLA9cM8nSdbaK6y1WyQ9L+mfRjuobAb3ZVGwRX5QdLnu3W5JD5emSRiDXPfv\nuhK1B2Mz0r+/X5amSXAp672Lf7DWfsbNScrxiT91kZ+/kKQ8F/lB8XDvyhv3r7xx/8pXQe9dWeTq\nZ5Gf8sW9K2/cv/LG/StfE3nvyiLwAwCAwijHrn4AADBOBH4AADyEwA8AgIcQ+AEA8BACPwAAHkLg\nBwDAQwj8AAB4SDmn7AWQwRgTk7Med8cYjtkjaYO19veFqJdxzEI52caOW2vPcnscgInDEz9QWSZb\nRq73SuoYS9A3xrA6IzCBeOIHKkvWoGmMmSpntcozJAUlPWCtvTqlyp8bY/5OUqOkf7DW3pDj/Fnr\nDQxPN6AAAAKjSURBVK2meLOkWkm7JV0iabGk2yVNM8Y8b639U2PMJyXdKMkvZ8GRy6y1e4wxX5Wz\n4MiZkn4s6VvGmBuHzlMjZw3yq6y1k+2LDVB2eOIHvOFySTOstUslnSXp08aYVSn7z7TW/qmksyVd\nbow5Pcd5htUb6s7/B0nrrbVtkv6fpO9ba5+QdK2kx4eC/nxJ90j6mLV2uaSHhj7HfVjSh6y13zLG\n/Fc5i5GcLelPhv73hUL8IgCvI/ADHmCt/Yakjw9td0l6WdKilCo/Gtp3SNJvJJ2b41TZ6n1I0v+z\n1u4YqnOPpI9l6bL/c0kPW2v3DH3+gaTVxpj436EnrbVHhrY/ImmrtbbbWhuTdJ+kT4ztpwaQDV39\ngAcYY9ok3W2MWSIpJukUSVtTqhxK2e6SNDPHqbLV80m60BjzylB5laQjkkIZxzYPlUuSrLXHhr4c\nNA0VHU6p2yBpkzFm49D5/JIOjvQzAnCHwA94w99JesZa+zFJMsb8LmN/Y8r2TKUH4dHqnZT0K2ut\nyaxsTFrRO0rpSTDGzJTzJaQzy3U6JG2z1n43RzsAjBNd/YA3NEt6XpKMMe+XdJqk+pT9nzLGVBlj\nZskZif9ojvNk1vutpF9KOn/oXb+MMecYY76V5dhfDdVbMPT585K2D3XlZ9om6b8ODUqUMWbj0Ht/\nAHniiR+oLIOS/jj0pF019PnDkr6m5Ej5ByXdJOlvjDHPD9V5auh/zZLuttbuzHHuzHqvSpIx5jJJ\n/2KMCUo6LunKzIOttfuNMZ+V9K/GmICkPZI2ZvshrLUPGmOWS3rOGDMo6XVJl4791wEgU9XgILNj\nAADwCrr6AQDwEAI/AAAeQuAHAMBDCPwAAHgIgR8AAA8h8AMA4CEEfgAAPITADwCAhxD4AQDwkP8f\naky9wnrSzJAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f936e249110>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#print imp_df[imp_df['Host'].isin(fraudtraffic)][\"IPadd\"].value_counts()[:10]\n",
    "#plt.hist(imp_df['Tsecs'].iloc[imp_df[\"IPadd\"]=='411.517.507.552'])\n",
    "#plt.subplot(411)\n",
    "#histdat = plt.hist(imp_df[imp_df[\"IPadd\"]=='411.517.507.552']['Tsecs'].values,bins=24)\n",
    "#times = imp_df[imp_df[\"Host\"].isin(fraudtraffic)]['Tsecs'].values\n",
    "times = imp_df[imp_df[\"Host\"]=='adsafeprotected.com']['Tsecs'].values\n",
    "diffs = np.array([times[i]-times[i-1] for i in range(1,len(times))])\n",
    "xcoords = diffs[:-1]\n",
    "ycoords = diffs[1:]\n",
    "\n",
    "fig,ax = plt.subplots(figsize=(8,8))\n",
    "#plt.subplot(131)\n",
    "plt.plot(np.append(xcoords,-10),np.append(ycoords,-10),'b.',alpha=.2)\n",
    "#plt.xlim(0,10**3)\n",
    "#plt.ylim(0,10**3)\n",
    "ax.set_xlim(-1e1,1e4)\n",
    "plt.xlim(-1e1,1e4)\n",
    "ax.set_xscale('log')\n",
    "plt.yscale('log')\n",
    "\n",
    "ax.set_xlim(-1e1,1e4)\n",
    "ax.set_ylim(0,10000)\n",
    "plt.xlim(0,10000)\n",
    "print plt.xlim()\n",
    "print plt.ylim()\n",
    "plt.ylabel('Lag after')\n",
    "plt.xlabel('Lag before')\n",
    "#plt.subplot(132)\n",
    "#plt.figure()\n",
    "'''myxlim = 30\n",
    "plt.hist(xcoords[xcoords<=myxlim],range(0,myxlim),normed=True,cumulative=True)\n",
    "import matplotlib.mlab as mlab\n",
    "#y=mlab.normpdf(range(0,myxlim),np.mean(xcoords[xcoords<=myxlim]),np.std(xcoords[xcoords<=myxlim])).cumsum()\n",
    "y=mlab.normpdf(range(0,np.max(xcoords)),np.mean(xcoords),np.std(xcoords)).cumsum()\n",
    "y/=y[-1]\n",
    "l=plt.plot(range(0,myxlim),y,'k--')\n",
    "plt.subplot(133)\n",
    "plt.hist(times,bins=96)\n",
    "plt.yscale('log')\n",
    "f = plt.gcf()\n",
    "f.set_size_inches(14,6)\n",
    "#import spectrum\n",
    "#p = spectrum.Periodogram()'''\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                Timestamp            IPadd  \\\n",
      "279   2014-08-25 00:02:08  574.491.567.341   \n",
      "311   2014-08-25 00:02:23  476.494.399.426   \n",
      "1186  2014-08-25 00:11:03  324.338.423.496   \n",
      "\n",
      "                                              UserAgent                Host  \\\n",
      "279   Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7....     rlinevideos.com   \n",
      "311   Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.3...    sprungliving.com   \n",
      "1186  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...  sweetxboxgames.com   \n",
      "\n",
      "      Inview  Nplugins        Wpossize  Latency  Fraud    UA_Browser_Ver  \\\n",
      "279        1       NaN  (0,0,1024,673)     3496      1             Other   \n",
      "311        0         3  (0,0,1024,706)        0      1  Chrome_35.0.1916   \n",
      "1186       0         1             NaN      NaN      1  Chrome_34.0.1847   \n",
      "\n",
      "     UA_Browser  Tsecs   Warea  WXpos  WYpos  Wwidth  Wheight  Browser_Chrome  \\\n",
      "279       Other    128  689152      0      0    1024      673               0   \n",
      "311      Chrome    143  722944      0      0    1024      706               1   \n",
      "1186     Chrome    663     NaN    NaN    NaN     NaN      NaN               1   \n",
      "\n",
      "      Browser_Firefox  Browser_Internet Explorer  Browser_Opera  \\\n",
      "279                 0                          1              0   \n",
      "311                 0                          0              0   \n",
      "1186                0                          0              0   \n",
      "\n",
      "      Browser_Other  Browser_Safari/Webkit  \n",
      "279               0                      0  \n",
      "311               0                      0  \n",
      "1186              0                      0  \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "324.338.423.496    344\n",
       "496.529.325.519    219\n",
       "574.491.567.341    208\n",
       "476.494.399.426    186\n",
       "529.366.487.475    156\n",
       "496.437.522.387    139\n",
       "574.452.484.501    136\n",
       "489.462.542.447     83\n",
       "496.325.356.347     76\n",
       "438.562.383.508     66\n",
       "476.391.537.495     58\n",
       "527.358.505.356     57\n",
       "324.525.562.389     54\n",
       "476.516.393.518     49\n",
       "529.324.449.376     47\n",
       "544.552.576.426     42\n",
       "446.488.377.562     42\n",
       "445.435.337.514     38\n",
       "445.438.339.329     33\n",
       "376.394.517.438     33\n",
       "476.356.528.363     33\n",
       "529.330.395.356     32\n",
       "412.498.354.369     31\n",
       "441.378.464.557     31\n",
       "518.368.391.462     30\n",
       "476.565.329.537     28\n",
       "525.537.550.349     27\n",
       "438.378.504.483     24\n",
       "525.470.434.544     19\n",
       "445.503.427.560     17\n",
       "                  ... \n",
       "496.361.565.416      9\n",
       "378.564.404.345      8\n",
       "445.555.342.420      7\n",
       "441.361.494.512      7\n",
       "452.434.392.353      6\n",
       "376.346.326.421      5\n",
       "476.516.393.572      5\n",
       "476.356.392.517      5\n",
       "324.453.396.518      4\n",
       "527.500.570.335      4\n",
       "496.438.406.528      3\n",
       "508.513.388.432      3\n",
       "445.566.570.515      3\n",
       "445.337.515.330      3\n",
       "446.530.381.455      3\n",
       "378.550.500.420      2\n",
       "547.467.380.356      2\n",
       "408.326.390.347      1\n",
       "544.394.560.398      1\n",
       "459.397.334.462      1\n",
       "325.531.380.407      1\n",
       "393.414.443.469      1\n",
       "381.463.412.504      1\n",
       "551.371.369.448      1\n",
       "445.324.461.380      1\n",
       "458.346.444.349      1\n",
       "378.576.348.399      1\n",
       "446.398.338.381      1\n",
       "512.533.327.391      1\n",
       "551.388.402.448      1\n",
       "Name: IPadd, dtype: int64"
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Split the data by fraud condition\n",
    "fdata = dum_imp_df[dum_imp_df[\"Host\"].isin(fraudtraffic)]\n",
    "nfdata = dum_imp_df[~dum_imp_df[\"Host\"].isin(fraudtraffic)]\n",
    "print fdata.head(3)\n",
    "fdata['IPadd'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Tsecs</th>\n",
       "      <th>Warea</th>\n",
       "      <th>WXpos</th>\n",
       "      <th>WYpos</th>\n",
       "      <th>Wwidth</th>\n",
       "      <th>Wheight</th>\n",
       "      <th>Browser_Chrome</th>\n",
       "      <th>Browser_Firefox</th>\n",
       "      <th>Browser_Internet Explorer</th>\n",
       "      <th>Browser_Opera</th>\n",
       "      <th>Browser_Other</th>\n",
       "      <th>Browser_Safari/Webkit</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fraud</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>53</td>\n",
       "      <td>50461</td>\n",
       "      <td>925056</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1296</td>\n",
       "      <td>730</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>127</td>\n",
       "      <td>49092</td>\n",
       "      <td>723968</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1024</td>\n",
       "      <td>706</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Inview  Nplugins  Latency  Tsecs   Warea  WXpos  WYpos  Wwidth  \\\n",
       "Fraud                                                                   \n",
       "0           0        10       53  50461  925056      0      0    1296   \n",
       "1           1         3      127  49092  723968      0      0    1024   \n",
       "\n",
       "       Wheight  Browser_Chrome  Browser_Firefox  Browser_Internet Explorer  \\\n",
       "Fraud                                                                        \n",
       "0          730               0                0                          0   \n",
       "1          706               0                0                          0   \n",
       "\n",
       "       Browser_Opera  Browser_Other  Browser_Safari/Webkit  \n",
       "Fraud                                                       \n",
       "0                  0              0                      0  \n",
       "1                  0              0                      0  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dum_imp_df.groupby('Fraud').median()\n",
    "#imp_df.Host.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Tsecs</th>\n",
       "      <th>Warea</th>\n",
       "      <th>WXpos</th>\n",
       "      <th>WYpos</th>\n",
       "      <th>Wwidth</th>\n",
       "      <th>Wheight</th>\n",
       "      <th>Browser_Chrome</th>\n",
       "      <th>Browser_Firefox</th>\n",
       "      <th>Browser_Internet Explorer</th>\n",
       "      <th>Browser_Opera</th>\n",
       "      <th>Browser_Other</th>\n",
       "      <th>Browser_Safari/Webkit</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fraud</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.428204</td>\n",
       "      <td>10.247782</td>\n",
       "      <td>283.620772</td>\n",
       "      <td>48853.426236</td>\n",
       "      <td>4.119939e+13</td>\n",
       "      <td>30754.749088</td>\n",
       "      <td>30747.681905</td>\n",
       "      <td>27810.803089</td>\n",
       "      <td>27278.966099</td>\n",
       "      <td>0.364143</td>\n",
       "      <td>0.135789</td>\n",
       "      <td>0.279755</td>\n",
       "      <td>0.000353</td>\n",
       "      <td>0.014416</td>\n",
       "      <td>0.205543</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.749683</td>\n",
       "      <td>5.516657</td>\n",
       "      <td>2381.751429</td>\n",
       "      <td>47430.848777</td>\n",
       "      <td>8.635001e+05</td>\n",
       "      <td>-0.079012</td>\n",
       "      <td>-0.079012</td>\n",
       "      <td>1158.570370</td>\n",
       "      <td>701.706173</td>\n",
       "      <td>0.469314</td>\n",
       "      <td>0.067389</td>\n",
       "      <td>0.456077</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.002006</td>\n",
       "      <td>0.005215</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Inview   Nplugins      Latency         Tsecs         Warea  \\\n",
       "Fraud                                                                 \n",
       "0      0.428204  10.247782   283.620772  48853.426236  4.119939e+13   \n",
       "1      0.749683   5.516657  2381.751429  47430.848777  8.635001e+05   \n",
       "\n",
       "              WXpos         WYpos        Wwidth       Wheight  Browser_Chrome  \\\n",
       "Fraud                                                                           \n",
       "0      30754.749088  30747.681905  27810.803089  27278.966099        0.364143   \n",
       "1         -0.079012     -0.079012   1158.570370    701.706173        0.469314   \n",
       "\n",
       "       Browser_Firefox  Browser_Internet Explorer  Browser_Opera  \\\n",
       "Fraud                                                              \n",
       "0             0.135789                   0.279755       0.000353   \n",
       "1             0.067389                   0.456077       0.000000   \n",
       "\n",
       "       Browser_Other  Browser_Safari/Webkit  \n",
       "Fraud                                        \n",
       "0           0.014416               0.205543  \n",
       "1           0.002006               0.005215  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dum_imp_df.groupby('Fraud').mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Fraud</th>\n",
       "      <th>Tsecs</th>\n",
       "      <th>Warea</th>\n",
       "      <th>WXpos</th>\n",
       "      <th>WYpos</th>\n",
       "      <th>Wwidth</th>\n",
       "      <th>Wheight</th>\n",
       "      <th>Browser_Chrome</th>\n",
       "      <th>Browser_Firefox</th>\n",
       "      <th>Browser_Internet Explorer</th>\n",
       "      <th>Browser_Opera</th>\n",
       "      <th>Browser_Other</th>\n",
       "      <th>Browser_Safari/Webkit</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Host</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bumclub.info</th>\n",
       "      <td>0.058824</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.823529</td>\n",
       "      <td>1</td>\n",
       "      <td>52148.508475</td>\n",
       "      <td>950340.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1348.000000</td>\n",
       "      <td>705.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>featureball.com</th>\n",
       "      <td>0.957066</td>\n",
       "      <td>6.254973</td>\n",
       "      <td>2985.175141</td>\n",
       "      <td>1</td>\n",
       "      <td>41581.533929</td>\n",
       "      <td>761686.504604</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1084.283610</td>\n",
       "      <td>678.399632</td>\n",
       "      <td>0.701786</td>\n",
       "      <td>0.126786</td>\n",
       "      <td>0.158929</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.012500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hourlyparent.com</th>\n",
       "      <td>0.455696</td>\n",
       "      <td>6.305556</td>\n",
       "      <td>2973.594937</td>\n",
       "      <td>1</td>\n",
       "      <td>43491.588235</td>\n",
       "      <td>807671.312500</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1148.812500</td>\n",
       "      <td>688.437500</td>\n",
       "      <td>0.364706</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.635294</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hunp.us</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>41193.137931</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.879310</td>\n",
       "      <td>0.120690</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>indoorlife.tv</th>\n",
       "      <td>0.929825</td>\n",
       "      <td>2.804348</td>\n",
       "      <td>346.107143</td>\n",
       "      <td>1</td>\n",
       "      <td>39963.216667</td>\n",
       "      <td>1280151.018868</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1423.075472</td>\n",
       "      <td>879.415094</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.950000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.050000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mammabay.couk</th>\n",
       "      <td>0.452991</td>\n",
       "      <td>3.625000</td>\n",
       "      <td>560.800000</td>\n",
       "      <td>1</td>\n",
       "      <td>49841.934959</td>\n",
       "      <td>1007426.715517</td>\n",
       "      <td>-0.068966</td>\n",
       "      <td>-0.068966</td>\n",
       "      <td>1361.862069</td>\n",
       "      <td>738.112069</td>\n",
       "      <td>0.065041</td>\n",
       "      <td>0.024390</td>\n",
       "      <td>0.910569</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>myhomesdesign.com</th>\n",
       "      <td>0.650794</td>\n",
       "      <td>1.050000</td>\n",
       "      <td>856.419355</td>\n",
       "      <td>1</td>\n",
       "      <td>52933.173913</td>\n",
       "      <td>1129792.390244</td>\n",
       "      <td>-1.170732</td>\n",
       "      <td>-1.170732</td>\n",
       "      <td>1497.268293</td>\n",
       "      <td>748.780488</td>\n",
       "      <td>0.608696</td>\n",
       "      <td>0.014493</td>\n",
       "      <td>0.362319</td>\n",
       "      <td>0</td>\n",
       "      <td>0.014493</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>psychoworld.tv</th>\n",
       "      <td>0.473684</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>321.487179</td>\n",
       "      <td>1</td>\n",
       "      <td>48530.894737</td>\n",
       "      <td>1947836.717949</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1780.871795</td>\n",
       "      <td>1017.025641</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rlinevideos.com</th>\n",
       "      <td>0.545455</td>\n",
       "      <td>3.450000</td>\n",
       "      <td>1082.181818</td>\n",
       "      <td>1</td>\n",
       "      <td>66072.018182</td>\n",
       "      <td>455706.036364</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>631.127273</td>\n",
       "      <td>412.145455</td>\n",
       "      <td>0.145455</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.854545</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sprungliving.com</th>\n",
       "      <td>0.634361</td>\n",
       "      <td>6.030769</td>\n",
       "      <td>682.125561</td>\n",
       "      <td>1</td>\n",
       "      <td>59687.820175</td>\n",
       "      <td>974671.718182</td>\n",
       "      <td>-0.472727</td>\n",
       "      <td>-0.472727</td>\n",
       "      <td>1311.695455</td>\n",
       "      <td>721.022727</td>\n",
       "      <td>0.179825</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.820175</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sweetxboxgames.com</th>\n",
       "      <td>0.163462</td>\n",
       "      <td>1.043478</td>\n",
       "      <td>3135.730769</td>\n",
       "      <td>1</td>\n",
       "      <td>63020.873333</td>\n",
       "      <td>278748.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>419.538462</td>\n",
       "      <td>204.461538</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ulterior-movies.com</th>\n",
       "      <td>0.951807</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2255.543210</td>\n",
       "      <td>1</td>\n",
       "      <td>52502.228916</td>\n",
       "      <td>920651.228916</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1221.710843</td>\n",
       "      <td>741.349398</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uvido.com</th>\n",
       "      <td>0.912664</td>\n",
       "      <td>7.630394</td>\n",
       "      <td>3646.675549</td>\n",
       "      <td>1</td>\n",
       "      <td>41286.648844</td>\n",
       "      <td>770641.558422</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1085.352049</td>\n",
       "      <td>693.772382</td>\n",
       "      <td>0.554913</td>\n",
       "      <td>0.124277</td>\n",
       "      <td>0.315029</td>\n",
       "      <td>0</td>\n",
       "      <td>0.001445</td>\n",
       "      <td>0.004335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>workingfathertv.com</th>\n",
       "      <td>0.865546</td>\n",
       "      <td>2.895238</td>\n",
       "      <td>123.091743</td>\n",
       "      <td>1</td>\n",
       "      <td>43133.783333</td>\n",
       "      <td>1245167.302752</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1424.532110</td>\n",
       "      <td>857.605505</td>\n",
       "      <td>0.125000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.850000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.025000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>worsthorrorgame.com</th>\n",
       "      <td>0.368421</td>\n",
       "      <td>1.014706</td>\n",
       "      <td>6700.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>64358.319149</td>\n",
       "      <td>448704.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>684.000000</td>\n",
       "      <td>328.000000</td>\n",
       "      <td>0.819149</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.180851</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       Inview  Nplugins      Latency  Fraud         Tsecs  \\\n",
       "Host                                                                        \n",
       "bumclub.info         0.058824       NaN    22.823529      1  52148.508475   \n",
       "featureball.com      0.957066  6.254973  2985.175141      1  41581.533929   \n",
       "hourlyparent.com     0.455696  6.305556  2973.594937      1  43491.588235   \n",
       "hunp.us              0.000000       NaN     0.000000      1  41193.137931   \n",
       "indoorlife.tv        0.929825  2.804348   346.107143      1  39963.216667   \n",
       "mammabay.couk        0.452991  3.625000   560.800000      1  49841.934959   \n",
       "myhomesdesign.com    0.650794  1.050000   856.419355      1  52933.173913   \n",
       "psychoworld.tv       0.473684  3.000000   321.487179      1  48530.894737   \n",
       "rlinevideos.com      0.545455  3.450000  1082.181818      1  66072.018182   \n",
       "sprungliving.com     0.634361  6.030769   682.125561      1  59687.820175   \n",
       "sweetxboxgames.com   0.163462  1.043478  3135.730769      1  63020.873333   \n",
       "ulterior-movies.com  0.951807  2.000000  2255.543210      1  52502.228916   \n",
       "uvido.com            0.912664  7.630394  3646.675549      1  41286.648844   \n",
       "workingfathertv.com  0.865546  2.895238   123.091743      1  43133.783333   \n",
       "worsthorrorgame.com  0.368421  1.014706  6700.000000      1  64358.319149   \n",
       "\n",
       "                              Warea     WXpos     WYpos       Wwidth  \\\n",
       "Host                                                                   \n",
       "bumclub.info          950340.000000  0.000000  0.000000  1348.000000   \n",
       "featureball.com       761686.504604  0.000000  0.000000  1084.283610   \n",
       "hourlyparent.com      807671.312500  0.000000  0.000000  1148.812500   \n",
       "hunp.us                    0.000000  0.000000  0.000000     0.000000   \n",
       "indoorlife.tv        1280151.018868  0.000000  0.000000  1423.075472   \n",
       "mammabay.couk        1007426.715517 -0.068966 -0.068966  1361.862069   \n",
       "myhomesdesign.com    1129792.390244 -1.170732 -1.170732  1497.268293   \n",
       "psychoworld.tv       1947836.717949  0.000000  0.000000  1780.871795   \n",
       "rlinevideos.com       455706.036364  0.000000  0.000000   631.127273   \n",
       "sprungliving.com      974671.718182 -0.472727 -0.472727  1311.695455   \n",
       "sweetxboxgames.com    278748.000000  0.000000  0.000000   419.538462   \n",
       "ulterior-movies.com   920651.228916  0.000000  0.000000  1221.710843   \n",
       "uvido.com             770641.558422  0.000000  0.000000  1085.352049   \n",
       "workingfathertv.com  1245167.302752  0.000000  0.000000  1424.532110   \n",
       "worsthorrorgame.com   448704.000000  0.000000  0.000000   684.000000   \n",
       "\n",
       "                         Wheight  Browser_Chrome  Browser_Firefox  \\\n",
       "Host                                                                \n",
       "bumclub.info          705.000000        0.000000         0.000000   \n",
       "featureball.com       678.399632        0.701786         0.126786   \n",
       "hourlyparent.com      688.437500        0.364706         0.000000   \n",
       "hunp.us                 0.000000        0.879310         0.120690   \n",
       "indoorlife.tv         879.415094        0.000000         0.000000   \n",
       "mammabay.couk         738.112069        0.065041         0.024390   \n",
       "myhomesdesign.com     748.780488        0.608696         0.014493   \n",
       "psychoworld.tv       1017.025641        0.000000         0.000000   \n",
       "rlinevideos.com       412.145455        0.145455         0.000000   \n",
       "sprungliving.com      721.022727        0.179825         0.000000   \n",
       "sweetxboxgames.com    204.461538        0.800000         0.000000   \n",
       "ulterior-movies.com   741.349398        0.000000         0.000000   \n",
       "uvido.com             693.772382        0.554913         0.124277   \n",
       "workingfathertv.com   857.605505        0.125000         0.000000   \n",
       "worsthorrorgame.com   328.000000        0.819149         0.000000   \n",
       "\n",
       "                     Browser_Internet Explorer  Browser_Opera  Browser_Other  \\\n",
       "Host                                                                           \n",
       "bumclub.info                          1.000000              0       0.000000   \n",
       "featureball.com                       0.158929              0       0.000000   \n",
       "hourlyparent.com                      0.635294              0       0.000000   \n",
       "hunp.us                               0.000000              0       0.000000   \n",
       "indoorlife.tv                         0.950000              0       0.050000   \n",
       "mammabay.couk                         0.910569              0       0.000000   \n",
       "myhomesdesign.com                     0.362319              0       0.014493   \n",
       "psychoworld.tv                        1.000000              0       0.000000   \n",
       "rlinevideos.com                       0.854545              0       0.000000   \n",
       "sprungliving.com                      0.820175              0       0.000000   \n",
       "sweetxboxgames.com                    0.200000              0       0.000000   \n",
       "ulterior-movies.com                   1.000000              0       0.000000   \n",
       "uvido.com                             0.315029              0       0.001445   \n",
       "workingfathertv.com                   0.850000              0       0.000000   \n",
       "worsthorrorgame.com                   0.180851              0       0.000000   \n",
       "\n",
       "                     Browser_Safari/Webkit  \n",
       "Host                                        \n",
       "bumclub.info                      0.000000  \n",
       "featureball.com                   0.012500  \n",
       "hourlyparent.com                  0.000000  \n",
       "hunp.us                           0.000000  \n",
       "indoorlife.tv                     0.000000  \n",
       "mammabay.couk                     0.000000  \n",
       "myhomesdesign.com                 0.000000  \n",
       "psychoworld.tv                    0.000000  \n",
       "rlinevideos.com                   0.000000  \n",
       "sprungliving.com                  0.000000  \n",
       "sweetxboxgames.com                0.000000  \n",
       "ulterior-movies.com               0.000000  \n",
       "uvido.com                         0.004335  \n",
       "workingfathertv.com               0.025000  \n",
       "worsthorrorgame.com               0.000000  "
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fdata.groupby('Host').mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Fraud</th>\n",
       "      <th>Tsecs</th>\n",
       "      <th>Warea</th>\n",
       "      <th>WXpos</th>\n",
       "      <th>WYpos</th>\n",
       "      <th>Wwidth</th>\n",
       "      <th>Wheight</th>\n",
       "      <th>Browser_Chrome</th>\n",
       "      <th>Browser_Firefox</th>\n",
       "      <th>Browser_Internet Explorer</th>\n",
       "      <th>Browser_Opera</th>\n",
       "      <th>Browser_Other</th>\n",
       "      <th>Browser_Safari/Webkit</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Host</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bumclub.info</th>\n",
       "      <td>0.058824</td>\n",
       "      <td>3.623420</td>\n",
       "      <td>22.823529</td>\n",
       "      <td>1</td>\n",
       "      <td>52148.508475</td>\n",
       "      <td>950340.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1348.000000</td>\n",
       "      <td>705.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>featureball.com</th>\n",
       "      <td>0.957066</td>\n",
       "      <td>6.254973</td>\n",
       "      <td>2985.175141</td>\n",
       "      <td>1</td>\n",
       "      <td>41581.533929</td>\n",
       "      <td>761686.504604</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1084.283610</td>\n",
       "      <td>678.399632</td>\n",
       "      <td>0.701786</td>\n",
       "      <td>0.126786</td>\n",
       "      <td>0.158929</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.012500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hourlyparent.com</th>\n",
       "      <td>0.455696</td>\n",
       "      <td>6.305556</td>\n",
       "      <td>2973.594937</td>\n",
       "      <td>1</td>\n",
       "      <td>43491.588235</td>\n",
       "      <td>807671.312500</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1148.812500</td>\n",
       "      <td>688.437500</td>\n",
       "      <td>0.364706</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.635294</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hunp.us</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.623420</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>41193.137931</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.879310</td>\n",
       "      <td>0.120690</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>indoorlife.tv</th>\n",
       "      <td>0.929825</td>\n",
       "      <td>2.804348</td>\n",
       "      <td>346.107143</td>\n",
       "      <td>1</td>\n",
       "      <td>39963.216667</td>\n",
       "      <td>1280151.018868</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1423.075472</td>\n",
       "      <td>879.415094</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.950000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.050000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mammabay.couk</th>\n",
       "      <td>0.452991</td>\n",
       "      <td>3.625000</td>\n",
       "      <td>560.800000</td>\n",
       "      <td>1</td>\n",
       "      <td>49841.934959</td>\n",
       "      <td>1007426.715517</td>\n",
       "      <td>-0.068966</td>\n",
       "      <td>-0.068966</td>\n",
       "      <td>1361.862069</td>\n",
       "      <td>738.112069</td>\n",
       "      <td>0.065041</td>\n",
       "      <td>0.024390</td>\n",
       "      <td>0.910569</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>myhomesdesign.com</th>\n",
       "      <td>0.650794</td>\n",
       "      <td>1.050000</td>\n",
       "      <td>856.419355</td>\n",
       "      <td>1</td>\n",
       "      <td>52933.173913</td>\n",
       "      <td>1129792.390244</td>\n",
       "      <td>-1.170732</td>\n",
       "      <td>-1.170732</td>\n",
       "      <td>1497.268293</td>\n",
       "      <td>748.780488</td>\n",
       "      <td>0.608696</td>\n",
       "      <td>0.014493</td>\n",
       "      <td>0.362319</td>\n",
       "      <td>0</td>\n",
       "      <td>0.014493</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>psychoworld.tv</th>\n",
       "      <td>0.473684</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>321.487179</td>\n",
       "      <td>1</td>\n",
       "      <td>48530.894737</td>\n",
       "      <td>1947836.717949</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1780.871795</td>\n",
       "      <td>1017.025641</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rlinevideos.com</th>\n",
       "      <td>0.545455</td>\n",
       "      <td>3.450000</td>\n",
       "      <td>1082.181818</td>\n",
       "      <td>1</td>\n",
       "      <td>66072.018182</td>\n",
       "      <td>455706.036364</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>631.127273</td>\n",
       "      <td>412.145455</td>\n",
       "      <td>0.145455</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.854545</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sprungliving.com</th>\n",
       "      <td>0.634361</td>\n",
       "      <td>6.030769</td>\n",
       "      <td>682.125561</td>\n",
       "      <td>1</td>\n",
       "      <td>59687.820175</td>\n",
       "      <td>974671.718182</td>\n",
       "      <td>-0.472727</td>\n",
       "      <td>-0.472727</td>\n",
       "      <td>1311.695455</td>\n",
       "      <td>721.022727</td>\n",
       "      <td>0.179825</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.820175</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sweetxboxgames.com</th>\n",
       "      <td>0.163462</td>\n",
       "      <td>1.043478</td>\n",
       "      <td>3135.730769</td>\n",
       "      <td>1</td>\n",
       "      <td>63020.873333</td>\n",
       "      <td>278748.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>419.538462</td>\n",
       "      <td>204.461538</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ulterior-movies.com</th>\n",
       "      <td>0.951807</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2255.543210</td>\n",
       "      <td>1</td>\n",
       "      <td>52502.228916</td>\n",
       "      <td>920651.228916</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1221.710843</td>\n",
       "      <td>741.349398</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uvido.com</th>\n",
       "      <td>0.912664</td>\n",
       "      <td>7.630394</td>\n",
       "      <td>3646.675549</td>\n",
       "      <td>1</td>\n",
       "      <td>41286.648844</td>\n",
       "      <td>770641.558422</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1085.352049</td>\n",
       "      <td>693.772382</td>\n",
       "      <td>0.554913</td>\n",
       "      <td>0.124277</td>\n",
       "      <td>0.315029</td>\n",
       "      <td>0</td>\n",
       "      <td>0.001445</td>\n",
       "      <td>0.004335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>workingfathertv.com</th>\n",
       "      <td>0.865546</td>\n",
       "      <td>2.895238</td>\n",
       "      <td>123.091743</td>\n",
       "      <td>1</td>\n",
       "      <td>43133.783333</td>\n",
       "      <td>1245167.302752</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1424.532110</td>\n",
       "      <td>857.605505</td>\n",
       "      <td>0.125000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.850000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.025000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>worsthorrorgame.com</th>\n",
       "      <td>0.368421</td>\n",
       "      <td>1.014706</td>\n",
       "      <td>6700.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>64358.319149</td>\n",
       "      <td>448704.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>684.000000</td>\n",
       "      <td>328.000000</td>\n",
       "      <td>0.819149</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.180851</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       Inview  Nplugins      Latency  Fraud         Tsecs  \\\n",
       "Host                                                                        \n",
       "bumclub.info         0.058824  3.623420    22.823529      1  52148.508475   \n",
       "featureball.com      0.957066  6.254973  2985.175141      1  41581.533929   \n",
       "hourlyparent.com     0.455696  6.305556  2973.594937      1  43491.588235   \n",
       "hunp.us              0.000000  3.623420     0.000000      1  41193.137931   \n",
       "indoorlife.tv        0.929825  2.804348   346.107143      1  39963.216667   \n",
       "mammabay.couk        0.452991  3.625000   560.800000      1  49841.934959   \n",
       "myhomesdesign.com    0.650794  1.050000   856.419355      1  52933.173913   \n",
       "psychoworld.tv       0.473684  3.000000   321.487179      1  48530.894737   \n",
       "rlinevideos.com      0.545455  3.450000  1082.181818      1  66072.018182   \n",
       "sprungliving.com     0.634361  6.030769   682.125561      1  59687.820175   \n",
       "sweetxboxgames.com   0.163462  1.043478  3135.730769      1  63020.873333   \n",
       "ulterior-movies.com  0.951807  2.000000  2255.543210      1  52502.228916   \n",
       "uvido.com            0.912664  7.630394  3646.675549      1  41286.648844   \n",
       "workingfathertv.com  0.865546  2.895238   123.091743      1  43133.783333   \n",
       "worsthorrorgame.com  0.368421  1.014706  6700.000000      1  64358.319149   \n",
       "\n",
       "                              Warea     WXpos     WYpos       Wwidth  \\\n",
       "Host                                                                   \n",
       "bumclub.info          950340.000000  0.000000  0.000000  1348.000000   \n",
       "featureball.com       761686.504604  0.000000  0.000000  1084.283610   \n",
       "hourlyparent.com      807671.312500  0.000000  0.000000  1148.812500   \n",
       "hunp.us                    0.000000  0.000000  0.000000     0.000000   \n",
       "indoorlife.tv        1280151.018868  0.000000  0.000000  1423.075472   \n",
       "mammabay.couk        1007426.715517 -0.068966 -0.068966  1361.862069   \n",
       "myhomesdesign.com    1129792.390244 -1.170732 -1.170732  1497.268293   \n",
       "psychoworld.tv       1947836.717949  0.000000  0.000000  1780.871795   \n",
       "rlinevideos.com       455706.036364  0.000000  0.000000   631.127273   \n",
       "sprungliving.com      974671.718182 -0.472727 -0.472727  1311.695455   \n",
       "sweetxboxgames.com    278748.000000  0.000000  0.000000   419.538462   \n",
       "ulterior-movies.com   920651.228916  0.000000  0.000000  1221.710843   \n",
       "uvido.com             770641.558422  0.000000  0.000000  1085.352049   \n",
       "workingfathertv.com  1245167.302752  0.000000  0.000000  1424.532110   \n",
       "worsthorrorgame.com   448704.000000  0.000000  0.000000   684.000000   \n",
       "\n",
       "                         Wheight  Browser_Chrome  Browser_Firefox  \\\n",
       "Host                                                                \n",
       "bumclub.info          705.000000        0.000000         0.000000   \n",
       "featureball.com       678.399632        0.701786         0.126786   \n",
       "hourlyparent.com      688.437500        0.364706         0.000000   \n",
       "hunp.us                 0.000000        0.879310         0.120690   \n",
       "indoorlife.tv         879.415094        0.000000         0.000000   \n",
       "mammabay.couk         738.112069        0.065041         0.024390   \n",
       "myhomesdesign.com     748.780488        0.608696         0.014493   \n",
       "psychoworld.tv       1017.025641        0.000000         0.000000   \n",
       "rlinevideos.com       412.145455        0.145455         0.000000   \n",
       "sprungliving.com      721.022727        0.179825         0.000000   \n",
       "sweetxboxgames.com    204.461538        0.800000         0.000000   \n",
       "ulterior-movies.com   741.349398        0.000000         0.000000   \n",
       "uvido.com             693.772382        0.554913         0.124277   \n",
       "workingfathertv.com   857.605505        0.125000         0.000000   \n",
       "worsthorrorgame.com   328.000000        0.819149         0.000000   \n",
       "\n",
       "                     Browser_Internet Explorer  Browser_Opera  Browser_Other  \\\n",
       "Host                                                                           \n",
       "bumclub.info                          1.000000              0       0.000000   \n",
       "featureball.com                       0.158929              0       0.000000   \n",
       "hourlyparent.com                      0.635294              0       0.000000   \n",
       "hunp.us                               0.000000              0       0.000000   \n",
       "indoorlife.tv                         0.950000              0       0.050000   \n",
       "mammabay.couk                         0.910569              0       0.000000   \n",
       "myhomesdesign.com                     0.362319              0       0.014493   \n",
       "psychoworld.tv                        1.000000              0       0.000000   \n",
       "rlinevideos.com                       0.854545              0       0.000000   \n",
       "sprungliving.com                      0.820175              0       0.000000   \n",
       "sweetxboxgames.com                    0.200000              0       0.000000   \n",
       "ulterior-movies.com                   1.000000              0       0.000000   \n",
       "uvido.com                             0.315029              0       0.001445   \n",
       "workingfathertv.com                   0.850000              0       0.000000   \n",
       "worsthorrorgame.com                   0.180851              0       0.000000   \n",
       "\n",
       "                     Browser_Safari/Webkit  \n",
       "Host                                        \n",
       "bumclub.info                      0.000000  \n",
       "featureball.com                   0.012500  \n",
       "hourlyparent.com                  0.000000  \n",
       "hunp.us                           0.000000  \n",
       "indoorlife.tv                     0.000000  \n",
       "mammabay.couk                     0.000000  \n",
       "myhomesdesign.com                 0.000000  \n",
       "psychoworld.tv                    0.000000  \n",
       "rlinevideos.com                   0.000000  \n",
       "sprungliving.com                  0.000000  \n",
       "sweetxboxgames.com                0.000000  \n",
       "ulterior-movies.com               0.000000  \n",
       "uvido.com                         0.004335  \n",
       "workingfathertv.com               0.025000  \n",
       "worsthorrorgame.com               0.000000  "
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nplug_fillna_val = np.nanmean(fdata.groupby('Host')['Nplugins'].mean().values)\n",
    "fhost_df = fdata.groupby('Host').mean()\n",
    "fhost_df['Nplugins'].fillna(nplug_fillna_val,inplace=True)\n",
    "fhost_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Fraud</th>\n",
       "      <th>Tsecs</th>\n",
       "      <th>Warea</th>\n",
       "      <th>WXpos</th>\n",
       "      <th>WYpos</th>\n",
       "      <th>Wwidth</th>\n",
       "      <th>Wheight</th>\n",
       "      <th>Browser_Chrome</th>\n",
       "      <th>Browser_Firefox</th>\n",
       "      <th>Browser_Internet Explorer</th>\n",
       "      <th>Browser_Opera</th>\n",
       "      <th>Browser_Other</th>\n",
       "      <th>Browser_Safari/Webkit</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Host</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1001jogos.pt</th>\n",
       "      <td>0.153846</td>\n",
       "      <td>16.000000</td>\n",
       "      <td>230.750000</td>\n",
       "      <td>0</td>\n",
       "      <td>38640.769231</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10beauty.com</th>\n",
       "      <td>0.181818</td>\n",
       "      <td>NaN</td>\n",
       "      <td>199.636364</td>\n",
       "      <td>0</td>\n",
       "      <td>35338.272727</td>\n",
       "      <td>948000.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1174.000000</td>\n",
       "      <td>798.800000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123greetings.com</th>\n",
       "      <td>0.261905</td>\n",
       "      <td>9.939394</td>\n",
       "      <td>467.352941</td>\n",
       "      <td>0</td>\n",
       "      <td>43512.680851</td>\n",
       "      <td>1128893.058824</td>\n",
       "      <td>-1.411765</td>\n",
       "      <td>1.176471</td>\n",
       "      <td>1436.294118</td>\n",
       "      <td>763.352941</td>\n",
       "      <td>0.255319</td>\n",
       "      <td>0.255319</td>\n",
       "      <td>0.340426</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.148936</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Inview   Nplugins     Latency  Fraud         Tsecs  \\\n",
       "Host                                                                     \n",
       "1001jogos.pt      0.153846  16.000000  230.750000      0  38640.769231   \n",
       "10beauty.com      0.181818        NaN  199.636364      0  35338.272727   \n",
       "123greetings.com  0.261905   9.939394  467.352941      0  43512.680851   \n",
       "\n",
       "                           Warea     WXpos     WYpos       Wwidth     Wheight  \\\n",
       "Host                                                                            \n",
       "1001jogos.pt                 NaN       NaN       NaN          NaN         NaN   \n",
       "10beauty.com       948000.000000  0.000000  0.000000  1174.000000  798.800000   \n",
       "123greetings.com  1128893.058824 -1.411765  1.176471  1436.294118  763.352941   \n",
       "\n",
       "                  Browser_Chrome  Browser_Firefox  Browser_Internet Explorer  \\\n",
       "Host                                                                           \n",
       "1001jogos.pt            1.000000         0.000000                   0.000000   \n",
       "10beauty.com            0.000000         0.000000                   1.000000   \n",
       "123greetings.com        0.255319         0.255319                   0.340426   \n",
       "\n",
       "                  Browser_Opera  Browser_Other  Browser_Safari/Webkit  \n",
       "Host                                                                   \n",
       "1001jogos.pt                  0              0               0.000000  \n",
       "10beauty.com                  0              0               0.000000  \n",
       "123greetings.com              0              0               0.148936  "
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nfdata.groupby('Host').mean().head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Fraud</th>\n",
       "      <th>Tsecs</th>\n",
       "      <th>Warea</th>\n",
       "      <th>WXpos</th>\n",
       "      <th>WYpos</th>\n",
       "      <th>Wwidth</th>\n",
       "      <th>Wheight</th>\n",
       "      <th>Browser_Chrome</th>\n",
       "      <th>Browser_Firefox</th>\n",
       "      <th>Browser_Internet Explorer</th>\n",
       "      <th>Browser_Opera</th>\n",
       "      <th>Browser_Other</th>\n",
       "      <th>Browser_Safari/Webkit</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Host</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>123greetings.com</th>\n",
       "      <td>0.261905</td>\n",
       "      <td>9.939394</td>\n",
       "      <td>467.352941</td>\n",
       "      <td>0</td>\n",
       "      <td>43512.680851</td>\n",
       "      <td>1128893.058824</td>\n",
       "      <td>-1.411765</td>\n",
       "      <td>1.176471</td>\n",
       "      <td>1436.294118</td>\n",
       "      <td>763.352941</td>\n",
       "      <td>0.255319</td>\n",
       "      <td>0.255319</td>\n",
       "      <td>0.340426</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.148936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14news.com</th>\n",
       "      <td>0.428571</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>350.166667</td>\n",
       "      <td>0</td>\n",
       "      <td>23959.500000</td>\n",
       "      <td>814549.142857</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1126.000000</td>\n",
       "      <td>715.571429</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>192.com</th>\n",
       "      <td>0.428571</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>69.571429</td>\n",
       "      <td>0</td>\n",
       "      <td>60819.571429</td>\n",
       "      <td>870142.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1366.000000</td>\n",
       "      <td>637.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Inview   Nplugins     Latency  Fraud         Tsecs  \\\n",
       "Host                                                                     \n",
       "123greetings.com  0.261905   9.939394  467.352941      0  43512.680851   \n",
       "14news.com        0.428571  11.000000  350.166667      0  23959.500000   \n",
       "192.com           0.428571   2.000000   69.571429      0  60819.571429   \n",
       "\n",
       "                           Warea     WXpos     WYpos       Wwidth     Wheight  \\\n",
       "Host                                                                            \n",
       "123greetings.com  1128893.058824 -1.411765  1.176471  1436.294118  763.352941   \n",
       "14news.com         814549.142857  0.000000  0.000000  1126.000000  715.571429   \n",
       "192.com            870142.000000  0.000000  0.000000  1366.000000  637.000000   \n",
       "\n",
       "                  Browser_Chrome  Browser_Firefox  Browser_Internet Explorer  \\\n",
       "Host                                                                           \n",
       "123greetings.com        0.255319         0.255319                   0.340426   \n",
       "14news.com              0.800000         0.000000                   0.000000   \n",
       "192.com                 0.000000         0.000000                   1.000000   \n",
       "\n",
       "                  Browser_Opera  Browser_Other  Browser_Safari/Webkit  \n",
       "Host                                                                   \n",
       "123greetings.com              0              0               0.148936  \n",
       "14news.com                    0              0               0.200000  \n",
       "192.com                       0              0               0.000000  "
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nfdata.groupby('Host').mean().dropna().head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Fraud</th>\n",
       "      <th>Tsecs</th>\n",
       "      <th>Warea</th>\n",
       "      <th>WXpos</th>\n",
       "      <th>WYpos</th>\n",
       "      <th>Wwidth</th>\n",
       "      <th>Wheight</th>\n",
       "      <th>Browser_Chrome</th>\n",
       "      <th>Browser_Firefox</th>\n",
       "      <th>Browser_Internet Explorer</th>\n",
       "      <th>Browser_Opera</th>\n",
       "      <th>Browser_Other</th>\n",
       "      <th>Browser_Safari/Webkit</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Host</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bumclub.info</th>\n",
       "      <td>0.117647</td>\n",
       "      <td>7.246840</td>\n",
       "      <td>45.647059</td>\n",
       "      <td>2</td>\n",
       "      <td>104297.016949</td>\n",
       "      <td>1900680.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2696.000000</td>\n",
       "      <td>1410.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>featureball.com</th>\n",
       "      <td>1.914132</td>\n",
       "      <td>12.509946</td>\n",
       "      <td>5970.350282</td>\n",
       "      <td>2</td>\n",
       "      <td>83163.067857</td>\n",
       "      <td>1523373.009208</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2168.567219</td>\n",
       "      <td>1356.799263</td>\n",
       "      <td>1.403571</td>\n",
       "      <td>0.253571</td>\n",
       "      <td>0.317857</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.025000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hourlyparent.com</th>\n",
       "      <td>0.911392</td>\n",
       "      <td>12.611111</td>\n",
       "      <td>5947.189873</td>\n",
       "      <td>2</td>\n",
       "      <td>86983.176471</td>\n",
       "      <td>1615342.625000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2297.625000</td>\n",
       "      <td>1376.875000</td>\n",
       "      <td>0.729412</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.270588</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hunp.us</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.246840</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2</td>\n",
       "      <td>82386.275862</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.758621</td>\n",
       "      <td>0.241379</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>indoorlife.tv</th>\n",
       "      <td>1.859649</td>\n",
       "      <td>5.608696</td>\n",
       "      <td>692.214286</td>\n",
       "      <td>2</td>\n",
       "      <td>79926.433333</td>\n",
       "      <td>2560302.037736</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2846.150943</td>\n",
       "      <td>1758.830189</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.900000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mammabay.couk</th>\n",
       "      <td>0.905983</td>\n",
       "      <td>7.250000</td>\n",
       "      <td>1121.600000</td>\n",
       "      <td>2</td>\n",
       "      <td>99683.869919</td>\n",
       "      <td>2014853.431034</td>\n",
       "      <td>-0.137931</td>\n",
       "      <td>-0.137931</td>\n",
       "      <td>2723.724138</td>\n",
       "      <td>1476.224138</td>\n",
       "      <td>0.130081</td>\n",
       "      <td>0.048780</td>\n",
       "      <td>1.821138</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>myhomesdesign.com</th>\n",
       "      <td>1.301587</td>\n",
       "      <td>2.100000</td>\n",
       "      <td>1712.838710</td>\n",
       "      <td>2</td>\n",
       "      <td>105866.347826</td>\n",
       "      <td>2259584.780488</td>\n",
       "      <td>-2.341463</td>\n",
       "      <td>-2.341463</td>\n",
       "      <td>2994.536585</td>\n",
       "      <td>1497.560976</td>\n",
       "      <td>1.217391</td>\n",
       "      <td>0.028986</td>\n",
       "      <td>0.724638</td>\n",
       "      <td>0</td>\n",
       "      <td>0.028986</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>psychoworld.tv</th>\n",
       "      <td>0.947368</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>642.974359</td>\n",
       "      <td>2</td>\n",
       "      <td>97061.789474</td>\n",
       "      <td>3895673.435897</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3561.743590</td>\n",
       "      <td>2034.051282</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rlinevideos.com</th>\n",
       "      <td>1.090909</td>\n",
       "      <td>6.900000</td>\n",
       "      <td>2164.363636</td>\n",
       "      <td>2</td>\n",
       "      <td>132144.036364</td>\n",
       "      <td>911412.072727</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1262.254545</td>\n",
       "      <td>824.290909</td>\n",
       "      <td>0.290909</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.709091</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sprungliving.com</th>\n",
       "      <td>1.268722</td>\n",
       "      <td>12.061538</td>\n",
       "      <td>1364.251121</td>\n",
       "      <td>2</td>\n",
       "      <td>119375.640351</td>\n",
       "      <td>1949343.436364</td>\n",
       "      <td>-0.945455</td>\n",
       "      <td>-0.945455</td>\n",
       "      <td>2623.390909</td>\n",
       "      <td>1442.045455</td>\n",
       "      <td>0.359649</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.640351</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sweetxboxgames.com</th>\n",
       "      <td>0.326923</td>\n",
       "      <td>2.086957</td>\n",
       "      <td>6271.461538</td>\n",
       "      <td>2</td>\n",
       "      <td>126041.746667</td>\n",
       "      <td>557496.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>839.076923</td>\n",
       "      <td>408.923077</td>\n",
       "      <td>1.600000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ulterior-movies.com</th>\n",
       "      <td>1.903614</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4511.086420</td>\n",
       "      <td>2</td>\n",
       "      <td>105004.457831</td>\n",
       "      <td>1841302.457831</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2443.421687</td>\n",
       "      <td>1482.698795</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uvido.com</th>\n",
       "      <td>1.825328</td>\n",
       "      <td>15.260788</td>\n",
       "      <td>7293.351097</td>\n",
       "      <td>2</td>\n",
       "      <td>82573.297688</td>\n",
       "      <td>1541283.116844</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2170.704097</td>\n",
       "      <td>1387.544765</td>\n",
       "      <td>1.109827</td>\n",
       "      <td>0.248555</td>\n",
       "      <td>0.630058</td>\n",
       "      <td>0</td>\n",
       "      <td>0.002890</td>\n",
       "      <td>0.008671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>workingfathertv.com</th>\n",
       "      <td>1.731092</td>\n",
       "      <td>5.790476</td>\n",
       "      <td>246.183486</td>\n",
       "      <td>2</td>\n",
       "      <td>86267.566667</td>\n",
       "      <td>2490334.605505</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2849.064220</td>\n",
       "      <td>1715.211009</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.700000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.050000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>worsthorrorgame.com</th>\n",
       "      <td>0.736842</td>\n",
       "      <td>2.029412</td>\n",
       "      <td>13400.000000</td>\n",
       "      <td>2</td>\n",
       "      <td>128716.638298</td>\n",
       "      <td>897408.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1368.000000</td>\n",
       "      <td>656.000000</td>\n",
       "      <td>1.638298</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.361702</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       Inview   Nplugins       Latency  Fraud          Tsecs  \\\n",
       "Host                                                                           \n",
       "bumclub.info         0.117647   7.246840     45.647059      2  104297.016949   \n",
       "featureball.com      1.914132  12.509946   5970.350282      2   83163.067857   \n",
       "hourlyparent.com     0.911392  12.611111   5947.189873      2   86983.176471   \n",
       "hunp.us              0.000000   7.246840      0.000000      2   82386.275862   \n",
       "indoorlife.tv        1.859649   5.608696    692.214286      2   79926.433333   \n",
       "mammabay.couk        0.905983   7.250000   1121.600000      2   99683.869919   \n",
       "myhomesdesign.com    1.301587   2.100000   1712.838710      2  105866.347826   \n",
       "psychoworld.tv       0.947368   6.000000    642.974359      2   97061.789474   \n",
       "rlinevideos.com      1.090909   6.900000   2164.363636      2  132144.036364   \n",
       "sprungliving.com     1.268722  12.061538   1364.251121      2  119375.640351   \n",
       "sweetxboxgames.com   0.326923   2.086957   6271.461538      2  126041.746667   \n",
       "ulterior-movies.com  1.903614   4.000000   4511.086420      2  105004.457831   \n",
       "uvido.com            1.825328  15.260788   7293.351097      2   82573.297688   \n",
       "workingfathertv.com  1.731092   5.790476    246.183486      2   86267.566667   \n",
       "worsthorrorgame.com  0.736842   2.029412  13400.000000      2  128716.638298   \n",
       "\n",
       "                              Warea     WXpos     WYpos       Wwidth  \\\n",
       "Host                                                                   \n",
       "bumclub.info         1900680.000000  0.000000  0.000000  2696.000000   \n",
       "featureball.com      1523373.009208  0.000000  0.000000  2168.567219   \n",
       "hourlyparent.com     1615342.625000  0.000000  0.000000  2297.625000   \n",
       "hunp.us                    0.000000  0.000000  0.000000     0.000000   \n",
       "indoorlife.tv        2560302.037736  0.000000  0.000000  2846.150943   \n",
       "mammabay.couk        2014853.431034 -0.137931 -0.137931  2723.724138   \n",
       "myhomesdesign.com    2259584.780488 -2.341463 -2.341463  2994.536585   \n",
       "psychoworld.tv       3895673.435897  0.000000  0.000000  3561.743590   \n",
       "rlinevideos.com       911412.072727  0.000000  0.000000  1262.254545   \n",
       "sprungliving.com     1949343.436364 -0.945455 -0.945455  2623.390909   \n",
       "sweetxboxgames.com    557496.000000  0.000000  0.000000   839.076923   \n",
       "ulterior-movies.com  1841302.457831  0.000000  0.000000  2443.421687   \n",
       "uvido.com            1541283.116844  0.000000  0.000000  2170.704097   \n",
       "workingfathertv.com  2490334.605505  0.000000  0.000000  2849.064220   \n",
       "worsthorrorgame.com   897408.000000  0.000000  0.000000  1368.000000   \n",
       "\n",
       "                         Wheight  Browser_Chrome  Browser_Firefox  \\\n",
       "Host                                                                \n",
       "bumclub.info         1410.000000        0.000000         0.000000   \n",
       "featureball.com      1356.799263        1.403571         0.253571   \n",
       "hourlyparent.com     1376.875000        0.729412         0.000000   \n",
       "hunp.us                 0.000000        1.758621         0.241379   \n",
       "indoorlife.tv        1758.830189        0.000000         0.000000   \n",
       "mammabay.couk        1476.224138        0.130081         0.048780   \n",
       "myhomesdesign.com    1497.560976        1.217391         0.028986   \n",
       "psychoworld.tv       2034.051282        0.000000         0.000000   \n",
       "rlinevideos.com       824.290909        0.290909         0.000000   \n",
       "sprungliving.com     1442.045455        0.359649         0.000000   \n",
       "sweetxboxgames.com    408.923077        1.600000         0.000000   \n",
       "ulterior-movies.com  1482.698795        0.000000         0.000000   \n",
       "uvido.com            1387.544765        1.109827         0.248555   \n",
       "workingfathertv.com  1715.211009        0.250000         0.000000   \n",
       "worsthorrorgame.com   656.000000        1.638298         0.000000   \n",
       "\n",
       "                     Browser_Internet Explorer  Browser_Opera  Browser_Other  \\\n",
       "Host                                                                           \n",
       "bumclub.info                          2.000000              0       0.000000   \n",
       "featureball.com                       0.317857              0       0.000000   \n",
       "hourlyparent.com                      1.270588              0       0.000000   \n",
       "hunp.us                               0.000000              0       0.000000   \n",
       "indoorlife.tv                         1.900000              0       0.100000   \n",
       "mammabay.couk                         1.821138              0       0.000000   \n",
       "myhomesdesign.com                     0.724638              0       0.028986   \n",
       "psychoworld.tv                        2.000000              0       0.000000   \n",
       "rlinevideos.com                       1.709091              0       0.000000   \n",
       "sprungliving.com                      1.640351              0       0.000000   \n",
       "sweetxboxgames.com                    0.400000              0       0.000000   \n",
       "ulterior-movies.com                   2.000000              0       0.000000   \n",
       "uvido.com                             0.630058              0       0.002890   \n",
       "workingfathertv.com                   1.700000              0       0.000000   \n",
       "worsthorrorgame.com                   0.361702              0       0.000000   \n",
       "\n",
       "                     Browser_Safari/Webkit  \n",
       "Host                                        \n",
       "bumclub.info                      0.000000  \n",
       "featureball.com                   0.025000  \n",
       "hourlyparent.com                  0.000000  \n",
       "hunp.us                           0.000000  \n",
       "indoorlife.tv                     0.000000  \n",
       "mammabay.couk                     0.000000  \n",
       "myhomesdesign.com                 0.000000  \n",
       "psychoworld.tv                    0.000000  \n",
       "rlinevideos.com                   0.000000  \n",
       "sprungliving.com                  0.000000  \n",
       "sweetxboxgames.com                0.000000  \n",
       "ulterior-movies.com               0.000000  \n",
       "uvido.com                         0.008671  \n",
       "workingfathertv.com               0.050000  \n",
       "worsthorrorgame.com               0.000000  "
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fraud data shape:  (15, 16)\n",
      "Nonfraud data shape:  (1939, 16)\n",
      "Combined data shape:  (1954, 16)\n"
     ]
    }
   ],
   "source": [
    "print 'Fraud data shape: ',fhost_df.shape\n",
    "nfhost_df = nfdata.groupby('Host').mean().dropna()\n",
    "print 'Nonfraud data shape: ',nfhost_df.shape\n",
    "allhost_df = pd.concat([nfhost_df, fhost_df])\n",
    "print 'Combined data shape: ',allhost_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "#from sklearn import preprocessing\n",
    "#allhost_df = pd.concat([nfhost, fhost])\n",
    "predictors = ['Inview','Nplugins','Latency','Tsecs','Warea',\n",
    "              'WXpos','WYpos',\n",
    "              'Wwidth','Wheight','Browser_Chrome','Browser_Firefox',\n",
    "              'Browser_Internet Explorer','Browser_Opera','Browser_Other','Browser_Safari/Webkit']\n",
    "this_split_df = pd.concat([nfhost_df[:13], fhost_df[:13]])\n",
    "clf = RandomForestClassifier(warm_start=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training 0 model\n",
      "0.866666666667\n",
      "[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.]\n",
      "Training 1 model\n",
      "1.0\n",
      "[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.]\n",
      "Training 2 model\n",
      "0.933333333333\n",
      "[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.]\n",
      "Training 3 model\n",
      "0.933333333333\n",
      "[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.]\n",
      "Training 4 model\n",
      "0.866666666667\n",
      "[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.]\n",
      "Training 5 model\n",
      "0.8\n",
      "[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.]\n",
      "Training 6 model\n",
      "1.0\n",
      "[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.]\n",
      "Training 7 model\n",
      "0.866666666667\n",
      "[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.]\n",
      "Training 8 model\n",
      "0.933333333333\n",
      "[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.]\n",
      "Training 9 model\n",
      "1.0\n",
      "[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.]\n"
     ]
    }
   ],
   "source": [
    "rows=np.random.choice(nfhost_df.index.values,15,replace=False)\n",
    "\n",
    "model=[]\n",
    "\n",
    "for x in range(10):\n",
    "    print 'Training', x, 'model'\n",
    "    rows=np.random.choice(nfhost_df.index.values,15,replace=False)\n",
    "    this_split_df = pd.concat([nfhost_df.ix[rows], fhost_df[:13]])\n",
    "    clf_fit = clf.fit(this_split_df[predictors],this_split_df['Fraud'])\n",
    "    rows=np.random.choice(nfhost_df.index.values,15,replace=False)\n",
    "    print clf_fit.score(allhost_df[predictors].ix[rows], allhost_df['Fraud'].ix[rows])\n",
    "    print clf_fit.predict(fhost_df[predictors])#.values[14])\n",
    "    model.append(clf_fit)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "('Number of features of the model must  match the input. Model n_features is 15 and  input n_features is 1954 ', u'occurred at index Inview')",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-163-09902be94f81>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmetrics\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mconfusion_matrix\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[0mpreds\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mallhost_df\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mpredictors\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mapply\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;32mlambda\u001b[0m \u001b[0mx\u001b[0m \u001b[1;33m:\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mm\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mm\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mmodel\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;31m#confusion_matrix(allhost_df,)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc\u001b[0m in \u001b[0;36mapply\u001b[1;34m(self, func, axis, broadcast, raw, reduce, args, **kwds)\u001b[0m\n\u001b[0;32m   3912\u001b[0m                     \u001b[1;32mif\u001b[0m \u001b[0mreduce\u001b[0m \u001b[1;32mis\u001b[0m \u001b[0mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3913\u001b[0m                         \u001b[0mreduce\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 3914\u001b[1;33m                     \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_apply_standard\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mf\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mreduce\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mreduce\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   3915\u001b[0m             \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3916\u001b[0m                 \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_apply_broadcast\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mf\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc\u001b[0m in \u001b[0;36m_apply_standard\u001b[1;34m(self, func, axis, ignore_failures, reduce)\u001b[0m\n\u001b[0;32m   4004\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   4005\u001b[0m                 \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mv\u001b[0m \u001b[1;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mseries_gen\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 4006\u001b[1;33m                     \u001b[0mresults\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mv\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   4007\u001b[0m                     \u001b[0mkeys\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mv\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   4008\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-163-09902be94f81>\u001b[0m in \u001b[0;36m<lambda>\u001b[1;34m(x)\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmetrics\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mconfusion_matrix\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[0mpreds\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mallhost_df\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mpredictors\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mapply\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;32mlambda\u001b[0m \u001b[0mx\u001b[0m \u001b[1;33m:\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mm\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mm\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mmodel\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;31m#confusion_matrix(allhost_df,)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/usr/local/lib/python2.7/dist-packages/sklearn/ensemble/forest.pyc\u001b[0m in \u001b[0;36mpredict\u001b[1;34m(self, X)\u001b[0m\n\u001b[0;32m    460\u001b[0m         \u001b[1;31m# for 1d.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    461\u001b[0m         \u001b[0mX\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcheck_array\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mensure_2d\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mFalse\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0maccept_sparse\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m\"csr\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 462\u001b[1;33m         \u001b[0mproba\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpredict_proba\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    463\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    464\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mn_outputs_\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/usr/local/lib/python2.7/dist-packages/sklearn/ensemble/forest.pyc\u001b[0m in \u001b[0;36mpredict_proba\u001b[1;34m(self, X)\u001b[0m\n\u001b[0;32m    511\u001b[0m                              \u001b[0mbackend\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m\"threading\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    512\u001b[0m             \u001b[0mdelayed\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_parallel_helper\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'predict_proba'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcheck_input\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mFalse\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 513\u001b[1;33m             for e in self.estimators_)\n\u001b[0m\u001b[0;32m    514\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    515\u001b[0m         \u001b[1;31m# Reduce\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/parallel.pyc\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, iterable)\u001b[0m\n\u001b[0;32m    657\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_iterating\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    658\u001b[0m             \u001b[1;32mfor\u001b[0m \u001b[0mfunction\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkwargs\u001b[0m \u001b[1;32min\u001b[0m \u001b[0miterable\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 659\u001b[1;33m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdispatch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfunction\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    660\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    661\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mpre_dispatch\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;34m\"all\"\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0mn_jobs\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/parallel.pyc\u001b[0m in \u001b[0;36mdispatch\u001b[1;34m(self, func, args, kwargs)\u001b[0m\n\u001b[0;32m    404\u001b[0m         \"\"\"\n\u001b[0;32m    405\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_pool\u001b[0m \u001b[1;32mis\u001b[0m \u001b[0mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 406\u001b[1;33m             \u001b[0mjob\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mImmediateApply\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    407\u001b[0m             \u001b[0mindex\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_jobs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    408\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0m_verbosity_filter\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mindex\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mverbose\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/parallel.pyc\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, func, args, kwargs)\u001b[0m\n\u001b[0;32m    138\u001b[0m         \u001b[1;31m# Don't delay the application, to avoid keeping the input\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    139\u001b[0m         \u001b[1;31m# arguments in memory\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 140\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mresults\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    141\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    142\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/usr/local/lib/python2.7/dist-packages/sklearn/ensemble/forest.pyc\u001b[0m in \u001b[0;36m_parallel_helper\u001b[1;34m(obj, methodname, *args, **kwargs)\u001b[0m\n\u001b[0;32m    104\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0m_parallel_helper\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmethodname\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    105\u001b[0m     \u001b[1;34m\"\"\"Private helper to workaround Python 2 pickle limitations\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 106\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmethodname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    107\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    108\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/usr/local/lib/python2.7/dist-packages/sklearn/tree/tree.pyc\u001b[0m in \u001b[0;36mpredict_proba\u001b[1;34m(self, X, check_input)\u001b[0m\n\u001b[0;32m    588\u001b[0m                              \u001b[1;34m\" match the input. Model n_features is %s and \"\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    589\u001b[0m                              \u001b[1;34m\" input n_features is %s \"\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 590\u001b[1;33m                              % (self.n_features_, n_features))\n\u001b[0m\u001b[0;32m    591\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    592\u001b[0m         \u001b[0mproba\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtree_\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: ('Number of features of the model must  match the input. Model n_features is 15 and  input n_features is 1954 ', u'occurred at index Inview')"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "preds = allhost_df[predictors].apply(lambda x : np.mean([m.predict(x) for m in model]))\n",
    "\n",
    "#confusion_matrix(allhost_df,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Inview</th>\n",
       "      <th>Nplugins</th>\n",
       "      <th>Latency</th>\n",
       "      <th>Fraud</th>\n",
       "      <th>Tsecs</th>\n",
       "      <th>Warea</th>\n",
       "      <th>WXpos</th>\n",
       "      <th>WYpos</th>\n",
       "      <th>Wwidth</th>\n",
       "      <th>Wheight</th>\n",
       "      <th>Browser_Chrome</th>\n",
       "      <th>Browser_Firefox</th>\n",
       "      <th>Browser_Internet Explorer</th>\n",
       "      <th>Browser_Opera</th>\n",
       "      <th>Browser_Other</th>\n",
       "      <th>Browser_Safari/Webkit</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Host</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>50connect.couk</th>\n",
       "      <td>0.730769</td>\n",
       "      <td>2.342105</td>\n",
       "      <td>301.205128</td>\n",
       "      <td>0</td>\n",
       "      <td>50263.666667</td>\n",
       "      <td>896810.933333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1343.666667</td>\n",
       "      <td>667.866667</td>\n",
       "      <td>0.012346</td>\n",
       "      <td>0.012346</td>\n",
       "      <td>0.938272</td>\n",
       "      <td>0</td>\n",
       "      <td>0.037037</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5mostpopular.com</th>\n",
       "      <td>0.214286</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1053.230769</td>\n",
       "      <td>0</td>\n",
       "      <td>52054.642857</td>\n",
       "      <td>922741.333333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1125.333333</td>\n",
       "      <td>807.833333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>advanceautoparts.com</th>\n",
       "      <td>0.230769</td>\n",
       "      <td>9.142857</td>\n",
       "      <td>874.875000</td>\n",
       "      <td>0</td>\n",
       "      <td>61069.206897</td>\n",
       "      <td>535928.250000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>982.083333</td>\n",
       "      <td>469.583333</td>\n",
       "      <td>0.206897</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.793103</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>advcm.com</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>1159.142857</td>\n",
       "      <td>0</td>\n",
       "      <td>43810.555556</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>affhealth.com</th>\n",
       "      <td>0.777778</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1079.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>41026.444444</td>\n",
       "      <td>1013165.714286</td>\n",
       "      <td>-1.142857</td>\n",
       "      <td>-1.142857</td>\n",
       "      <td>1436.571429</td>\n",
       "      <td>698.428571</td>\n",
       "      <td>0.777778</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ahaber.comtr</th>\n",
       "      <td>0.333333</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>697.571429</td>\n",
       "      <td>0</td>\n",
       "      <td>58776.222222</td>\n",
       "      <td>826119.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1269.000000</td>\n",
       "      <td>651.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>alldayfashions.com</th>\n",
       "      <td>0.181818</td>\n",
       "      <td>3.250000</td>\n",
       "      <td>1852.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>41642.000000</td>\n",
       "      <td>812886.571429</td>\n",
       "      <td>-2.000000</td>\n",
       "      <td>-2.000000</td>\n",
       "      <td>1246.285714</td>\n",
       "      <td>626.000000</td>\n",
       "      <td>0.272727</td>\n",
       "      <td>0.090909</td>\n",
       "      <td>0.636364</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>americanprofile.com</th>\n",
       "      <td>0.518248</td>\n",
       "      <td>7.153846</td>\n",
       "      <td>1866.311111</td>\n",
       "      <td>0</td>\n",
       "      <td>54460.000000</td>\n",
       "      <td>971511.945946</td>\n",
       "      <td>-0.504505</td>\n",
       "      <td>-0.504505</td>\n",
       "      <td>1324.198198</td>\n",
       "      <td>711.117117</td>\n",
       "      <td>0.133803</td>\n",
       "      <td>0.014085</td>\n",
       "      <td>0.852113</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>artinstitutes.edu</th>\n",
       "      <td>0.166667</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>233.916667</td>\n",
       "      <td>0</td>\n",
       "      <td>45196.083333</td>\n",
       "      <td>1028208.000000</td>\n",
       "      <td>-8.000000</td>\n",
       "      <td>-8.000000</td>\n",
       "      <td>1382.000000</td>\n",
       "      <td>744.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>avimoya.com</th>\n",
       "      <td>0.583333</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1349.166667</td>\n",
       "      <td>0</td>\n",
       "      <td>58844.750000</td>\n",
       "      <td>1245209.600000</td>\n",
       "      <td>-4.800000</td>\n",
       "      <td>-4.800000</td>\n",
       "      <td>1494.400000</td>\n",
       "      <td>821.600000</td>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.916667</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>babiesrus.com</th>\n",
       "      <td>0.333333</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>103.727273</td>\n",
       "      <td>0</td>\n",
       "      <td>44402.153846</td>\n",
       "      <td>1018560.000000</td>\n",
       "      <td>-3.000000</td>\n",
       "      <td>-3.000000</td>\n",
       "      <td>1335.142857</td>\n",
       "      <td>695.857143</td>\n",
       "      <td>0.076923</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.923077</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bangordailynews.com</th>\n",
       "      <td>0.272727</td>\n",
       "      <td>2.909091</td>\n",
       "      <td>634.444444</td>\n",
       "      <td>0</td>\n",
       "      <td>26939.181818</td>\n",
       "      <td>840435.142857</td>\n",
       "      <td>5.857143</td>\n",
       "      <td>5.857143</td>\n",
       "      <td>1180.571429</td>\n",
       "      <td>712.142857</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.090909</td>\n",
       "      <td>0.909091</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>befunky.com</th>\n",
       "      <td>0.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>370.700000</td>\n",
       "      <td>0</td>\n",
       "      <td>38910.952381</td>\n",
       "      <td>854757.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1301.000000</td>\n",
       "      <td>657.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.952381</td>\n",
       "      <td>0</td>\n",
       "      <td>0.047619</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bratano.com</th>\n",
       "      <td>0.558824</td>\n",
       "      <td>1.125000</td>\n",
       "      <td>618.529412</td>\n",
       "      <td>0</td>\n",
       "      <td>41740.789474</td>\n",
       "      <td>1210221.818182</td>\n",
       "      <td>-4.727273</td>\n",
       "      <td>-4.727273</td>\n",
       "      <td>1518.272727</td>\n",
       "      <td>789.227273</td>\n",
       "      <td>0.447368</td>\n",
       "      <td>0.026316</td>\n",
       "      <td>0.526316</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>breakfastdailynews.com</th>\n",
       "      <td>0.638889</td>\n",
       "      <td>1.275862</td>\n",
       "      <td>935.400000</td>\n",
       "      <td>0</td>\n",
       "      <td>47864.050633</td>\n",
       "      <td>910621.000000</td>\n",
       "      <td>-0.190476</td>\n",
       "      <td>-0.190476</td>\n",
       "      <td>1317.190476</td>\n",
       "      <td>673.238095</td>\n",
       "      <td>0.594937</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.379747</td>\n",
       "      <td>0</td>\n",
       "      <td>0.012658</td>\n",
       "      <td>0.012658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>brolycow.com</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.857143</td>\n",
       "      <td>3365.750000</td>\n",
       "      <td>0</td>\n",
       "      <td>39555.416667</td>\n",
       "      <td>1080518.500000</td>\n",
       "      <td>1.250000</td>\n",
       "      <td>13.750000</td>\n",
       "      <td>1171.750000</td>\n",
       "      <td>909.250000</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.416667</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cableone.net</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.961538</td>\n",
       "      <td>155.730769</td>\n",
       "      <td>0</td>\n",
       "      <td>51175.653846</td>\n",
       "      <td>1490606.384615</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1730.423077</td>\n",
       "      <td>859.653846</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cincinnatibell.net</th>\n",
       "      <td>0.266667</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>424.933333</td>\n",
       "      <td>0</td>\n",
       "      <td>65421.266667</td>\n",
       "      <td>1212472.727273</td>\n",
       "      <td>-0.363636</td>\n",
       "      <td>-0.363636</td>\n",
       "      <td>1497.454545</td>\n",
       "      <td>810.181818</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.066667</td>\n",
       "      <td>0.933333</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cinesport.com</th>\n",
       "      <td>0.514925</td>\n",
       "      <td>1.545455</td>\n",
       "      <td>1427.545455</td>\n",
       "      <td>0</td>\n",
       "      <td>51404.255474</td>\n",
       "      <td>1044896.441558</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1213.532468</td>\n",
       "      <td>851.272727</td>\n",
       "      <td>0.029197</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.919708</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.051095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>clearguides.com</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.400000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>52870.909091</td>\n",
       "      <td>828660.272727</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1159.000000</td>\n",
       "      <td>688.636364</td>\n",
       "      <td>0.181818</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>collegedose.com</th>\n",
       "      <td>0.500000</td>\n",
       "      <td>7.333333</td>\n",
       "      <td>1537.800000</td>\n",
       "      <td>0</td>\n",
       "      <td>48817.789474</td>\n",
       "      <td>796222.800000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1294.800000</td>\n",
       "      <td>609.000000</td>\n",
       "      <td>0.473684</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.526316</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cruisecritic.com</th>\n",
       "      <td>0.545455</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>442.909091</td>\n",
       "      <td>0</td>\n",
       "      <td>40096.272727</td>\n",
       "      <td>1258912.000000</td>\n",
       "      <td>5.666667</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>1568.000000</td>\n",
       "      <td>794.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.181818</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cupidspulse.com</th>\n",
       "      <td>0.200000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>142.681818</td>\n",
       "      <td>0</td>\n",
       "      <td>42805.888889</td>\n",
       "      <td>810875.052632</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1101.157895</td>\n",
       "      <td>724.684211</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dailycaller.com</th>\n",
       "      <td>0.428571</td>\n",
       "      <td>6.500000</td>\n",
       "      <td>354.457143</td>\n",
       "      <td>0</td>\n",
       "      <td>45227.333333</td>\n",
       "      <td>1052013.304348</td>\n",
       "      <td>0.304348</td>\n",
       "      <td>0.956522</td>\n",
       "      <td>1381.260870</td>\n",
       "      <td>738.782609</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>direttanews.it</th>\n",
       "      <td>0.250000</td>\n",
       "      <td>5.250000</td>\n",
       "      <td>1487.107143</td>\n",
       "      <td>0</td>\n",
       "      <td>39686.714286</td>\n",
       "      <td>1196951.185185</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1549.851852</td>\n",
       "      <td>774.000000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dressupdream.com</th>\n",
       "      <td>0.914286</td>\n",
       "      <td>2.205882</td>\n",
       "      <td>12660.235294</td>\n",
       "      <td>0</td>\n",
       "      <td>67852.028571</td>\n",
       "      <td>578384.393939</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>929.000000</td>\n",
       "      <td>615.969697</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eenadu.net</th>\n",
       "      <td>0.631579</td>\n",
       "      <td>6.066667</td>\n",
       "      <td>201.763158</td>\n",
       "      <td>0</td>\n",
       "      <td>45022.238095</td>\n",
       "      <td>971442.250000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1298.250000</td>\n",
       "      <td>733.000000</td>\n",
       "      <td>0.214286</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.714286</td>\n",
       "      <td>0</td>\n",
       "      <td>0.047619</td>\n",
       "      <td>0.023810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>essentialkids.comau</th>\n",
       "      <td>0.428571</td>\n",
       "      <td>3.465517</td>\n",
       "      <td>280.985507</td>\n",
       "      <td>0</td>\n",
       "      <td>76138.514286</td>\n",
       "      <td>1070651.800000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1470.828571</td>\n",
       "      <td>711.828571</td>\n",
       "      <td>0.128571</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.871429</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>famousbirthdays.com</th>\n",
       "      <td>0.055556</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>310.888889</td>\n",
       "      <td>0</td>\n",
       "      <td>65316.000000</td>\n",
       "      <td>757760.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>880.000000</td>\n",
       "      <td>715.333333</td>\n",
       "      <td>0.722222</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.277778</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fashion-feed.net</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>23.100000</td>\n",
       "      <td>0</td>\n",
       "      <td>48163.800000</td>\n",
       "      <td>2014920.000000</td>\n",
       "      <td>-5.000000</td>\n",
       "      <td>-5.000000</td>\n",
       "      <td>1930.000000</td>\n",
       "      <td>1044.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>supgamer.com</th>\n",
       "      <td>0.615385</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>630.280000</td>\n",
       "      <td>0</td>\n",
       "      <td>51230.862069</td>\n",
       "      <td>861305.000000</td>\n",
       "      <td>-1.333333</td>\n",
       "      <td>-1.333333</td>\n",
       "      <td>1141.166667</td>\n",
       "      <td>757.000000</td>\n",
       "      <td>0.034483</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.965517</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>surlybirds.com</th>\n",
       "      <td>0.142857</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>414.583333</td>\n",
       "      <td>0</td>\n",
       "      <td>49069.941176</td>\n",
       "      <td>960624.000000</td>\n",
       "      <td>-3.200000</td>\n",
       "      <td>-3.200000</td>\n",
       "      <td>1214.000000</td>\n",
       "      <td>784.800000</td>\n",
       "      <td>0.058824</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.941176</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>swagbucks.com</th>\n",
       "      <td>0.750000</td>\n",
       "      <td>10.041667</td>\n",
       "      <td>1382.272727</td>\n",
       "      <td>0</td>\n",
       "      <td>32593.840000</td>\n",
       "      <td>1035259.250000</td>\n",
       "      <td>103.750000</td>\n",
       "      <td>135.500000</td>\n",
       "      <td>1423.750000</td>\n",
       "      <td>717.500000</td>\n",
       "      <td>0.520000</td>\n",
       "      <td>0.040000</td>\n",
       "      <td>0.360000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.040000</td>\n",
       "      <td>0.040000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tds.net</th>\n",
       "      <td>0.181818</td>\n",
       "      <td>7.818182</td>\n",
       "      <td>2559.900000</td>\n",
       "      <td>0</td>\n",
       "      <td>56078.272727</td>\n",
       "      <td>728661.333333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1052.444444</td>\n",
       "      <td>691.888889</td>\n",
       "      <td>0.363636</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.636364</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>technicultr.com</th>\n",
       "      <td>0.787234</td>\n",
       "      <td>1.619048</td>\n",
       "      <td>1107.787234</td>\n",
       "      <td>0</td>\n",
       "      <td>54231.375000</td>\n",
       "      <td>911274.157895</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1273.184211</td>\n",
       "      <td>708.078947</td>\n",
       "      <td>0.104167</td>\n",
       "      <td>0.062500</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>techniqueworld.com</th>\n",
       "      <td>0.473684</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1479.533333</td>\n",
       "      <td>0</td>\n",
       "      <td>47216.869565</td>\n",
       "      <td>961754.615385</td>\n",
       "      <td>-1.230769</td>\n",
       "      <td>-1.230769</td>\n",
       "      <td>1170.230769</td>\n",
       "      <td>661.538462</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.956522</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.043478</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>thenewscenter.tv</th>\n",
       "      <td>0.294118</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>223.500000</td>\n",
       "      <td>0</td>\n",
       "      <td>62349.529412</td>\n",
       "      <td>1116000.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1440.000000</td>\n",
       "      <td>775.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>theplaystationgames.com</th>\n",
       "      <td>0.406250</td>\n",
       "      <td>1.145455</td>\n",
       "      <td>6296.645161</td>\n",
       "      <td>0</td>\n",
       "      <td>59259.303030</td>\n",
       "      <td>926875.636364</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1279.272727</td>\n",
       "      <td>653.090909</td>\n",
       "      <td>0.767677</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.232323</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>theseaapp.com</th>\n",
       "      <td>0.296296</td>\n",
       "      <td>1.882353</td>\n",
       "      <td>980.571429</td>\n",
       "      <td>0</td>\n",
       "      <td>70668.214286</td>\n",
       "      <td>3161842.000000</td>\n",
       "      <td>-14.000000</td>\n",
       "      <td>55.500000</td>\n",
       "      <td>2399.500000</td>\n",
       "      <td>1222.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.964286</td>\n",
       "      <td>0</td>\n",
       "      <td>0.035714</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>thosefreegames.com</th>\n",
       "      <td>0.640000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>210.041667</td>\n",
       "      <td>0</td>\n",
       "      <td>50822.607143</td>\n",
       "      <td>1025941.333333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1209.250000</td>\n",
       "      <td>836.000000</td>\n",
       "      <td>0.071429</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.892857</td>\n",
       "      <td>0</td>\n",
       "      <td>0.035714</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tomandlorenzo.com</th>\n",
       "      <td>0.645161</td>\n",
       "      <td>4.491525</td>\n",
       "      <td>699.372881</td>\n",
       "      <td>0</td>\n",
       "      <td>55221.852941</td>\n",
       "      <td>1484018.571429</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1661.642857</td>\n",
       "      <td>862.678571</td>\n",
       "      <td>0.382353</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.558824</td>\n",
       "      <td>0</td>\n",
       "      <td>0.058824</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tooshocking.com</th>\n",
       "      <td>0.428571</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1347.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>57728.083333</td>\n",
       "      <td>1857216.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>89.333333</td>\n",
       "      <td>1269.333333</td>\n",
       "      <td>1556.000000</td>\n",
       "      <td>0.138889</td>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.027778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>topplanetnews.com</th>\n",
       "      <td>0.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>509.769231</td>\n",
       "      <td>0</td>\n",
       "      <td>70171.277778</td>\n",
       "      <td>1202951.285714</td>\n",
       "      <td>-2.857143</td>\n",
       "      <td>-2.857143</td>\n",
       "      <td>1538.571429</td>\n",
       "      <td>777.071429</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.944444</td>\n",
       "      <td>0</td>\n",
       "      <td>0.055556</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>toshiba.com</th>\n",
       "      <td>0.152493</td>\n",
       "      <td>4.671429</td>\n",
       "      <td>463.409496</td>\n",
       "      <td>0</td>\n",
       "      <td>50507.986188</td>\n",
       "      <td>924603.742857</td>\n",
       "      <td>-0.685714</td>\n",
       "      <td>-0.685714</td>\n",
       "      <td>1339.281633</td>\n",
       "      <td>665.302041</td>\n",
       "      <td>0.196133</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.787293</td>\n",
       "      <td>0</td>\n",
       "      <td>0.016575</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>traileraddict.com</th>\n",
       "      <td>0.526316</td>\n",
       "      <td>1.764706</td>\n",
       "      <td>1739.078947</td>\n",
       "      <td>0</td>\n",
       "      <td>42015.105263</td>\n",
       "      <td>727885.217391</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1081.434783</td>\n",
       "      <td>648.695652</td>\n",
       "      <td>0.342105</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.657895</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ubore.com</th>\n",
       "      <td>0.259259</td>\n",
       "      <td>2.142857</td>\n",
       "      <td>528.555556</td>\n",
       "      <td>0</td>\n",
       "      <td>59535.441176</td>\n",
       "      <td>675779.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1013.642857</td>\n",
       "      <td>664.785714</td>\n",
       "      <td>0.058824</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.882353</td>\n",
       "      <td>0</td>\n",
       "      <td>0.058824</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uinterview.com</th>\n",
       "      <td>0.250000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>1356.300000</td>\n",
       "      <td>0</td>\n",
       "      <td>67620.642857</td>\n",
       "      <td>790563.200000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1039.100000</td>\n",
       "      <td>771.700000</td>\n",
       "      <td>0.428571</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ussportbox.com</th>\n",
       "      <td>0.125000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>201.111111</td>\n",
       "      <td>0</td>\n",
       "      <td>50236.823529</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.058824</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.941176</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>vaccint.com</th>\n",
       "      <td>0.700000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1101.764706</td>\n",
       "      <td>0</td>\n",
       "      <td>50659.181818</td>\n",
       "      <td>717769.823529</td>\n",
       "      <td>177.941176</td>\n",
       "      <td>41.235294</td>\n",
       "      <td>1116.058824</td>\n",
       "      <td>643.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.909091</td>\n",
       "      <td>0</td>\n",
       "      <td>0.090909</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>veevr.com</th>\n",
       "      <td>0.493827</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>994.358974</td>\n",
       "      <td>0</td>\n",
       "      <td>64527.864198</td>\n",
       "      <td>1094013.000000</td>\n",
       "      <td>-7.000000</td>\n",
       "      <td>-7.000000</td>\n",
       "      <td>1342.750000</td>\n",
       "      <td>721.750000</td>\n",
       "      <td>0.074074</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.913580</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.012346</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>viki.com</th>\n",
       "      <td>0.657143</td>\n",
       "      <td>14.857143</td>\n",
       "      <td>2306.800000</td>\n",
       "      <td>0</td>\n",
       "      <td>29213.885714</td>\n",
       "      <td>975844.444444</td>\n",
       "      <td>-1.777778</td>\n",
       "      <td>-1.777778</td>\n",
       "      <td>1369.555556</td>\n",
       "      <td>712.444444</td>\n",
       "      <td>0.942857</td>\n",
       "      <td>0.057143</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>virgilio.it</th>\n",
       "      <td>0.375000</td>\n",
       "      <td>6.800000</td>\n",
       "      <td>618.304348</td>\n",
       "      <td>0</td>\n",
       "      <td>40758.750000</td>\n",
       "      <td>796759.470588</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1241.235294</td>\n",
       "      <td>629.176471</td>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>waff.com</th>\n",
       "      <td>0.777778</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>293.272727</td>\n",
       "      <td>0</td>\n",
       "      <td>50996.432432</td>\n",
       "      <td>1075680.476190</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1352.904762</td>\n",
       "      <td>818.047619</td>\n",
       "      <td>0.081081</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.891892</td>\n",
       "      <td>0</td>\n",
       "      <td>0.027027</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wagbrag.com</th>\n",
       "      <td>0.734694</td>\n",
       "      <td>13.566667</td>\n",
       "      <td>4729.085106</td>\n",
       "      <td>0</td>\n",
       "      <td>69017.568627</td>\n",
       "      <td>853344.837209</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1232.279070</td>\n",
       "      <td>694.209302</td>\n",
       "      <td>0.274510</td>\n",
       "      <td>0.019608</td>\n",
       "      <td>0.627451</td>\n",
       "      <td>0</td>\n",
       "      <td>0.039216</td>\n",
       "      <td>0.039216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weeklyfit.com</th>\n",
       "      <td>0.180000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>512.200000</td>\n",
       "      <td>0</td>\n",
       "      <td>47315.574074</td>\n",
       "      <td>855479.466667</td>\n",
       "      <td>-0.533333</td>\n",
       "      <td>-0.533333</td>\n",
       "      <td>1080.533333</td>\n",
       "      <td>786.133333</td>\n",
       "      <td>0.018519</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.981481</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wmur.com</th>\n",
       "      <td>0.130435</td>\n",
       "      <td>1.956522</td>\n",
       "      <td>152.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>54151.782609</td>\n",
       "      <td>1129618.636364</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1517.090909</td>\n",
       "      <td>722.318182</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>womenwholovebooks.com</th>\n",
       "      <td>0.714286</td>\n",
       "      <td>1.307692</td>\n",
       "      <td>1876.692308</td>\n",
       "      <td>0</td>\n",
       "      <td>34002.400000</td>\n",
       "      <td>971032.000000</td>\n",
       "      <td>-2.000000</td>\n",
       "      <td>-2.000000</td>\n",
       "      <td>1344.000000</td>\n",
       "      <td>714.500000</td>\n",
       "      <td>0.933333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.066667</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wpcentral.com</th>\n",
       "      <td>0.426471</td>\n",
       "      <td>2.763547</td>\n",
       "      <td>226.171569</td>\n",
       "      <td>0</td>\n",
       "      <td>48564.980676</td>\n",
       "      <td>1213569.059406</td>\n",
       "      <td>-0.039604</td>\n",
       "      <td>0.504950</td>\n",
       "      <td>1451.123762</td>\n",
       "      <td>849.559406</td>\n",
       "      <td>0.033816</td>\n",
       "      <td>0.106280</td>\n",
       "      <td>0.859903</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wvydeo.com</th>\n",
       "      <td>0.142857</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>460.571429</td>\n",
       "      <td>0</td>\n",
       "      <td>71849.928571</td>\n",
       "      <td>195203.714286</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>209.571429</td>\n",
       "      <td>124.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.928571</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.071429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zanru.com</th>\n",
       "      <td>0.576923</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>254.120000</td>\n",
       "      <td>0</td>\n",
       "      <td>46349.481481</td>\n",
       "      <td>800961.062500</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1095.937500</td>\n",
       "      <td>706.562500</td>\n",
       "      <td>0.037037</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.962963</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>125 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                           Inview   Nplugins       Latency  Fraud  \\\n",
       "Host                                                                \n",
       "50connect.couk           0.730769   2.342105    301.205128      0   \n",
       "5mostpopular.com         0.214286   2.000000   1053.230769      0   \n",
       "advanceautoparts.com     0.230769   9.142857    874.875000      0   \n",
       "advcm.com                0.000000  11.000000   1159.142857      0   \n",
       "affhealth.com            0.777778   1.000000   1079.000000      0   \n",
       "ahaber.comtr             0.333333   2.000000    697.571429      0   \n",
       "alldayfashions.com       0.181818   3.250000   1852.000000      0   \n",
       "americanprofile.com      0.518248   7.153846   1866.311111      0   \n",
       "artinstitutes.edu        0.166667   2.000000    233.916667      0   \n",
       "avimoya.com              0.583333   3.000000   1349.166667      0   \n",
       "babiesrus.com            0.333333   2.000000    103.727273      0   \n",
       "bangordailynews.com      0.272727   2.909091    634.444444      0   \n",
       "befunky.com              0.500000   1.000000    370.700000      0   \n",
       "bratano.com              0.558824   1.125000    618.529412      0   \n",
       "breakfastdailynews.com   0.638889   1.275862    935.400000      0   \n",
       "brolycow.com             0.000000   1.857143   3365.750000      0   \n",
       "cableone.net             0.000000   1.961538    155.730769      0   \n",
       "cincinnatibell.net       0.266667   2.000000    424.933333      0   \n",
       "cinesport.com            0.514925   1.545455   1427.545455      0   \n",
       "clearguides.com          1.000000   4.400000      0.000000      0   \n",
       "collegedose.com          0.500000   7.333333   1537.800000      0   \n",
       "cruisecritic.com         0.545455   8.000000    442.909091      0   \n",
       "cupidspulse.com          0.200000   2.000000    142.681818      0   \n",
       "dailycaller.com          0.428571   6.500000    354.457143      0   \n",
       "direttanews.it           0.250000   5.250000   1487.107143      0   \n",
       "dressupdream.com         0.914286   2.205882  12660.235294      0   \n",
       "eenadu.net               0.631579   6.066667    201.763158      0   \n",
       "essentialkids.comau      0.428571   3.465517    280.985507      0   \n",
       "famousbirthdays.com      0.055556   1.000000    310.888889      0   \n",
       "fashion-feed.net         0.000000   1.000000     23.100000      0   \n",
       "...                           ...        ...           ...    ...   \n",
       "supgamer.com             0.615385   2.000000    630.280000      0   \n",
       "surlybirds.com           0.142857   2.000000    414.583333      0   \n",
       "swagbucks.com            0.750000  10.041667   1382.272727      0   \n",
       "tds.net                  0.181818   7.818182   2559.900000      0   \n",
       "technicultr.com          0.787234   1.619048   1107.787234      0   \n",
       "techniqueworld.com       0.473684   1.000000   1479.533333      0   \n",
       "thenewscenter.tv         0.294118   2.000000    223.500000      0   \n",
       "theplaystationgames.com  0.406250   1.145455   6296.645161      0   \n",
       "theseaapp.com            0.296296   1.882353    980.571429      0   \n",
       "thosefreegames.com       0.640000   2.000000    210.041667      0   \n",
       "tomandlorenzo.com        0.645161   4.491525    699.372881      0   \n",
       "tooshocking.com          0.428571   2.000000   1347.000000      0   \n",
       "topplanetnews.com        0.500000   2.000000    509.769231      0   \n",
       "toshiba.com              0.152493   4.671429    463.409496      0   \n",
       "traileraddict.com        0.526316   1.764706   1739.078947      0   \n",
       "ubore.com                0.259259   2.142857    528.555556      0   \n",
       "uinterview.com           0.250000   8.000000   1356.300000      0   \n",
       "ussportbox.com           0.125000   8.000000    201.111111      0   \n",
       "vaccint.com              0.700000   2.000000   1101.764706      0   \n",
       "veevr.com                0.493827   2.000000    994.358974      0   \n",
       "viki.com                 0.657143  14.857143   2306.800000      0   \n",
       "virgilio.it              0.375000   6.800000    618.304348      0   \n",
       "waff.com                 0.777778   2.000000    293.272727      0   \n",
       "wagbrag.com              0.734694  13.566667   4729.085106      0   \n",
       "weeklyfit.com            0.180000   2.000000    512.200000      0   \n",
       "wmur.com                 0.130435   1.956522    152.333333      0   \n",
       "womenwholovebooks.com    0.714286   1.307692   1876.692308      0   \n",
       "wpcentral.com            0.426471   2.763547    226.171569      0   \n",
       "wvydeo.com               0.142857   2.000000    460.571429      0   \n",
       "zanru.com                0.576923   2.000000    254.120000      0   \n",
       "\n",
       "                                Tsecs           Warea       WXpos       WYpos  \\\n",
       "Host                                                                            \n",
       "50connect.couk           50263.666667   896810.933333    0.000000    0.000000   \n",
       "5mostpopular.com         52054.642857   922741.333333    0.000000    0.000000   \n",
       "advanceautoparts.com     61069.206897   535928.250000    0.000000    0.000000   \n",
       "advcm.com                43810.555556        0.000000    0.000000    0.000000   \n",
       "affhealth.com            41026.444444  1013165.714286   -1.142857   -1.142857   \n",
       "ahaber.comtr             58776.222222   826119.000000    0.000000    0.000000   \n",
       "alldayfashions.com       41642.000000   812886.571429   -2.000000   -2.000000   \n",
       "americanprofile.com      54460.000000   971511.945946   -0.504505   -0.504505   \n",
       "artinstitutes.edu        45196.083333  1028208.000000   -8.000000   -8.000000   \n",
       "avimoya.com              58844.750000  1245209.600000   -4.800000   -4.800000   \n",
       "babiesrus.com            44402.153846  1018560.000000   -3.000000   -3.000000   \n",
       "bangordailynews.com      26939.181818   840435.142857    5.857143    5.857143   \n",
       "befunky.com              38910.952381   854757.000000    0.000000    0.000000   \n",
       "bratano.com              41740.789474  1210221.818182   -4.727273   -4.727273   \n",
       "breakfastdailynews.com   47864.050633   910621.000000   -0.190476   -0.190476   \n",
       "brolycow.com             39555.416667  1080518.500000    1.250000   13.750000   \n",
       "cableone.net             51175.653846  1490606.384615    0.000000    0.000000   \n",
       "cincinnatibell.net       65421.266667  1212472.727273   -0.363636   -0.363636   \n",
       "cinesport.com            51404.255474  1044896.441558    0.000000    0.000000   \n",
       "clearguides.com          52870.909091   828660.272727    0.000000    0.000000   \n",
       "collegedose.com          48817.789474   796222.800000    0.000000    0.000000   \n",
       "cruisecritic.com         40096.272727  1258912.000000    5.666667   50.000000   \n",
       "cupidspulse.com          42805.888889   810875.052632    0.000000    0.000000   \n",
       "dailycaller.com          45227.333333  1052013.304348    0.304348    0.956522   \n",
       "direttanews.it           39686.714286  1196951.185185    0.000000    0.000000   \n",
       "dressupdream.com         67852.028571   578384.393939    0.000000    0.000000   \n",
       "eenadu.net               45022.238095   971442.250000    0.000000    0.000000   \n",
       "essentialkids.comau      76138.514286  1070651.800000    0.000000    0.000000   \n",
       "famousbirthdays.com      65316.000000   757760.000000    0.000000    0.000000   \n",
       "fashion-feed.net         48163.800000  2014920.000000   -5.000000   -5.000000   \n",
       "...                               ...             ...         ...         ...   \n",
       "supgamer.com             51230.862069   861305.000000   -1.333333   -1.333333   \n",
       "surlybirds.com           49069.941176   960624.000000   -3.200000   -3.200000   \n",
       "swagbucks.com            32593.840000  1035259.250000  103.750000  135.500000   \n",
       "tds.net                  56078.272727   728661.333333    0.000000    0.000000   \n",
       "technicultr.com          54231.375000   911274.157895    0.000000    0.000000   \n",
       "techniqueworld.com       47216.869565   961754.615385   -1.230769   -1.230769   \n",
       "thenewscenter.tv         62349.529412  1116000.000000    0.000000    0.000000   \n",
       "theplaystationgames.com  59259.303030   926875.636364    0.000000    0.000000   \n",
       "theseaapp.com            70668.214286  3161842.000000  -14.000000   55.500000   \n",
       "thosefreegames.com       50822.607143  1025941.333333    0.000000    0.000000   \n",
       "tomandlorenzo.com        55221.852941  1484018.571429    0.000000    0.000000   \n",
       "tooshocking.com          57728.083333  1857216.000000    6.000000   89.333333   \n",
       "topplanetnews.com        70171.277778  1202951.285714   -2.857143   -2.857143   \n",
       "toshiba.com              50507.986188   924603.742857   -0.685714   -0.685714   \n",
       "traileraddict.com        42015.105263   727885.217391    0.000000    0.000000   \n",
       "ubore.com                59535.441176   675779.500000    0.000000    0.000000   \n",
       "uinterview.com           67620.642857   790563.200000    0.000000    0.000000   \n",
       "ussportbox.com           50236.823529        0.000000    0.000000    0.000000   \n",
       "vaccint.com              50659.181818   717769.823529  177.941176   41.235294   \n",
       "veevr.com                64527.864198  1094013.000000   -7.000000   -7.000000   \n",
       "viki.com                 29213.885714   975844.444444   -1.777778   -1.777778   \n",
       "virgilio.it              40758.750000   796759.470588    0.000000    0.000000   \n",
       "waff.com                 50996.432432  1075680.476190    0.000000    0.000000   \n",
       "wagbrag.com              69017.568627   853344.837209    0.000000    0.000000   \n",
       "weeklyfit.com            47315.574074   855479.466667   -0.533333   -0.533333   \n",
       "wmur.com                 54151.782609  1129618.636364    0.000000    0.000000   \n",
       "womenwholovebooks.com    34002.400000   971032.000000   -2.000000   -2.000000   \n",
       "wpcentral.com            48564.980676  1213569.059406   -0.039604    0.504950   \n",
       "wvydeo.com               71849.928571   195203.714286    0.000000    0.000000   \n",
       "zanru.com                46349.481481   800961.062500    0.000000    0.000000   \n",
       "\n",
       "                              Wwidth      Wheight  Browser_Chrome  \\\n",
       "Host                                                                \n",
       "50connect.couk           1343.666667   667.866667        0.012346   \n",
       "5mostpopular.com         1125.333333   807.833333        0.000000   \n",
       "advanceautoparts.com      982.083333   469.583333        0.206897   \n",
       "advcm.com                   0.000000     0.000000        1.000000   \n",
       "affhealth.com            1436.571429   698.428571        0.777778   \n",
       "ahaber.comtr             1269.000000   651.000000        0.000000   \n",
       "alldayfashions.com       1246.285714   626.000000        0.272727   \n",
       "americanprofile.com      1324.198198   711.117117        0.133803   \n",
       "artinstitutes.edu        1382.000000   744.000000        0.000000   \n",
       "avimoya.com              1494.400000   821.600000        0.083333   \n",
       "babiesrus.com            1335.142857   695.857143        0.076923   \n",
       "bangordailynews.com      1180.571429   712.142857        0.000000   \n",
       "befunky.com              1301.000000   657.000000        0.000000   \n",
       "bratano.com              1518.272727   789.227273        0.447368   \n",
       "breakfastdailynews.com   1317.190476   673.238095        0.594937   \n",
       "brolycow.com             1171.750000   909.250000        0.333333   \n",
       "cableone.net             1730.423077   859.653846        0.000000   \n",
       "cincinnatibell.net       1497.454545   810.181818        0.000000   \n",
       "cinesport.com            1213.532468   851.272727        0.029197   \n",
       "clearguides.com          1159.000000   688.636364        0.181818   \n",
       "collegedose.com          1294.800000   609.000000        0.473684   \n",
       "cruisecritic.com         1568.000000   794.000000        0.000000   \n",
       "cupidspulse.com          1101.157895   724.684211        0.000000   \n",
       "dailycaller.com          1381.260870   738.782609        0.250000   \n",
       "direttanews.it           1549.851852   774.000000        0.250000   \n",
       "dressupdream.com          929.000000   615.969697        0.200000   \n",
       "eenadu.net               1298.250000   733.000000        0.214286   \n",
       "essentialkids.comau      1470.828571   711.828571        0.128571   \n",
       "famousbirthdays.com       880.000000   715.333333        0.722222   \n",
       "fashion-feed.net         1930.000000  1044.000000        0.000000   \n",
       "...                              ...          ...             ...   \n",
       "supgamer.com             1141.166667   757.000000        0.034483   \n",
       "surlybirds.com           1214.000000   784.800000        0.058824   \n",
       "swagbucks.com            1423.750000   717.500000        0.520000   \n",
       "tds.net                  1052.444444   691.888889        0.363636   \n",
       "technicultr.com          1273.184211   708.078947        0.104167   \n",
       "techniqueworld.com       1170.230769   661.538462        0.000000   \n",
       "thenewscenter.tv         1440.000000   775.000000        0.000000   \n",
       "theplaystationgames.com  1279.272727   653.090909        0.767677   \n",
       "theseaapp.com            2399.500000  1222.000000        0.000000   \n",
       "thosefreegames.com       1209.250000   836.000000        0.071429   \n",
       "tomandlorenzo.com        1661.642857   862.678571        0.382353   \n",
       "tooshocking.com          1269.333333  1556.000000        0.138889   \n",
       "topplanetnews.com        1538.571429   777.071429        0.000000   \n",
       "toshiba.com              1339.281633   665.302041        0.196133   \n",
       "traileraddict.com        1081.434783   648.695652        0.342105   \n",
       "ubore.com                1013.642857   664.785714        0.058824   \n",
       "uinterview.com           1039.100000   771.700000        0.428571   \n",
       "ussportbox.com              0.000000     0.000000        0.058824   \n",
       "vaccint.com              1116.058824   643.000000        0.000000   \n",
       "veevr.com                1342.750000   721.750000        0.074074   \n",
       "viki.com                 1369.555556   712.444444        0.942857   \n",
       "virgilio.it              1241.235294   629.176471        0.083333   \n",
       "waff.com                 1352.904762   818.047619        0.081081   \n",
       "wagbrag.com              1232.279070   694.209302        0.274510   \n",
       "weeklyfit.com            1080.533333   786.133333        0.018519   \n",
       "wmur.com                 1517.090909   722.318182        0.000000   \n",
       "womenwholovebooks.com    1344.000000   714.500000        0.933333   \n",
       "wpcentral.com            1451.123762   849.559406        0.033816   \n",
       "wvydeo.com                209.571429   124.500000        0.000000   \n",
       "zanru.com                1095.937500   706.562500        0.037037   \n",
       "\n",
       "                         Browser_Firefox  Browser_Internet Explorer  \\\n",
       "Host                                                                  \n",
       "50connect.couk                  0.012346                   0.938272   \n",
       "5mostpopular.com                0.000000                   1.000000   \n",
       "advanceautoparts.com            0.000000                   0.793103   \n",
       "advcm.com                       0.000000                   0.000000   \n",
       "affhealth.com                   0.000000                   0.222222   \n",
       "ahaber.comtr                    0.000000                   1.000000   \n",
       "alldayfashions.com              0.090909                   0.636364   \n",
       "americanprofile.com             0.014085                   0.852113   \n",
       "artinstitutes.edu               0.000000                   1.000000   \n",
       "avimoya.com                     0.000000                   0.916667   \n",
       "babiesrus.com                   0.000000                   0.923077   \n",
       "bangordailynews.com             0.090909                   0.909091   \n",
       "befunky.com                     0.000000                   0.952381   \n",
       "bratano.com                     0.026316                   0.526316   \n",
       "breakfastdailynews.com          0.000000                   0.379747   \n",
       "brolycow.com                    0.250000                   0.416667   \n",
       "cableone.net                    0.000000                   1.000000   \n",
       "cincinnatibell.net              0.066667                   0.933333   \n",
       "cinesport.com                   0.000000                   0.919708   \n",
       "clearguides.com                 0.000000                   0.818182   \n",
       "collegedose.com                 0.000000                   0.526316   \n",
       "cruisecritic.com                0.181818                   0.818182   \n",
       "cupidspulse.com                 0.000000                   1.000000   \n",
       "dailycaller.com                 0.083333                   0.666667   \n",
       "direttanews.it                  0.000000                   0.750000   \n",
       "dressupdream.com                0.000000                   0.800000   \n",
       "eenadu.net                      0.000000                   0.714286   \n",
       "essentialkids.comau             0.000000                   0.871429   \n",
       "famousbirthdays.com             0.000000                   0.277778   \n",
       "fashion-feed.net                0.000000                   1.000000   \n",
       "...                                  ...                        ...   \n",
       "supgamer.com                    0.000000                   0.965517   \n",
       "surlybirds.com                  0.000000                   0.941176   \n",
       "swagbucks.com                   0.040000                   0.360000   \n",
       "tds.net                         0.000000                   0.636364   \n",
       "technicultr.com                 0.062500                   0.833333   \n",
       "techniqueworld.com              0.000000                   0.956522   \n",
       "thenewscenter.tv                0.000000                   1.000000   \n",
       "theplaystationgames.com         0.000000                   0.232323   \n",
       "theseaapp.com                   0.000000                   0.964286   \n",
       "thosefreegames.com              0.000000                   0.892857   \n",
       "tomandlorenzo.com               0.000000                   0.558824   \n",
       "tooshocking.com                 0.083333                   0.750000   \n",
       "topplanetnews.com               0.000000                   0.944444   \n",
       "toshiba.com                     0.000000                   0.787293   \n",
       "traileraddict.com               0.000000                   0.657895   \n",
       "ubore.com                       0.000000                   0.882353   \n",
       "uinterview.com                  0.000000                   0.571429   \n",
       "ussportbox.com                  0.000000                   0.941176   \n",
       "vaccint.com                     0.000000                   0.909091   \n",
       "veevr.com                       0.000000                   0.913580   \n",
       "viki.com                        0.057143                   0.000000   \n",
       "virgilio.it                     0.083333                   0.833333   \n",
       "waff.com                        0.000000                   0.891892   \n",
       "wagbrag.com                     0.019608                   0.627451   \n",
       "weeklyfit.com                   0.000000                   0.981481   \n",
       "wmur.com                        0.000000                   1.000000   \n",
       "womenwholovebooks.com           0.000000                   0.066667   \n",
       "wpcentral.com                   0.106280                   0.859903   \n",
       "wvydeo.com                      0.000000                   0.928571   \n",
       "zanru.com                       0.000000                   0.962963   \n",
       "\n",
       "                         Browser_Opera  Browser_Other  Browser_Safari/Webkit  \n",
       "Host                                                                          \n",
       "50connect.couk                       0       0.037037               0.000000  \n",
       "5mostpopular.com                     0       0.000000               0.000000  \n",
       "advanceautoparts.com                 0       0.000000               0.000000  \n",
       "advcm.com                            0       0.000000               0.000000  \n",
       "affhealth.com                        0       0.000000               0.000000  \n",
       "ahaber.comtr                         0       0.000000               0.000000  \n",
       "alldayfashions.com                   0       0.000000               0.000000  \n",
       "americanprofile.com                  0       0.000000               0.000000  \n",
       "artinstitutes.edu                    0       0.000000               0.000000  \n",
       "avimoya.com                          0       0.000000               0.000000  \n",
       "babiesrus.com                        0       0.000000               0.000000  \n",
       "bangordailynews.com                  0       0.000000               0.000000  \n",
       "befunky.com                          0       0.047619               0.000000  \n",
       "bratano.com                          0       0.000000               0.000000  \n",
       "breakfastdailynews.com               0       0.012658               0.012658  \n",
       "brolycow.com                         0       0.000000               0.000000  \n",
       "cableone.net                         0       0.000000               0.000000  \n",
       "cincinnatibell.net                   0       0.000000               0.000000  \n",
       "cinesport.com                        0       0.000000               0.051095  \n",
       "clearguides.com                      0       0.000000               0.000000  \n",
       "collegedose.com                      0       0.000000               0.000000  \n",
       "cruisecritic.com                     0       0.000000               0.000000  \n",
       "cupidspulse.com                      0       0.000000               0.000000  \n",
       "dailycaller.com                      0       0.000000               0.000000  \n",
       "direttanews.it                       0       0.000000               0.000000  \n",
       "dressupdream.com                     0       0.000000               0.000000  \n",
       "eenadu.net                           0       0.047619               0.023810  \n",
       "essentialkids.comau                  0       0.000000               0.000000  \n",
       "famousbirthdays.com                  0       0.000000               0.000000  \n",
       "fashion-feed.net                     0       0.000000               0.000000  \n",
       "...                                ...            ...                    ...  \n",
       "supgamer.com                         0       0.000000               0.000000  \n",
       "surlybirds.com                       0       0.000000               0.000000  \n",
       "swagbucks.com                        0       0.040000               0.040000  \n",
       "tds.net                              0       0.000000               0.000000  \n",
       "technicultr.com                      0       0.000000               0.000000  \n",
       "techniqueworld.com                   0       0.000000               0.043478  \n",
       "thenewscenter.tv                     0       0.000000               0.000000  \n",
       "theplaystationgames.com              0       0.000000               0.000000  \n",
       "theseaapp.com                        0       0.035714               0.000000  \n",
       "thosefreegames.com                   0       0.035714               0.000000  \n",
       "tomandlorenzo.com                    0       0.058824               0.000000  \n",
       "tooshocking.com                      0       0.000000               0.027778  \n",
       "topplanetnews.com                    0       0.055556               0.000000  \n",
       "toshiba.com                          0       0.016575               0.000000  \n",
       "traileraddict.com                    0       0.000000               0.000000  \n",
       "ubore.com                            0       0.058824               0.000000  \n",
       "uinterview.com                       0       0.000000               0.000000  \n",
       "ussportbox.com                       0       0.000000               0.000000  \n",
       "vaccint.com                          0       0.090909               0.000000  \n",
       "veevr.com                            0       0.000000               0.012346  \n",
       "viki.com                             0       0.000000               0.000000  \n",
       "virgilio.it                          0       0.000000               0.000000  \n",
       "waff.com                             0       0.027027               0.000000  \n",
       "wagbrag.com                          0       0.039216               0.039216  \n",
       "weeklyfit.com                        0       0.000000               0.000000  \n",
       "wmur.com                             0       0.000000               0.000000  \n",
       "womenwholovebooks.com                0       0.000000               0.000000  \n",
       "wpcentral.com                        0       0.000000               0.000000  \n",
       "wvydeo.com                           0       0.000000               0.071429  \n",
       "zanru.com                            0       0.000000               0.000000  \n",
       "\n",
       "[125 rows x 16 columns]"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nfhost_df.loc[model[0].predict(nfhost_df[predictors]).astype(bool)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.99183673  0.99183673  0.99183673  0.99180328  0.99180328  0.99180328\n",
      "  0.99180328  0.99588477]\n",
      "0.992326011562\n"
     ]
    }
   ],
   "source": [
    "from sklearn import cross_validation\n",
    "alg = RandomForestClassifier(random_state=1, n_estimators=10, min_samples_split=2, min_samples_leaf=1)\n",
    "scores = cross_validation.cross_val_score(alg, allhost_df[predictors], allhost_df[\"Fraud\"], cv=8)\n",
    "print scores\n",
    "print scores.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.99795291709314227"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_fit.score(allhost_df[predictors], allhost_df['Fraud'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15.0\n",
      "[0 1 2 3 4]\n",
      "15.0\n",
      "[652 653 654 655 656]\n",
      "0.0\n",
      "[1303 1304 1305 1306 1307]\n"
     ]
    }
   ],
   "source": [
    "kf_nf = cross_validation.KFold(allhost_df.shape[0], n_folds=3, random_state=1)\n",
    "for train, test in kf_nf:\n",
    "    print allhost_df['Fraud'].iloc[train].sum()\n",
    "    print test[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectKBest, f_classif\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "rows=np.random.choice(df.index.values,15,replace=False)\n",
    "\n",
    "\n",
    "cross_validation.StratifiedShuffleSplit\n",
    "# The algorithms we want to ensemble.\n",
    "# We're using the more linear predictors for the logistic regression, and everything with the gradient boosting classifier.\n",
    "algorithms = [\n",
    "    [GradientBoostingClassifier(random_state=1, n_estimators=25, max_depth=3), predictors],\n",
    "    [LogisticRegression(random_state=1), predictors]\n",
    "]\n",
    "\n",
    "# Initialize the cross validation folds\n",
    "kf_nf = KFold(titanic.shape[0], n_folds=3, random_state=1)\n",
    "kf_f = KFold()\n",
    "predictions = []\n",
    "for train, test in kf:\n",
    "    train_target = titanic[\"Survived\"].iloc[train]\n",
    "    full_test_predictions = []\n",
    "    # Make predictions for each algorithm on each fold\n",
    "    for alg, predictors in algorithms:\n",
    "        # Fit the algorithm on the training data.\n",
    "        alg.fit(titanic[predictors].iloc[train,:], train_target)\n",
    "        # Select and predict on the test fold.  \n",
    "        # The .astype(float) is necessary to convert the dataframe to all floats and avoid an sklearn error.\n",
    "        test_predictions = alg.predict_proba(titanic[predictors].iloc[test,:].astype(float))[:,1]\n",
    "        full_test_predictions.append(test_predictions)\n",
    "    # Use a simple ensembling scheme -- just average the predictions to get the final classification.\n",
    "    test_predictions = (full_test_predictions[0] + full_test_predictions[1]) / 2\n",
    "    # Any value over .5 is assumed to be a 1 prediction, and below .5 is a 0 prediction.\n",
    "    test_predictions[test_predictions <= .5] = 0\n",
    "    test_predictions[test_predictions > .5] = 1\n",
    "    predictions.append(test_predictions)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1952, 16)\n",
      "            Inview     Nplugins       Latency        Fraud         Tsecs  \\\n",
      "count  1952.000000  1952.000000   1952.000000  1952.000000   1952.000000   \n",
      "mean      0.457692     9.597534    379.542030     0.006660  48746.020906   \n",
      "std       0.239451     5.585981    744.015352     0.081356  13205.936813   \n",
      "min       0.000000     1.000000      0.000000     0.000000   2108.250000   \n",
      "25%       0.285714     5.500000     97.904227     0.000000  41007.492898   \n",
      "50%       0.444444     9.803813    181.245833     0.000000  49650.151030   \n",
      "75%       0.624403    13.000000    364.135417     0.000000  57279.096154   \n",
      "max       1.000000    37.000000  12660.235294     1.000000  84369.000000   \n",
      "\n",
      "              Warea         WXpos         WYpos        Wwidth       Wheight  \\\n",
      "count  1.952000e+03  1.952000e+03  1.952000e+03  1.952000e+03  1.952000e+03   \n",
      "mean   1.041620e+14  7.655938e+04  7.655125e+04  7.040713e+04  6.990065e+04   \n",
      "std    4.586037e+15  3.359440e+06  3.359440e+06  3.071944e+06  3.071946e+06   \n",
      "min    0.000000e+00 -3.200000e+04 -3.200000e+04 -8.052228e+05 -8.057189e+05   \n",
      "25%    8.310409e+05 -5.333333e-01 -5.091575e-01  1.139553e+03  6.768250e+02   \n",
      "50%    1.000746e+06  0.000000e+00  0.000000e+00  1.300763e+03  7.488712e+02   \n",
      "75%    1.191766e+06  0.000000e+00  0.000000e+00  1.420582e+03  8.396023e+02   \n",
      "max    2.026170e+17  1.484231e+08  1.484231e+08  1.357204e+08  1.357200e+08   \n",
      "\n",
      "       Browser_Chrome  Browser_Firefox  Browser_Internet Explorer  \\\n",
      "count     1952.000000      1952.000000                1952.000000   \n",
      "mean         0.350541         0.153001                   0.277418   \n",
      "std          0.300579         0.238711                   0.300963   \n",
      "min          0.000000         0.000000                   0.000000   \n",
      "25%          0.083333         0.000000                   0.000000   \n",
      "50%          0.300000         0.052178                   0.171057   \n",
      "75%          0.555556         0.200000                   0.454545   \n",
      "max          1.000000         1.000000                   1.000000   \n",
      "\n",
      "       Browser_Opera  Browser_Other  Browser_Safari/Webkit  \n",
      "count    1952.000000    1952.000000            1952.000000  \n",
      "mean        0.001506       0.008282               0.209253  \n",
      "std         0.027665       0.032870               0.266631  \n",
      "min         0.000000       0.000000               0.000000  \n",
      "25%         0.000000       0.000000               0.000000  \n",
      "50%         0.000000       0.000000               0.101839  \n",
      "75%         0.000000       0.000000               0.300000  \n",
      "max         1.000000       0.600000               1.000000  \n",
      "Number of orig fraud host:  15\n",
      "Number of fraud hosts now:  13\n"
     ]
    }
   ],
   "source": [
    "final_df = dum_imp_df.groupby('Host').mean().dropna()\n",
    "print final_df.shape\n",
    "print final_df.describe()\n",
    "print 'Number of orig fraud host: ', len(fraudtraffic)\n",
    "print 'Number of fraud hosts now: ', sum(final_df['Fraud']==1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              Inview       Nplugins         Latency         Warea  \\\n",
      "count  211810.000000  162303.000000   200279.000000  1.184500e+05   \n",
      "mean        0.431996      10.139424      345.189685  3.765405e+13   \n",
      "std         0.495355       7.637193    13292.948056  9.158285e+15   \n",
      "min         0.000000       1.000000        0.000000  0.000000e+00   \n",
      "25%         0.000000       2.000000        0.000000  7.144180e+05   \n",
      "50%         0.000000      10.000000       54.000000  9.247820e+05   \n",
      "75%         1.000000      15.000000      179.000000  1.262400e+06   \n",
      "max         1.000000      61.000000  5866142.000000  2.228787e+18   \n",
      "\n",
      "               Tsecs          Fraud         WXpos         WYpos        Wwidth  \\\n",
      "count  234945.000000  234945.000000  1.184500e+05  1.184500e+05  1.184500e+05   \n",
      "mean    48780.596480       0.010611  2.810657e+04  2.809878e+04  2.552057e+04   \n",
      "std     21935.404297       0.102462  6.709700e+06  6.709700e+06  6.136262e+06   \n",
      "min         0.000000       0.000000 -4.000000e+04 -4.000000e+04 -2.257769e+07   \n",
      "25%     34279.000000       0.000000  0.000000e+00  0.000000e+00  1.032000e+03   \n",
      "50%     50375.000000       0.000000  0.000000e+00  0.000000e+00  1.292000e+03   \n",
      "75%     65784.000000       0.000000  0.000000e+00  0.000000e+00  1.462000e+03   \n",
      "max     86398.000000       1.000000  1.632654e+09  1.632654e+09  1.492912e+09   \n",
      "\n",
      "            Wheight  \n",
      "count  1.184500e+05  \n",
      "mean   2.499656e+04  \n",
      "std    6.136265e+06  \n",
      "min   -2.257769e+07  \n",
      "25%    6.310000e+02  \n",
      "50%    7.280000e+02  \n",
      "75%    8.850000e+02  \n",
      "max    1.492912e+09  \n",
      "             Inview      Nplugins        Latency         Warea         Tsecs  \\\n",
      "count  86850.000000  86850.000000   86850.000000  8.685000e+04  86850.000000   \n",
      "mean       0.477524     10.356661     269.620956  5.134844e+13  48518.845607   \n",
      "std        0.499497      7.630489    1752.692684  1.069538e+16  21796.881922   \n",
      "min        0.000000      1.000000       0.000000  0.000000e+00      0.000000   \n",
      "25%        0.000000      2.000000       0.000000  7.594560e+05  34372.000000   \n",
      "50%        0.000000     10.000000       0.000000  9.376640e+05  49836.500000   \n",
      "75%        1.000000     16.000000     141.000000  1.275456e+06  65573.500000   \n",
      "max        1.000000     61.000000  221849.000000  2.228787e+18  86398.000000   \n",
      "\n",
      "              Fraud         WXpos         WYpos        Wwidth       Wheight  \n",
      "count  86850.000000  8.685000e+04  8.685000e+04  8.685000e+04  8.685000e+04  \n",
      "mean       0.014727  3.817669e+04  3.816885e+04  3.466045e+04  3.409590e+04  \n",
      "std        0.120457  7.835602e+06  7.835602e+06  7.165740e+06  7.165743e+06  \n",
      "min        0.000000 -4.000000e+04 -4.000000e+04 -2.257769e+07 -2.257769e+07  \n",
      "25%        0.000000  0.000000e+00  0.000000e+00  1.111000e+03  6.430000e+02  \n",
      "50%        0.000000  0.000000e+00  0.000000e+00  1.344000e+03  7.310000e+02  \n",
      "75%        0.000000  0.000000e+00  0.000000e+00  1.536000e+03  8.960000e+02  \n",
      "max        1.000000  1.632654e+09  1.632654e+09  1.492912e+09  1.492912e+09  \n"
     ]
    }
   ],
   "source": [
    "cols_OI = ['Browser','Host','Inview','Nplugins','Latency','Warea','Tsecs','Fraud','WXpos','WYpos','Wwidth','Wheight']\n",
    "int_imp_df = imp_df.loc[:,cols_OI]\n",
    "print int_imp_df.describe()\n",
    "int_imp_df.dropna(inplace=True)\n",
    "print int_imp_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    85571\n",
       "1     1279\n",
       "Name: Fraud, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_imp_df.Fraud.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "axes1 = pd.tools.plotting.scatter_matrix(int_imp_df.loc[int_imp_df['Fraud']==1,:], color=\"brown\")\n",
    "f1 = plt.gcf()\n",
    "f1.set_size_inches(10,8)\n",
    "axes2 = pd.tools.plotting.scatter_matrix(int_imp_df.loc[int_imp_df['Fraud']==0,:], color=\"blue\")\n",
    "f2 = plt.gcf()\n",
    "f2.set_size_inches(10,8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "imp_df['Fraud']=0\n",
    "imp_df.loc[imp_df[\"Host\"].isin(fraudtraffic),'Fraud']=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fdata = imp_df[imp_df[\"Host\"].isin(fraudtraffic)]\n",
    "nfdata = imp_df[~imp_df[\"Host\"].isin(fraudtraffic)]\n",
    "#ax = fdata.plot(kind='scatter', x=\"UA_Browser_Ver\", y=\"Warea\", style=\"o\")\n",
    "#ax2 = nfdata.plot(kind='scatter', x='UA_Browser_Ver', y='Warea')\n",
    "#f, axs = plt.subplots(1, 2)\n",
    "#fdata[\"Nplugins\"].value_counts(normalize=True).plot(kind='bar',ax=axs[0])\n",
    "axes = fdata.boxplot(column=\"Nplugins\",by=\"Browser\")\n",
    "axes.set_title('Nplugins (fraud)')\n",
    "plt.ylim(0,35)\n",
    "#nfdata[\"Nplugins\"].value_counts(normalize=True).plot(kind='bar',ax=axs[1])\n",
    "axes2 = nfdata.boxplot(column=\"Nplugins\",by=\"Browser\")\n",
    "axes2.set_title('Nplugins (nonfraud)')\n",
    "plt.ylim(0,35)\n",
    "\n",
    "plt.figure()\n",
    "fdata[\"Browser\"].value_counts(normalize=True).sort_index().plot(kind='bar',color=\"brown\")\n",
    "plt.axis.set_title('Browser Counts (Fraud)')\n",
    "plt.figure()\n",
    "nfdata[\"Browser\"].value_counts(normalize=True).sort_index().plot(kind='bar',color=\"blue\")\n",
    "\n",
    "#plt.figure()\n",
    "#imp_df['Browser'].value_counts().plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pd.tools.plotting.scatter_matrix(fdata[['Nplugins','Warea','Browser','Inview']], color=\"brown\")\n",
    "pd.tools.plotting.scatter_matrix(nfdata[['Nplugins','Warea','Browser','Inview']], color=\"blue\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print imp_df.describe()\n",
    "print fdata['Warea'].describe()\n",
    "print nfdata['Warea'].describe()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assumptions\n",
    "* logs are correct. I assume that when there are observations of one IP address clicking on the same host 3 times in a second, that this is what occurred. It may be the case that some entries are simply repeated due to errors in the recording.\n",
    "* ERROR hosts are to be disregarded. As there is no valid host, the only information is IP address and Browser/User Agent info\n",
    "* That greatboxgames.com should be greatxboxgames.com\n",
    "\n",
    "### Cleaning and Feature Engineering\n",
    "* Remove ERROR hosts\n",
    "* Standardize hosts by removing preceeding 'http://www.|go.|video.'\n",
    "* Convert timestamp to seconds in the day\n",
    "* Create browser window area feature\n",
    "* Create inter-impression interval (III) feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fraudtraffic=[\"featureplay.com\",\"uvidi.com\",\"spryliving.com\",\"greatxboxgames.com\",\n",
    "              \"mmabay.co.uk\",\"workingmothertv.com\",\"besthorrorgame.com\",\"dailyparent.com\",\"superior-movies.com\",\n",
    "              \"yourhousedesign.com\",\"outdoorlife.tv\",\"drumclub.info\",\"cycleworld.tv\",\"hmnp.us\",\"nlinevideos.com\"]\n",
    "\n",
    "print \"IP counts:\\n\", data[data[\"Host\"].isin(fraudtraffic)][\"IPadd\"].value_counts().describe()\n",
    "print \"Host counts:\\n\", data[data[\"Host\"].isin(fraudtraffic)][\"Host\"].value_counts().describe()\n",
    "\n",
    "fdata = data[data[\"Host\"].isin(fraudtraffic)]\n",
    "nfdata = data[~data[\"Host\"].isin(fraudtraffic)]\n",
    "\n",
    "f, axs = plt.subplots(1, 2)\n",
    "fdata[\"Tsecs\"].value_counts(normalize=True).plot(kind='bar',ax=axs[0])\n",
    "axs[0].set_title('Seconds (fraud)')\n",
    "nfdata[\"Tsecs\"].value_counts(normalize=True).plot(kind='bar',ax=axs[1])\n",
    "axs[1].set_title('Seconds (nonfraud)')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "#Reduce the data - Remove Hosts that are present less than 8 times\n",
    "host_counts = data[\"Host\"].value_counts()\n",
    "host_counts\n",
    "host_counts.describe()\n",
    "sighost_counts = host_counts[host_counts>8]\n",
    "keepHosts = sighost_counts.index.map(lambda x: str(x))\n",
    "data = data[data[\"Host\"].isin(keepHosts)]\n",
    "\n",
    "\n",
    "#Reduce the data - Remove IPs that are present less than 17 times\n",
    "IP_counts = data[\"IPadd\"].value_counts()\n",
    "IP_counts\n",
    "IP_counts.describe()\n",
    "sigIP_counts = IP_counts[IP_counts>17]\n",
    "keepIPs = sigIP_counts.index.map(lambda x: str(x))\n",
    "data = data[data[\"IPadd\"].isin(keepIPs)]\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_host_counts(df,hostname,feat_name):\n",
    "    '''\n",
    "        Selects feature column out given hostname\n",
    "    '''\n",
    "    count = df[df[\"Host\"]==hostname][feat_name].value_counts()\n",
    "    return count \n",
    "\n",
    "def get_hosts_counts(df,hostnames,feat_name):\n",
    "    '''\n",
    "        Selects feature column out for list of hostnames\n",
    "    '''\n",
    "    count=data[data[\"Host\"].isin(hostnames)][feat_name].value_counts()\n",
    "    return count \n",
    "\n",
    "def naive_bayes_fraud(Fcounts,NFcounts,Tcounts):\n",
    "    '''\n",
    "        Runs naive bayes, or a modified version that returns a score, not a probability\n",
    "    '''\n",
    "    vocab = NFcounts.add(Fcounts,fill_value=0)\n",
    "    Fprior = .625\n",
    "    NFprior = .375\n",
    "    prob_fraud = 0.0\n",
    "    log_prob_fraud = 0.0\n",
    "    prob_notfraud = 0.0\n",
    "    log_prob_notfraud = 0.0    \n",
    "    for ind, val in Tcounts.iteritems():\n",
    "        if ind in vocab.index:\n",
    "            p_value = (vocab[ind]+0.0)/vocab.sum()\n",
    "            #print \"Prob. of value: \", p_value\n",
    "            #p_v_given_fraud = (Fcounts.get(ind,0.0)+0.0)/Fcounts.sum()\n",
    "            p_v_given_fraud = (Fcounts.get(ind,0.0)+1.0)/(Fcounts.sum()+len(vocab))\n",
    "            #print \"Prob. of val | fraud: \", p_v_given_fraud\n",
    "            #p_v_given_notfraud = (NFcounts.get(ind,0.0)+0.0)/NFcounts.sum()\n",
    "            p_v_given_notfraud = (NFcounts.get(ind,0.0)+1.0)/(NFcounts.sum()+len(vocab))\n",
    "            #print \"Prob. of val | not fraud: \", p_v_given_notfraud\n",
    "        else:\n",
    "            p_value = 1.0/(vocab.sum()+1)\n",
    "            p_v_given_fraud = 1.0/(Fcounts.sum()+len(vocab))\n",
    "            p_v_given_notfraud = 1.0/(NFcounts.sum()+len(vocab))\n",
    "        if p_v_given_fraud > 0:\n",
    "            prob_fraud += (val * p_v_given_fraud) / p_value\n",
    "            log_prob_fraud += math.log(val * p_v_given_fraud / p_value)\n",
    "        if p_v_given_notfraud >0:\n",
    "            prob_notfraud += (val * p_v_given_notfraud) / p_value\n",
    "            log_prob_notfraud += math.log(val * p_v_given_notfraud / p_value)\n",
    "    '''\n",
    "    print \"\\nFraud Score:  \", (prob_fraud*Fprior)/(prob_fraud*Fprior+prob_notfraud*NFprior)\n",
    "    print \"SumProb. (fraud):  \", prob_fraud + Fprior\n",
    "    print \"SumProb. (not fraud):  \", prob_notfraud + NFprior\n",
    "    print \"LogScore (fraud):  \", log_prob_fraud + math.log(Fprior)\n",
    "    print \"LogScore (not fraud):  \", log_prob_notfraud + math.log(NFprior)\n",
    "    print \"Fscore : \", np.exp(log_prob_fraud+math.log(Fprior)-(log_prob_fraud+math.log(Fprior)+log_prob_notfraud+math.log(NFprior)))   \n",
    "    exp_prob_fraud = np.exp(log_prob_fraud + math.log(Fprior))\n",
    "    exp_prob_notfraud = np.exp(log_prob_notfraud + math.log(NFprior))\n",
    "    print \"Likelihood of Fraud(exp):  \", exp_prob_fraud/(exp_prob_fraud+exp_prob_notfraud)#(log_prob_fraud + math.log(Fprior))/(log_prob_fraud + math.log(Fprior)+log_prob_notfraud + math.log(NFprior))\n",
    "    print \"Likelihood of Fraud(log):  \", (log_prob_fraud + math.log(Fprior))/(log_prob_fraud + math.log(Fprior)+log_prob_notfraud + math.log(NFprior))\n",
    "    '''\n",
    "    return prob_fraud/(prob_fraud+prob_notfraud)\n",
    "\n",
    "def run_fraudscore(df,host,feat_name):\n",
    "    '''\n",
    "        Runs naive_bayes_fraud for a host. Checks to see if it is in training set and removes it if True\n",
    "    '''\n",
    "    fraudtraffic=[\"featureplay.com\",\"uvidi.com\",\"spryliving.com\",\"greatxboxgames.com\",\n",
    "              \"mmabay.co.uk\",\"workingmothertv.com\",\"besthorrorgame.com\",\"dailyparent.com\",\"superior-movies.com\",\n",
    "              \"yourhousedesign.com\",\"outdoorlife.tv\",\"drumclub.info\",\"cycleworld.tv\",\"hmnp.us\",\"nlinevideos.com\"]\n",
    "    nfraudtraffic=[\"google.com\",\"foxsports.com\",\"washingtonpost.com\",\"amazon.com\",\n",
    "               \"nytimes.com\",\"tvguide.com\",\"pandora.com\",\"youtube.com\",\"cnn.com\"]\n",
    "    if host in fraudtraffic:\n",
    "        fraudtraffic.remove(host)\n",
    "    if host in nfraudtraffic:\n",
    "        nfraudtraffic.remove(host)\n",
    "    F_count = get_hosts_counts(data,fraudtraffic,feat_name)\n",
    "    NF_count = get_hosts_counts(data,nfraudtraffic,feat_name)\n",
    "    T_count = get_host_counts(data,host,feat_name)\n",
    "    return naive_bayes_fraud(F_count,NF_count,T_count)\n",
    "\n",
    "def clean_plot(color='w',ax=plt.gca(), leftAxisOn=True):\n",
    "\t# Make a cleaner, prettier plot\n",
    "\tax.set_axis_bgcolor('w')\n",
    "\tax.spines['bottom'].set_color(color)\n",
    "\tax.spines['top'].set_color(color) \n",
    "\tax.spines['right'].set_color(color)\n",
    "\tax.spines['left'].set_color(color)\n",
    "\tif leftAxisOn is True:\n",
    "\t\tax.spines['left'].set_color((0.5,0.5,0.5))\n",
    "\t\tax.yaxis.set_ticks_position('left')\n",
    "\t\tax.get_yaxis().set_tick_params(direction='out',color=(0.5,0.5,0.5),length=3.5)\n",
    "\n",
    "\n",
    "def runallhost_fraudscore(data,feat_name):\n",
    "    # Run fraudscores for each host   \n",
    "    allhosts = pd.Series(data[\"Host\"].ravel()).unique()\n",
    "    fraudscores = np.zeros_like(allhosts)\n",
    "    for i,h in enumerate(allhosts):\n",
    "        #print i, \"out of\", len(allhosts)\n",
    "        fraudscores[i]=run_fraudscore(data,h,feat_name)\n",
    "\n",
    "    host_score_prob = pd.DataFrame({'host' : allhosts,'fscore' : fraudscores})\n",
    "    host_score_prob.sort('fscore',ascending=False)\n",
    "    \n",
    "    #sort the scores and reindex    \n",
    "    sort_host_score_prob = host_score_prob.sort_index(by='fscore',ascending=False)\n",
    "    sort_host_score_prob.index=range(1,len(sort_host_score_prob)+1)\n",
    "\n",
    "    fraudtraffic=[\"featureplay.com\",\"uvidi.com\",\"spryliving.com\",\"greatxboxgames.com\",\n",
    "              \"mmabay.co.uk\",\"workingmothertv.com\",\"besthorrorgame.com\",\"dailyparent.com\",\"superior-movies.com\",\n",
    "              \"yourhousedesign.com\",\"outdoorlife.tv\",\"drumclub.info\",\"cycleworld.tv\",\"hmnp.us\",\"nlinevideos.com\"]\n",
    "       \n",
    "    #plot results\n",
    "    fig = plt.figure(figsize=(16,8))\n",
    "    ax = plt.axes([.125, .2, .775, .7])\n",
    "    plt.title('Fraud Scores from '+feat_name+' for known Fraudulent')\n",
    "    plt.ylabel('Fraud Score Percentile', size=12)\n",
    "    textFont  = {'family' : u'sans-serif',\n",
    "         'size'   : 12,\n",
    "         'style'  : u'italic' }\t\t\n",
    "    rects = plt.bar(np.arange(15),100-(sort_host_score_prob[sort_host_score_prob['host'].isin(fraudtraffic)].index/(len(sort_host_score_prob)+0.0))*100,color='r')\n",
    "    plt.xticks(np.arange(15)+0.5, sort_host_score_prob[sort_host_score_prob['host'].isin(fraudtraffic)].host.values, rotation=60,\n",
    "    \thorizontalalignment='right', **textFont)\n",
    "    \n",
    "    for rect in rects:\n",
    "        height = int(rect.get_height())\n",
    "        rankStr = str(height)\n",
    "        xloc = rect.get_x()+rect.get_width()/2.0\n",
    "        yloc = 0.95*height\n",
    "        ax.text(xloc, yloc, rankStr, horizontalalignment='center',\n",
    "                verticalalignment='center', color='white', weight='bold')\n",
    "    clean_plot(ax=ax)\n",
    "    \n",
    "    f = open('Integral_data_host_fraud_ranks'+feat_name+'_noreduction.txt', 'w')\n",
    "    for ind, fsc, hst in sort_host_score_prob.itertuples():\n",
    "        f.write(str(ind) + ' ' + hst + '\\n')\n",
    "    f.close()\n",
    "    \n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    print 'Loading data'\n",
    "    log_file = \"D:\\Downloads\\Integral_data_set.tsv\"\n",
    "    data = pd.io.parsers.read_csv(log_file,sep='\\t',names=['Timestamp','IPadd','Browser','UserA','Host','Iinview','Nplugins','Bwinpossize','NetLat'],header=None)\n",
    "    \n",
    "    print 'Cleaning data'\n",
    "    data = clean_data(data)\n",
    "    print 'Adding features'\n",
    "    data = add_features(data)\n",
    "    print 'Reducing data'\n",
    "    data = reduce_data(data)\n",
    "    \n",
    "    for f in ['Browser','UserA','Iinview','Nplugins','Bwinarea']:\n",
    "        print 'Computing fraud scores for '+f\n",
    "        runallhost_fraudscore(data,f)\n",
    "        plt.savefig('KnownFraudulent'+f+'FraudScore.pdf')\n",
    "    \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}