hypothesis_testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hypothesis Testing\n",
    "\n",
    "[Hypothesis testing](#Hypothesis)<br/>\n",
    "[AB testing](#AB)<br/>\n",
    "[Z-test](#Z)<br/>\n",
    "[T-test](#T)<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import division\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='Hypothesis'></a>\n",
    "### Hypothesis Testing\n",
    "\n",
    "- State null hypothesis H0 (typically status quo, no effect)\n",
    "- Choose a significance level alpha\n",
    "- Choose and compute appropriate test statistics\n",
    "- Compute p-value and 'reject' or 'fail to reject' H0\n",
    "\n",
    "**Two sided tests**\n",
    "- Reject H0 if test statistic is in upper or lower tail\n",
    "- Compute p-value using probability of being in either tail\n",
    "\n",
    "**One sided tests**\n",
    "- Reject H0 if test statistic is in the wrong tail (advertising does not decrease sales)\n",
    "- Compute p-value using probability of being in only one tail\n",
    "\n",
    "**type I error** - `alpha`, reject H0 when it is true (convict innocent)<br/>\n",
    "**type II error** - `beta`, fail to reject H0 when it is false (release guilty)\n",
    "\n",
    "<img src=\"img/hypothesis_testing.png\" width=\"300\">\n",
    "\n",
    "|             |          | Truth        |              |\n",
    "|-------------|----------|--------------|--------------|\n",
    "|             |          | H0 true      | H0 false     |\n",
    "|**Findings** | H0 false | Type I (TN)  | Correct (TN) |\n",
    "|             | H0 true  | Correct (TP) | Type II (FN) |\n",
    "\n",
    "**p-value** - probability of observing data which is at least as extreme as hwat was observed<br/>\n",
    "**confidence interval** - if we compute CI from multiple random samples from population, then 95% will contain the true population value\n",
    "\n",
    "**power (sensitivity)** - `1-beta`, probability of not making type II error<br/>\n",
    "**significance level (type I eror)** - `alpha`, probability of rejecting H0 given that it is true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='AB'></a>\n",
    "### AB Testing\n",
    "\n",
    "Can we say if one of the landing pages is better (i.e. gets more registrations) than the other with statistical significane?\n",
    "\n",
    "```\n",
    "                 Visitors   Registrations\n",
    "Landing Page 2\t1,012,285   349,643\n",
    "Landing Page 3\t  995,750   320,432\n",
    "\n",
    "H0: Landing pages 2 and 3 are the same.\n",
    "H1: One of the pages is better.\n",
    "\n",
    "Visit > register ratio\n",
    "2: 349,643 / 1,012,285 = 0.3454\n",
    "3: 320,432 / 995,750 = 0.3218\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lp2 = np.zeros(1012285)\n",
    "lp2[0:349643] = 1\n",
    "lp3 = np.zeros(995750)\n",
    "lp3[0:320432] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ttest_indResult(statistic=35.476606227821215, pvalue=1.3745040544084776e-275)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.stats import ttest_ind\n",
    "ttest_ind(lp2, lp3, equal_var=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "P-value 1.374e-275 < 0.05, we can reject that the pages are the same (H0)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='Z'></a>\n",
    "### Z-test\n",
    "\n",
    "**z-test** - use when variance is known\n",
    "\n",
    "To Compare means (assuming they are independent) with the same standard deviation `statsmodels.ztest`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-1.0954451150103321, 0.27332167829229814)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from statsmodels.stats.weightstats import ztest\n",
    "\n",
    "ztest([2,3,4,5], [3,4,5,6])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compare means from distributions with different standard deviation use `CompareMeans.ztest_ind`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-1.0954451150103321, 0.27332167829229814)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from statsmodels.stats.weightstats import CompareMeans, DescrStatsW\n",
    "\n",
    "cm = CompareMeans(DescrStatsW([2,3,4,5]), DescrStatsW([3,4,5,6]))\n",
    "cm.ztest_ind(alternative='two-sided')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='T'></a>\n",
    "## T-test\n",
    "\n",
    "**t-test** - use when variance is unknown\n",
    "\n",
    "Calculate the T-test for the means of two independent samples of scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ttest_indResult(statistic=-0.36514837167011088, pvalue=0.72450697149417942)\n",
      "Ttest_indResult(statistic=-0.36514837167011083, pvalue=0.72446582573474294)\n"
     ]
    }
   ],
   "source": [
    "from scipy.stats import ttest_ind\n",
    "print ttest_ind([0,0,1,0,2], [0,1,1,0,2], equal_var=False) #Welch’s t-test (do not assume equal population variance)\n",
    "print ttest_ind([0,0,1,0,2], [0,1,1,0,2], equal_var=True)  #standard independent 2 sample test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}