bayesian_testing_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1: AB test \n",
    "#### We have run an email campaign testing 2 different call to actions. As a result, we obtained a click-through rate of 5% for call to action A and 4% for call to action B. Overall, we sent 5,000 emails, 3,500 in variant A and 1,500 in variant B of the call to action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>B</td>\n",
       "      <td>0.18</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A     B\n",
       "A   NaN  0.82\n",
       "B  0.18   NaN"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CI,P = bayes.computeBayesTest([0.2857142857142857,0.25],sample_size=[280,240])\n",
    "P"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "CI,P = bayes.computeBayesTest([0.05,0.04],sample_size=[3500,1500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### There is a 94% probability that variant A is better than variant B:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>B</td>\n",
       "      <td>0.06</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A     B\n",
       "A   NaN  0.94\n",
       "B  0.06   NaN"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The uplift we would get by using variant A over B is between 10% and 41% (with a 80% probability):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[1.1, 1.41]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>B</td>\n",
       "      <td>[0.71, 0.91]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              A            B\n",
       "A           NaN  [1.1, 1.41]\n",
       "B  [0.71, 0.91]          NaN"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 2: AB test with a prior \n",
    "#### Similar set up as before, but this time we want to use some previous information. Before this campaign we had run two campaigns, one with call to action A (volume of 700 emails, obtaining a click-through rate of 4%) and another one with call to action B (volume of 300 emails, click-through rate of 4.5%). We can capture this information in the form of prior probabilities through the prior_settings dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "CI,P = bayes.computeBayesTest([0.05,0.04],sample_size=[3500,1500],\n",
    "                              prior_settings={\"positive_rate_prior\":[0.04,0.045],\"sample_size_prior\":[700,300]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### There is now a 90% probability that variant A is better than variant B. \n",
    "#### This is lower than in the previous example because we now have a prior belief (derived from the two previous campaigns) that variant B should actually perform slightly better than A (4.5% vs 4%). The previous campaigns had smaller sample sizes than the latest one (500 and 300, as opposed to 1,000 for the latest campaign) so we still conclude that A is better than B with a 90% probability. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>B</td>\n",
       "      <td>0.1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B\n",
       "A  NaN  0.9\n",
       "B  0.1  NaN"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[1.05, 1.32]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>B</td>\n",
       "      <td>[0.76, 0.95]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              A             B\n",
       "A           NaN  [1.05, 1.32]\n",
       "B  [0.76, 0.95]           NaN"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 3: Multivariate test with 3 variants\n",
    "#### We have run an email campaign testing 3 different subject lines. As a result, we obtained an open rate of 35% for subject line A, 32% for subject line B and 33% for subject line C. Overall, we sent 6,000 emails, equally split among the three subject lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "CI,P = bayes.computeBayesTest([0.35,0.32,0.33],sample_size=6000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The uplift we would get by using variant A over B is between 5% and 13% (with a 80% probability). The uplift we would get by using variant A over C is between 2% and 10% (with a 80% probability)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[1.05, 1.13]</td>\n",
       "      <td>[1.02, 1.1]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>B</td>\n",
       "      <td>[0.88, 0.95]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[0.93, 1.01]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>C</td>\n",
       "      <td>[0.91, 0.98]</td>\n",
       "      <td>[0.99, 1.08]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              A             B             C\n",
       "A           NaN  [1.05, 1.13]   [1.02, 1.1]\n",
       "B  [0.88, 0.95]           NaN  [0.93, 1.01]\n",
       "C  [0.91, 0.98]  [0.99, 1.08]           NaN"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### There is a ~98% probability that variant A is better than variant B. There is a 75% probability that variant C is better than B."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.98</td>\n",
       "      <td>0.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>B</td>\n",
       "      <td>0.02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>C</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.75</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A     B     C\n",
       "A   NaN  0.98  0.91\n",
       "B  0.02   NaN  0.25\n",
       "C  0.09  0.75   NaN"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 4: Sample size calculation for an AB test\n",
    "#### We are planning an AB test to see if a new recommendation engine would drive a higher cross-sell rate across our customers. The current cross-sell rate using the old engine is 10% and we want to make sure we collect enough volume to detect a relative uplift of at least 20% (i.e. a new cross-sell of 12%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "relative_uplift = 0.2\n",
    "current_xsell_rate = 0.1\n",
    "hypotetical_new_xsell_rate = 0.1 + 0.1*relative_uplift"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The minimum sample size required is 2,237:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2237"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bayes.computeBayesTestSampleSize([current_xsell_rate,hypotetical_new_xsell_rate],min_confidence=80.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### If instead we wanted to detect a 6x relative uplift (600%), the function will inform us that the uplift we want to measure is so large that any sample size of at least 10 data points would do"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "relative_uplift = 6.\n",
    "current_xsell_rate = 0.1\n",
    "hypotetical_new_xsell_rate = 0.1 + 0.1*relative_uplift"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The values in the explored size range are all large enough to confirm that the positive rates are truly different with a 80.0% Bayesian confidence. The smallest sample size (10.0) corresponds to a Bayesian confidence of 96.9%.\n"
     ]
    }
   ],
   "source": [
    "bayes.computeBayesTestSampleSize([current_xsell_rate,hypotetical_new_xsell_rate])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### If instead we wanted to detect a very small relative uplift (0.01%), the function will inform us that the uplift we want to measure is so small that not even the largest sample size analysed (100,000,000) would be sufficient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "relative_uplift = 0.0001\n",
    "current_xsell_rate = 0.1\n",
    "hypotetical_new_xsell_rate = 0.1 + 0.1*relative_uplift"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The values in the explored size range are too small to confirm that the positive rates are truly different with a 80.0% Bayesian confidence. The largest value in the range (100000000.0) corresponds to a Bayesian confidence of 19.7%. Try smaller Bayesian confidence\n"
     ]
    }
   ],
   "source": [
    "bayes.computeBayesTestSampleSize([current_xsell_rate,hypotetical_new_xsell_rate])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More info from the docstring of the function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function computeBayesTestSampleSize in module bayes:\n",
      "\n",
      "computeBayesTestSampleSize(true_positive_rates, min_confidence=80.0, accuracy=10, prior_settings={'positive_rate_prior': None, 'sample_size_prior': 0})\n",
      "    Calculate minimum sample size required to detect a given difference between positive rates.\n",
      "    \n",
      "    When designing an AB or Multivariate test, you want to make sure that the sample size you will \n",
      "    collect is large enough to confidently conclude that the variations are different when their true \n",
      "    positive rates are sufficiently dissimilar.\n",
      "    \n",
      "    For an AB test, a way to do this within a Bayesian framework is measuring the required minimum \n",
      "    sample size that detects a difference between the two variants with at least a x% Bayesian confidence\n",
      "    at least p% of the times, given the two expected positive rates (true_positive_rates).\n",
      "    In this function, we chose p = 80% (fixed) whereas \"x\" can be selected by the user (min_confidence,\n",
      "    default value is 80%).\n",
      "    \n",
      "    For a multivariate test, the function looks at the largest and smallest positive rates in the input\n",
      "    list and then works out the sample size as in the AB test case.\n",
      "    \n",
      "    The function will search for a solution by using the bisection method. The initial search range\n",
      "    is size_range (default (100,10000)), but if the solution is not in this range the function will\n",
      "    change it accordingly up to 3 times. If no solution is found after 3 times, the function will\n",
      "    return a message explaining why.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    true_positive_rates : list of float\n",
      "        List of positive rates that are expected for each variant in the experiment.\n",
      "    min_confidence : float, optional (default=80.)\n",
      "        Minimum Bayesian posterior probability value required for a test to be considered \"passed\".\n",
      "    accuracy : int, optional (default=10)\n",
      "        Required accuracy for the min_sample_size. Smaller values correspond to higher \n",
      "        accuracy and longer computation times.\n",
      "    \n",
      "    Returns\n",
      "    -------\n",
      "    min_sample_size : float\n",
      "        The smallest sample size for which a Bayesian test would detect a difference with at\n",
      "        least min_confidence Bayesian confidence at least 80% of the times, assuming that \n",
      "        true_positive_rates are the true positive rates.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(bayes.computeBayesTestSampleSize)"
   ]
  }
 ],
 "metadata": {
  "creator": "musmecn",
  "customFields": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "tags": []
 },
 "nbformat": 4,
 "nbformat_minor": 1
}