Website A/B Testing - Lab

Introduction

In this lab, you'll get another chance to practice your skills at conducting a full A/B test analysis. It will also be a chance to practice your data exploration and processing skills! The scenario you'll be investigating is data collected from the homepage of a music app page for audacity.

Objectives

You will be able to:

• Analyze the data from a website A/B test to draw relevant conclusions
• Explore and analyze web action data

Exploratory Analysis

Start by loading in the dataset stored in the file 'homepage_actions.csv'. Then conduct an exploratory analysis to get familiar with the data.

Hints: * Start investigating the id column: * How many viewers also clicked? * Are there any anomalies with the data; did anyone click who didn't view? * Is there any overlap between the control and experiment groups? * If so, how do you plan to account for this in your experimental design?

#Your code here

Conduct a Statistical Test

Conduct a statistical test to determine whether the experimental homepage was more effective than that of the control group.

#Your code here

Verifying Results

One sensible formulation of the data to answer the hypothesis test above would be to create a binary variable representing each individual in the experiment and control group. This binary variable would represent whether or not that individual clicked on the homepage; 1 for they did and 0 if they did not.

The variance for the number of successes in a sample of a binomial variable with n observations is given by:

$n\bullet p (1-p)$

Given this, perform 3 steps to verify the results of your statistical test:

1. Calculate the expected number of clicks for the experiment group, if it had the same click-through rate as that of the control group.
2. Calculate the number of standard deviations that the actual number of clicks was from this estimate.
3. Finally, calculate a p-value using the normal distribution based on this z-score.

Step 1:

Calculate the expected number of clicks for the experiment group, if it had the same click-through rate as that of the control group.

#Your code here

Step 2:

Calculate the number of standard deviations that the actual number of clicks was from this estimate.

#Your code here

Step 3:

Finally, calculate a p-value using the normal distribution based on this z-score.

#Your code here

Analysis:

Does this result roughly match that of the previous statistical test?

Comment: Your analysis here

Summary

In this lab, you continued to get more practice designing and conducting AB tests. This required additional work preprocessing and formulating the initial problem in a suitable manner. Additionally, you also saw how to verify results, strengthening your knowledge of binomial variables, and reviewing initial statistical concepts of the central limit theorem, standard deviation, z-scores, and their accompanying p-values.