Hypothesis Testing & Experimentation Primer (Notebook)

This repo contains a single Jupyter notebook:

randomized_experiments_playbook.ipynb — a practical, code-first primer on hypothesis testing and experiment design, including:

• fixed-horizon confidence intervals and sample-size planning (means and proportions)
• one-sample, two-sample, A/B testing, and paired (matched) tests
• sequential testing (always-valid / mSPRT-style ideas) and a short discussion of multi-arm bandits

The notebook includes reusable helper functions for sample-size computation (margin-of-error and minimum-detectable-effect). References for the material and further reading are also provided.

Keywords: A/B testing, experiment design, hypothesis testing, sample size, sequential testing (mSPRT), multi-armed bandits

Read-only HTML (no setup required)

A rendered HTML with all cells executed and outputs included is available at:

• Playbook (web)

If you only want to read, use the link above. To run or modify the code, download the .ipynb notebook and follow the setup instructions below.

Requirements

• Python 3.10+ recommended.
• The dependencies (including JupyterLab) are listed in requirements.txt.

Setup

macOS / Linux (bash/zsh)

python -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
pip install -r requirements.txt

jupyter lab

On some Linux systems, use python3 -m venv .venv instead of python -m venv .venv.

Windows (PowerShell)

python -m venv .venv
.\.venv\Scripts\Activate.ps1

python -m pip install --upgrade pip
pip install -r requirements.txt

jupyter lab

Windows (Command Prompt)

python -m venv .venv
.venv\Scripts\activate.bat

python -m pip install --upgrade pip
pip install -r requirements.txt

jupyter lab

Disclaimer & Terms of Use

Disclaimer

This repository is provided for educational purposes only. It is not statistical, legal, medical, or financial advice, and it does not substitute for review by a qualified professional. Any decisions you make based on this material (e.g., experiment design, rollout decisions, business actions) are your responsibility.

The code and calculations are provided “as is”, without warranty of any kind. Results may be sensitive to assumptions (e.g., independence, distributional approximations, variance stability, instrumentation quality, multiple testing, and sequential monitoring). You are responsible for validating assumptions, testing the code in your environment, and ensuring correctness for your use case.

See the LICENSE file for full license terms.

Data & Privacy

Do not commit confidential data, user identifiers, or proprietary metrics to this repository. If you run experiments on real users, you are responsible for complying with applicable privacy/security policies and laws.

Credit (optional)

Attribution is not required, but appreciated: Keyvan Rahmani, Randomized Experiments Playbook (2026).