08-Inference.Rmd

# Statistical inference {#statistical-inference}

```{r setup9, include=FALSE}
knitr::opts_chunk$set(echo = FALSE,
                      prompt = FALSE,
                      tidy = TRUE,
                      collapse = TRUE)
library("tidyverse")
```

In a previous chapter, we learned about
[estimation](#estimation): the use of data and statistics
to construct the best possible guess at the value of some parameter.  

In this chapter, we will pursue a different goal.  Instead of estimating
the single "most likely" value of the parameter, we will construct
statistics that can be used to classify particular parameter values as
plausible (could be the true value) or implausible (unlikely to be the 
true value).

- A hypothesis test determines whether a *particular* value *can* be 
  ruled out.
- A confidence interval determines a *range* of values that *cannot*
  be ruled out.

The set of procedures for constructing confidence intervals and 
hypothesis tests is called statistical ***inference***.

::: {.goals data-latex=""}
**Chapter goals**

In this chapter we will learn how to:

 - Construct and perform a simple hypothesis test
 - Construct a confidence interval
 - Correctly interpret both hypothesis tests and confidence intervals
:::

## Principles of inference

### Evidence

The purpose of statistical inference is to systematically account for the
uncertainty associated with limited evidence.  That is, there are important
aspects of the data generating process we do not know. The data provide
some evidence about those unknown aspects, but the evidence they provide
may not be strong.  Statistical inference asks us what statements about 
the data generating process can be made with confidence based on the data.

::: example
**Fair and unfair roulette games**

Suppose you work as a casino regulator for the BCLC (British Columbia
Lottery Corporation, the crown corporation that regulates all commercial
gambling in B.C.).  You have been given data with recent roulette 
results from a particular casino and are tasked with determining whether
the casino is running a fair game.

Before getting caught up in math, let's think about how we might assess 
evidence: 

- We might have results from many games, or only a few games.  
- Our results may have a win rate close to the expected rate for a fair
  game, or far from that rate.

We can put those possibilities into a table:

| Win rate                |  Many games     |    Few games            |
|:------------------------|:---------------:|:-----------------------:|
| Close to fair game rate | Probably fair   | Could be fair or unfair |
| Far from fair game rate | Probably unfair | Could be fair or unfair |

That is, we can make a fairly confident conclusion if we have a lot
of evidence, and our conclusion depends on what the evidence shows. But
if we do not have a lot of evidence, we cannot make a confident conclusion
either way.

In this chapter we will formalize these basic ideas about evidence.
:::

### A basic framework

For the remainder of this chapter, suppose we have a data set 
$D_n = (x_1,x_2,\ldots,x_n)$ of size $n$.  The data comes from an
unknown data generating process that includes an unknown 
parameter of interest $\theta$. 

::: example
**DGP and parameter of interest for roulette**

Let $D_n = (x_1,\ldots,x_n)$ be a data set of results from 
$n = 100$ games of roulette at a local casino. More specifically,
let 
  $$x_i = I(\textrm{Red wins})$$
Our parameter of interest is the probability that
red wins:
  $$p_{red} = \Pr(x_i = 1) = E(x_i)$$
We know that red wins in a fair game with probability
$p_{red} = 18/37 \approx 0.486$. 
:::

## Hypothesis tests

We will start with hypothesis tests. The idea of a hypothesis test
is to determine whether the data rule out or ***reject*** a specific
value of the unknown parameter $\theta$.

Intuitively, if we have no (useful) data we cannot rule anything
out, but as we obtain more data, we can rule out more values.

### The null and alternative hypotheses

The first step in a hypothesis test is to define the 
***null hypothesis***.  The null hypothesis is a statement
about our parameter $\theta$ that takes the form:
  $$H_0: \theta = \theta_0$$
where $\theta_0$ is a specific number. This is the value of 
$\theta$ we are interested in ruling out.

The next step is to define the ***alternative hypothesis***. The 
alternative hypothesis defines every other value of $\theta$ 
we are allowing, and is usually written as:
  $$H_1: \theta \neq \theta_0$$
where $\theta_0$ is the same number as used in the null.

::: example
**Null and alternative for $p_{red}$**

In our roulette example, our null hypothesis is
that the game is fair:
  $$H_0: p_{red} = 18/37 \approx 0.486$$
and the alternative hypothesis is that it is not fair:
  $$H_1: p_{red} \neq 18/37$$
:::

Notice that there is something of an asymmetry between the null and 
alternative hypothesis: the null is typically (though not necessarily)
a single value and the alternative is every other possible value. 

::: {.fyi data-latex=""}
**What null hypothesis to choose?**

Our framework here assumes that you already know what null hypothesis you
wish to test, but we might briefly consider how we might choose a null
hypothesis to test.

In some applications there are null hypotheses that are of clear interest
for that specific case:

- In our roulette example, the natural null to test is whether the 
  win probability matches that of a fair game ($p = p_{fair}$).
- When measuring the effect $\beta$ of one variable on another, the natural
  null to test is "no effect at all" ($\beta = 0$).
- When comparing the mean of some characteristic or outcome across two
  groups (for example, average wages of men and women), the natural null
  to test is that they are the same ($\mu_m = \mu_W$)
- In epidemiology, a contagious disease will tend to spread if its
  reproduction rate $R$ is greater than one, and decline if it
    is less than one, so $R = 1$ is a natural null to test.

If there is no obvious null hypothesis, it may make sense to test many 
null hypotheses and report all of the results.  There is nothing 
wrong with doing that.
:::

### The test statistic

Our next step is to construct a ***test statistic*** that can be 
calculated from our data. A valid test statistic for a given null 
hypothesis is a statistic $t_n$ that has the following two 
properties:

1. The probability distribution of $t_n$ ***under the null*** 
   (i.e., when $H_0$ is true) is *known*.
2. The probability distribution of $t_n$ ***under the alternative***
   (i.e., when $H_1$ is true) is *different* from its
   probability distribution under the null.

The test statistic is usually based on an estimator of the parameter, 
and is usually constructed so that it is typically close to zero 
when the null is true, and far from zero when the null is false. But
it does not need to be.

::: example
***A test statistic for roulette***

A natural test statistic for the win probability of a bet on red 
would be the corresponding win frequency in our data.  We could
use either the relative win frequency (which also happens to 
be the sample average):
  $$\hat{f}_{red} = \bar{x}_n = \frac{1}{n} \sum_{i=1}^n x_i$$
but it will be more convenient to use the absolute win frequency:
  $$t_n = n\hat{f}_{red} = n\bar{x}_n =\sum_{i=1}^n x_i$$
Next we need to find the probability distribution of $t_n$ 
under the null, and under the alternative.

In general, since $x_i \sim Bernoulli(p_{red})$ we have:
  $$t_n \sim Binomial(n,p_{red})$$
Remember that the $Binomial(n,p)$ distribution is the distribution corresponding
to the number of times an event with probability $p$ happens in $n$ independent
trials.

Under the null (when $H_0$ is true), $p_{red} = 18/37$ and so:
  $$t_n \sim Binomial(100,18/37)$$
Since this distribution does not involve any unknown parameters,
our test statistic satisfies the requirement of having a known
distribution under the null.

Under the alternative (when $H_1$ is true), $p_{red}$ can take on any value
*other* than $18/37$.  The sample size is still $n=100$, so the
distribution of the test statistic is:
  $$t_n \sim Binomial(100,p_{red}) \textrm{ where $p_{red} \neq 18/37$ }$$
Notice that the distribution of our test statistic under the alternative
is not known, since $p_{red}$ is not known.  But the distribution is 
*different* under the alternative, and that is what we require from
our test statistic.
:::

### Significance and critical values

After choosing a test statistic $t_n$, the next step is to choose 
***critical values***.  The critical values are two numbers
$c_L$ and $c_H$ (where $c_L < c_H$) such that 

- $t_n$ has a *high* probability of being between $c_L$ and $c_H$ when
  the null is true.
- $t_n$ has a *lower* probability of being between $c_L$ and $c_H$ when
  the alternative is true.
  
The range of values from $c_L$ to $c_H$ is called the ***critical range***
of our test.

Given the test statistic and critical values:

- We ***reject the null*** if $t_n$ is outside of  the critical range.
  - This means we have strong evidence that $H_0$ is false.
  - The reason we reject here is that we know we would be unlikely to
    observe such a value of $t_n$ if $H_0$ were true.
- We ***fail to reject the null*** if $t_n$ is inside of the critical
  range.
  - This means we do not have strong evidence that $H_0$ is false.
  - This does not mean we have strong evidence that $H_0$ is true. We 
    may just not have enough evidence to reach a conclusion.

Notice that there is an asymmetry here: in the absence of evidence,
we will not reject *any* null hypotheses.

![Critical values and rejecting the null hypothesis](bin/reject.png)

How do we choose critical values? You can think of critical values as setting
a standard of evidence, so we need to balance two considerations:

- The probability of rejecting a false null is called the ***power***
  of the test.
  - We want our test to reject the null when it is false, so power is good.
- The probability of rejecting a true null is called the ***size***
  or ***significance*** of a test.
  - We do not want our test to reject the null when it is true, so size 
    is bad.
- There is always a trade off between power and size 
  - A *narrower* critical range (higher $c_L$ or lower $c_H$) will increase 
    the rejection rate, increasing both power (good) and size (bad).
  - A *wider* critical range (lower $c_L$ or higher $c_H$) will reduce 
    the rejection rate, reducing both power (bad) and size (good).
  
Given this trade off between power and size, we might construct some
criterion that includes both (just like MSE includes both variance and
bias) and choose critical values to maximize that criterion.  In practice, 
we do not typically do that.  

Instead, we follow a simple convention:

1. Set the size to a fixed value $\alpha$.
   - In general, the conventional size varies by field, and typically
     varies with how much data is typical in that field.
   - In economics and most other social sciences, the usual convention 
     is to use a size of 5\% ($\alpha = 0.05$).
   - We sometimes see 1\% ($\alpha = 0.01$) when working with larger data sets 
     or 10\% ($\alpha = 0.10$) when working with small data sets.
   - In physics or genetics, where data sets are much larger, the 
     conventional size is much lower. 
2. Calculate critical values that imply the desired size.  
   - With a size of 5\% $(\alpha = 0.05)$, we would:
     - set $c_L$ to the 2.5 percentile (0.025 quantile) of the null distribution
     - set $c_H$ to the 97.5 percentile (0.975 quantile) of the null distribution
   - With a size of 10\% $(\alpha = 0.10)$, we would:
     - set $c_L$ to the 5 percentile (0.05 quantile) of the null distribution
     - set $c_H$ to the 95 percentile (0.95 quantile) of the null distribution
   - With a size of $\alpha$, we would:
     - set $c_L$ to the $\alpha/2$ quantile of the null distribution
     - set $c_H$ to the $1-\alpha/2$ quantile of the null distribution

In other words, we set size equal to a conventional value, and let the 
power be whatever is implied by that.

::: example
**Critical values for roulette**

We earlier showed that the distribution of $t_n$ under the null is:
  $$t_n \sim Binomial(100,18/37)$$
We can get a size of 5\% by choosing:
  $$c_L = 2.5 \textrm{ percentile of } Binomial(100,18/37)$$
  $$c_H = 97.5 \textrm{ percentile of } Binomial(100,18/37)$$
We can then use Excel or R to calculate these critical values. In Excel,
the function you would use is `BINOM.INV()`

- The formula to calculate $c_L$ is `=BINOM.INV(100,18/37,0.025)`
- The formula to calculate $c_H$ is `=BINOM.INV(100,18/37,0.975)`

The calculations below were done in R:
```{r BinomialCriticalValues}
cat("2.5 percentile of binomial(100,18/37) =",
    qbinom(0.025,100,18/37),
    "\n")
cat("97.5 percentile of binomial(100,18/37) =",
    qbinom(0.975,100,18/37),
    "\n")
```
In other words we reject the null (at 5\% significance) that 
the roulette wheel is fair if red wins fewer than 39 games 
or more than 58 games.
:::



::: {.fyi data-latex=""}
**A general test for a single probability**

We can generalize the test we have constructed so far to the case
of the probability of any event:

| Test component         |  Roulette example                    |General case               |
|------------------------|--------------------------------------|---------------------------|
| Parameter              | $p_{red} = \Pr(\textrm{Red wins})$   | $p = \Pr(\textrm{event})$ |
| Null hypothesis        | $H_0:p_{red} = 18/37$                | $H_0:p = p_0$             | 
| Alternative hypothesis | $H_1: p_{red} \neq 18/37$            |$H_1: p \neq p_0$          | 
| Test statistic         | $t = n\hat{f}_{RED}$           | $t = n\hat{f}_{\textrm{event}}$ | 
| Null distribution      | $Binomial(100,18/37)$                |$Binomial(n,p_0)$          | 
| Critical value $c_L$   | 39                        | 2.5 percentile of $Binomial(n,p_0)$  |
| Critical value $c_H$   | 58                        | 97.5 percentile of $Binomial(n,p_0)$ |
| Decision               | Reject if $t \notin [39,58]$    | Reject if $t \notin [c_L,c_H]$ |
:::

### The power of a test

As mentioned above, the power of a test is defined as the probability 
of rejecting the null when it is false, and the alternative is true.

The size of a test is a number, since the distribution of the test statistic
is known under the null.  Since the alternative typically allows
more than one value of the parameter $\theta$, the power
of a test is not a number but a *function* of the unknown true value
of $\theta$ (and sometimes other unknown features of the DGP):
  $$power(\theta) = \Pr(\textrm{reject $H_0$})$$
In some cases we can actually calculate this function.

::: example
**The power curve for roulette**

Power curves can be tricky to calculate, and I will not ask you to calculate
them for this course.  But they can be calculated, and it is useful to
see what they look like.

Figure \@ref(fig:PowerCurves) below depicts the power curve for the roulette test we have
just constructed; that is, we are testing the null that $p_{red} = 18/37$
at a 5\% size.  The blue line depicts the power curve for $n=100$ 
as in our example, while the green line depicts the power curve for
$n=20$.

```{r PowerCurves, fig.cap = "*Power curves for the roulette example*"}
PowerCurveData <- tibble(theta = seq(0,1,length.out=380),
                         power20 = pbinom(qbinom(0.025,
                                                 20,
                                                 18/37),
                                          20,
                                          theta) +
                             (1 - pbinom(qbinom(0.975,
                                              20,
                                              18/37),
                                       20,
                                       theta)),
                         power100 = pbinom(qbinom(0.025,
                                                  100,
                                                  18/37),
                                           100,
                                           theta) +
                           (1 - pbinom(qbinom(0.975,
                                              100,
                                              18/37),
                                       100,
                                       theta)))
ggplot(data = PowerCurveData, 
       mapping = aes(x = theta,y = power20)) +
  geom_line(col="green") +
  geom_line(aes(y = power100), col="blue") +
  xlab("true probability of winning") +
  ylab("power") +
  geom_text(label="n=100",x=0.7,y=0.8,col="blue") +
  geom_text(label="n=20",x=0.9,y=0.8,col="green") +
  geom_hline(yintercept = 0.05,col="gray") +
  geom_vline(xintercept = 18/37,col="gray") +
  labs(title = "Power curve for fair roulette wheel", 
       subtitle = "", 
       caption = "H0: Pr(red wins) = 18/37, significance = 0.05", 
       tag = "")
```

There are a few features I would like you to notice, all of which
are common to most regularly used tests:

- The power curve reaches its lowest value at the red point 
  $(18/37,0.05)$.  Note that $18/37$ is the parameter value
  under the null, and $0.05$ is the size of the test. In other words:
  - The power is always at least as big as the size, and is usually bigger.
  - We are more likely to reject the null when it is false than when it 
    is true. That's good!
  - When a test has this desirable property, we call 
    it an ***unbiased*** test. 
- The power increases as $\theta$ gets further from the null.  
  - That is, we are more likely to detect unfairness in a game 
    that is *very* unfair than when in one that is *a little* unfair.
- Power also increases with the sample size; the blue line
  ($n = 100$) is above the green line ($n = 20$).
  
Power analysis is often used by researchers to determine how much data 
to collect.  Each additional observation increases power but costs money,
so it is important to spend enough to get clear results but not
much more than that.
:::

::: {.fyi data-latex=""}
**P values**

The convention of always using a 5\% significance level for 
hypothesis tests is somewhat arbitrary and has some negative 
unintended consequences:

  1. Sometimes a test statistic falls just below or just above the
    critical value, and small changes in the analysis can change 
    a result from reject to cannot-reject.
  2. In many fields, unsophisticated researchers and journal 
    editors misinterpret "cannot reject the null" as "the null is true."
    
One common response to these issues is to report what is called
the ***p-value*** of a test.  The p-value of a test is defined
as the significance level at which one would switch from rejecting
to not-rejecting the null.  For example:

- If the p-value is 0.43 (43\%) we would not reject the null at 10\%,
  5\% or 1\%.
- If the p-value is 0.06 (6\%) we would reject the null at 10\% but not
  at 5\% or 1\%.
- If the p-value is 0.02 (2\%) we would reject the null at 10\% and 5\%
  but not at 1\%.
- If the p-value is 0.001 (0.1\%) we would reject the null at 10\%, 5\%,
  and 1\%.
  
The p-value of a test is simple to calculate from the test statistic and 
its distribution under the null.  I won't go through that calculation 
here.
:::

## The central limit theorem

In order for a test statistic to work, its exact probability distribution must be known
under the null hypothesis.  The example test in the previous section worked because
it was based on a sample frequency, a statistic whose probability distribution is 
relatively easy to calculate.  Unfortunately, most statistics do not have a probability 
distribution that is easy to calculate.

Fortunately, we have a very powerful asymptotic result called the 
***Central Limit Theorem (CLT)***.  The CLT roughly says that we can approximate 
the entire probability distribution of the sample average $\bar{x}_n$ by a normal distribution
if the sample size is sufficiently large.

As with the LLN, we need to invest in some terminology before we can
state the CLT.  Let $s_n$ be a statistic calculated from $D_n$ and 
let $F_n(\cdot)$ be its CDF. We say that $s_n$ 
***converges in distribution*** to a random variable $s$ with CDF $F(\cdot)$, or 
  $$s_n \rightarrow^D s$$
if
  $$\lim_{n \rightarrow \infty} |F_n(a) - F(a)| = 0$$
for every $a \in \mathbb{R}$.

Convergence in distribution means we can approximate the actual CDF
$F_n(\cdot)$ of $s_n$ with its limit $F(\cdot)$.  As with most approximations,
this is useful whenever $F_n(\cdot)$ is difficult to calculate and $F(\cdot)$
is easy to calculate. 

::: {.fyi data-latex=""}
**CENTRAL LIMIT THEOREM**: Let $\bar{x}_n$ be the sample average 
from a random sample of size $n$ on the random variable $x_i$ with
mean $E(x_i) = \mu_x$ and variance $var(x_i) = \sigma_x^2$. Then
  $$z_n \rightarrow^D z \sim N(0,1)$$
:::

What does the central limit theorem mean?

  - Fundamentally, it means that if $n$ is big enough then the probability
    distribution of $\bar{x}_n$ is approximately normal 
    *no matter what the original distribution of $x_i$ looks like.*
  - In order for the CLT to apply, we need to rescale $\bar{x}_n$
    so that it has zero mean (by subtracting $E(\bar{x}_n) = \mu_x$) 
    and constant variance as $n$ increases (by dividing by 
    $sd(\bar{x}_n) = \sigma_x/\sqrt{n}$)).  That rescaled sample
    average is $z_n$.  
  - In practice, we don't usually know $\mu_x$ or $\sigma_x$ so 
    we can't calculate $z_n$ from data. Fortunately, there are some
    tricks for getting around this problem that we will talk
    about later.

What about statistics other than the sample average?  Well it turns out that 
Slutsky's theorem also extends to convergence in distribution.

::: {.fyi data-latex=""}
**SLUTSKY THEOREM**: Let $g(\cdot)$ be a continuous function. Then:
  $$s_n \rightarrow^D s \implies g(s_n) \rightarrow^D g(s)$$
:::

The implication here is that *nearly all* statistics have a sampling
distribution that can be approximated using the normal distribution if
the sample size is large enough.

## Inference on the mean

Having described the general framework and a single example, we now
move on to the most common application: constructing hypothesis
tests and confidence intervals on the mean in a random sample.

Let $D = (x_1,\ldots,x_n)$ be a random sample of size $n$
on some random variable $x_i$ with unknown mean $E(x_i) = \mu_x$
and variance $var(x_i) = \sigma_x^2$.  

Let the sample average be:
  $$\bar{x}_n = \frac{1}{n} \sum_{i=1}^n x_i$$
let the sample variance be:
  $$s_x^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2$$
and let the sample standard deviation be:
  $$s_x = \sqrt{s_x^2}$$
These statistics are easily calculated from the data,
and we have previously discussed their properties in detail.

### The null and alternative hypotheses

Suppose that you want to test the null hypothesis:
  $$H_0: \mu_x = 1$$
against the alternative hypothesis 
  $$H_1: \mu_x \neq 1$$
Having stated our null and alternative hypotheses, we need to
construct a test statistic.

Remember that our test statistic needs to have a *known* distribution
under the null, and a *different* distribution under the alternative.

### The T statistic

The typical test statistic we use in this setting is called the 
***T statistic***, and takes the form:
  $$t_n = \frac{\bar{x}_n - 1}{s_x/\sqrt{n}}$$
The idea here is that we take our estimate of the parameter ($\bar{x}$),
subtract its expected value under the null ($1$), and divide
by an estimate of its standard deviation ($s_x/\sqrt{n}$). We can
add and subtract the unknown true mean $\mu_x$ to get: 
  \begin{align}
    t_n &= \frac{\bar{x}_n - \mu_x + \mu_x - 1}{s_x/\sqrt{n}} \\
      &= \frac{\bar{x}_n - \mu_x}{s_x/\sqrt{n}} + \frac{\mu_x - 1}{s_x/\sqrt{n}} 
  \end{align}
The first part of this expression is a random variable with a mean of zero
and a variance of (about) one.  The second part of the expression is
exactly zero when $H_0$ is true, and not exactly zero when it is false. 

Recall that we need the probability distribution of $t_n$ to be known when
$H_0$ is true, and different when it is false.  The second criterion is
clearly met, and the first criterion is met if we can find the 
probability distribution of $\frac{\bar{x}_n - \mu_x}{s_x/\sqrt{n}}$.

Unfortunately, if we don't know the exact probability distribution
of $x_i$ we don't know the exact probability distribution of statistics
calculated from it.  Once we have a potential test statistic, there 
are two standard solutions to this problem:
  
  1. Assume a specific probability distribution (usually a normal
    distribution) for $x_i$.  We can (or at least a 
    professional statistician can) then mathematically derive 
    the distribution of any test statistic from this distribution.
  2. Use the central limit theorem to get an approximate probability
    distribution.

We will explore both of those options.

### Asymptotic critical values

We will start with the asymptotic solution to the problem. As we learned in 
Chapter \@ref(statistics), the Central Limit Theorem tells us that:
  $$\frac{\bar{x}_n - \mu_x}{\sigma_x/\sqrt{n}} \rightarrow N(0,1)$$
Under the null our test statistic looks just like this, but with the sample
standard deviation $s_x$ in place of the population standard deviation 
$\sigma_x$.  It turns out that Slutsky's theorem allows us to make
this substitution, and it can be proved that:
  $$\frac{\bar{x}_n - \mu_x}{s_x/\sqrt{n}} \rightarrow N(0,1)$$

Therefore, under the null:
  $$t_n \rightarrow N(0,1)$$
In other words, while we do not know the exact (finite-sample) distribution
of $t_n$ we know that $N(0,1)$ provides a useful asymptotic approximation 
to that distribution.

```{r NormalDistribution, fig.cap = "*Asymptotic distribution of t_n under the null*"}
simdata <- tibble(x = seq(-3,3,length.out = 100),
                  tinf = dnorm(x))
ggplot(data=simdata,
       mapping = aes(x= x,
                     y= tinf)) +
  geom_line() +
  xlab("value") +
  ylab("PDF of t_n") +
  labs(title = "Asymptotic distribution of t_n under null", 
       subtitle = "", 
       caption = "", 
       tag = "")
```

Therefore, if we want a test that has the ***asymptotic size*** of 5\%, we
can use Excel or R to calculate critical values. In Excel, the function
would be `NORM.INV` or `NORM.S.INV`, and the formulas would be:

- $c_L$: `=NORM.S.INV(0.025)` or `=NORM.INV(0.025,0,1)`.
- $c_H$: `=NORM.S.INV(0.975)` or `=NORM.INV(0.975,0,1)`.

The calculations below were done in R:
```{r AsymptoticCriticalValues}
  cat("cL = 2.5 percentile of N(0,1) = ",
      round(qnorm(0.025),3),
      "\n")
  cat("cH = 97.5 percentile of N(0,1) = ",
      round(qnorm(0.975),3),
      "\n")
```
These particular critical values are so commonly used that I want you
to remember them.

### Exact critical values 

Most economic data comes in sufficiently large samples that the asymptotic
distribution of $t_n$ is a reasonable approximation and the asymptotic
test works well. But occasionally we have samples that are small enough
that it doesn't.  

Another option is to assume that the $x_i$ variables are normally
distributed:
  $$x_i \sim N(\mu_x,\sigma_x^2)$$
where $\mu_x$ and $\sigma_x^2$ are unknown parameters.

Now at this point it is important to remind you: many interesting 
variables are *not* normally distributed (for example, our roulette outcome
is discrete uniform, and the result of a given bet is Bernoulli)
and so this assumption may very well be incorrect.

If this was a more advanced course I would derive the 
distribution of $t_n$ under the null.  But for ECON 233, I will
just ask you to understand that it *can* be derived once we 
assume normality of the $x_i$.

The null distribution of this particular test statistic under these particular assumptions was derived in the 1920's by William Sealy Gosset, a statistician 
working at the Guiness brewery.  To avoid getting in trouble at work (Guinness 
did not want to give away trade secrets) Gosset published under the pseudonym 
"Student".  As a result, the family of distributions he derived is called
"Student's T distribution".

When the null is true, the test statistic $t_n$ as described above
has Student's T distribution with $n-1$ degrees of freedom:
  $$t_n \sim T_{n-1}$$
As always, it has a different distribution (sometimes called the "noncentral
T distribution") when the null is false.

The $T_{n-1}$ distribution looks a lot like the $N(0,1)$
distribution, but has slightly higher probability of extreme
positive or negative values.  As $n$ increases the 
$T_{n-1}$ distribution converges to the 
$N(0,1)$ distribution, just as predicted by the central 
limit theorem.

```{r TDistribution, fig.cap = "*Exact distribution of t_n under the null*"}
simdata <- tibble(x = seq(-3,3,length.out = 100),
                  t5 = dt(x,df=4),
                  t10 = dt(x,df=9),
                  t30 = dt(x,df=29),
                  tinf = dnorm(x))
ggplot(data = simdata,
       mapping = aes(x = x)) +
  geom_line(aes(y = t5), col = "green") +
  geom_line(aes(y = t10), col = "blue") +
  geom_line(aes(y = t30), col = "red") +
  geom_line(aes(y = tinf), col = "black") +
  geom_text(label="n = infinity",x=0.75,y=0.4,col="black") +
  geom_text(label="n = 5",x=2.5,y=0.1,col="green") +
  geom_text(label="n = 10",x=1.7,y=0.2,col="blue") +
  geom_text(label="n = 30",x=1.2,y=0.3,col="red") +
  xlab("value") +
  ylab("PDF") +
  labs(title = "Exact distribution of t_n under null", 
       subtitle = "", 
       caption = "(x assumed to be normally distributed)", 
       tag = "")
```

Having found our test statistic and its distribution under the null, 
we can calculate our critical values:
  $$c_L = 2.5 \textrm{ percentile of } T_{n-1}$$
  $$c_H = 97.5 \textrm{ percentile of } T_{n-1}$$
We can obtain these percentiles using Excel or R. In Excel, the
relevant function is `T.INV`.  For example, if we have $n = 5$ observations, then:

- We would calculate $c_L$ by the formula `=T.INV(0.025,5-1)`.
- We would calculate $c_H$ by the formula `=T.INV(0.975,5-1)`.

The results (calculated below using R) would be:
```{r T4CriticalValues}
  cat("cL = 2.5 percentile of T_4 = ",
      round(qt(0.025,df=4),3),
      "\n")
  cat("cH = 97.5 percentile of T_4 = ",
      round(qt(0.975,df=4),3),
      "\n")
```
In contrast, if we have 30 observations, then:

- We would calculate $c_L$ by the formula `=T.INV(0.025,30-1)`.
- We would calculate $c_H$ by the formula `=T.INV(0.975,30-1)`.

The results (calculated below using R) would be:
```{r T29CriticalValues}
  cat("cL = 2.5 percentile of T_29 = ",
      round(qt(0.025,df=29),3),
      "\n")
  cat("cH = 97.5 percentile of T_29 = ",
      round(qt(0.975,df=29),3),
      "\n")
```
and if we have 1,000 observations:

- We would calculate $c_L$ by the formula `=T.INV(0.025,1000-1)`.
- We would calculate $c_H$ by the formula `=T.INV(0.975,1000-1)`.

The results (calculated below using R) would be:
```{r T999CriticalValues}
  cat("cL = 2.5 percentile of T_999 = ",
      round(qt(0.025,df=999),3),
      "\n")
  cat("cH = 97.5 percentile of T_999 = ",
      round(qt(0.975,df=999),3),
      "\n")
```


### Which test to use? 

Statisticians often call the finite-sample test the ***T test*** and the 
asymptotic test the ***Z test***, as a result of the notation typically
used to represent the test statistic.  The two tests have the same
underlying test statistic, but different critical values.

Which test should we use in practice?  The T test is a more ***conservative***
test than the Z test, meaning it has larger critical values and is less
likely to reject the null. As a result it has lower power and lower size. But 
at some point (around $n = 30$) the difference 
between the two tests becomes too small to make much of a difference.

As a result, statisticians typically recommend using the T test for smaller samples
(less than 30 or so), and then using whichever test is more convenient
with larger samples.  Most data sets in economics have well over 30 observations so
economists tend to use asymptotic tests unless they have a very small 
sample.


## Confidence intervals 

Hypothesis tests have one very important limitation: although they 
allow us to rule out $\theta = \theta_0$ for a single value of $\theta_0$, 
they say nothing about other values very close to $\theta_0$.

For example, suppose you are a medical researcher trying to measure the
effect of a particular treatment, let $\theta$ be the treatment, and 
suppose that you have tested the null hypothesis that the treatment
has no effect ($\theta = 0$).

- If you reject this null, you have concluded that the effect has *some*
  effect. However, that does not rule out the possibility that
  the effect of the treatment is *very small*.
- If you fail to reject this null, you cannot rule out the possibility
  that the treatment has *no* effect.  However, this does not rule out
  the possibility that the effect is *very large*.

The solution to this would be to do a hypothesis test for every possible
value of $\theta$, and classify them into values that were rejected 
and not rejected.  This is the idea of a ***confidence interval***.

A confidence interval for the parameter $\theta$ with coverage 
rate $CP$ is an interval with lower bound $CI_L$ and 
upper bound $CI_H$ constructed from the data in such a way that
  $$\Pr(CI_L < \theta < CI_H) = CP$$
In economics and most other social sciences, the convention is to report
confidence intervals with a coverage rate of 95\%. 
  $$\Pr(CI_L < \theta < CI_H) = 0.95$$
Note that $\theta$ is a fixed (but unknown) parameter, while $CI_L$ 
and $CI_H$ are statistics calculated from the data.  

How do we calculate confidence intervals? It turns out to be 
entirely straightforward: confidence intervals can be constructed
by inverting hypothesis tests: 

- The 95\% confidence interval is all values that cannot be 
  rejected at a 5\% level of significance.
- The 90\% confidence interval all values that cannot be 
  rejected at a 10\% level of significance. 
  - It is *narrower* than the 95\% confidence interval.
- The 99\% confidence interval is all values that cannot be 
  rejected at a 1\% level of significance. 
  - It is *wider* than the 95\% confidence interval.

::: example
**A confidence interval for the win probability**

Calculating a confidence interval for $p_{red}$ is somewhat 
tricky to do by hand, but easy to do on a computer:

1. Construct a grid of many values between 0 and 1.
2. For each value $p_0$ in the grid, test the null hypothesis
   $H_0: p_{red} = p_0$ against the alternative hypothesis
   $H_1: p_{red} \neq p_0$.
3. The confidence interval is the range of values for $p_0$
   that are not rejected.

For example, suppose that red wins on 40 of the 100 games. Then 
a 95\% confidence interval for $p_{red}$ is:
```{r RouletteCI}
theta <- seq(0,1,length.out=101)
cL <- qbinom(0.025,100,theta)
cH <- qbinom(0.975,100,theta)
thetaCI <- range(theta[cL < 40 & cH > 40])
cat(thetaCI[1]," to ",thetaCI[2],"\n")
```
Notice that the confidence interval includes the fair value of $0.486$
but it also includes some very unfair values.  In other words, while
we are unable to rule out the possibility that we have a fair game,
the evidence that we have a fair game is not very strong.
:::

### Confidence intervals for the mean

Confidence intervals for the mean are very easy to calculate. Again
we construct them by inverting the hypothesis test.

Pick any $\mu_0$.  To test the null 
  $$H_0: \mu_x = \mu_0$$
our test statistic is:
  $$t_n = \sqrt{n}\frac{\bar{x}-\mu_0}{s_x}$$
and we fail to reject the null if
  $$c_L < t_n < c_H$$
where $c_L$ and $c_H$ are our critical values.

Plugging $t_n$ to this expression we fail to reject the null whenever:
  $$c_L < \sqrt{n}\frac{\bar{x}-\mu_0}{s_x} < c_H$$
Solving for $\mu_0$ we fail to reject whenever:
  $$\bar{x} - c_H s_x/\sqrt{n} < \mu_0 < \bar{x} - c_L s_x/\sqrt{n}$$
All that remains is to choose a confidence/size level, and 
decide whether to use an asymptotic or finite sample test.

If we are using the asymptotic approximation to construct
a 95\% confidence interval, then the 5\% asymptotic critical values
are $c_L = -1.96$ and $c_H \approx 1.96$ and the 
confidence interval is:
  $$CI = \bar{x} \pm 1.96 s_x/\sqrt{n}$$
In other words, the 95\% confidence interval for $\mu_x$ 
is just the point estimate plus or minus roughly 2 standard
errors.  

If we have a small sample, and choose to assume normality rather
than using the asymptotic approximation, then we need to use the
slightly larger critical values from the $T_{n-1}$ distribution.
For example, if $n=5$, then $c_L \approx -2.78$,
$c_H \approx 2.78$ and the 95\% confidence interval is:
  $$CI = \bar{x} \pm 2.78 s_x/\sqrt{n}$$
As with hypothesis tests, finite sample confidence intervals
are typically more conservative (wider) than their asymptotic
cousins, but the difference becomes negligible as 
the sample size increases.

## Chapter review {-#review-inference}

In this chapter we have learned to formulate and test hypotheses, and to construct
confidence intervals. The mechanics of doing so are complicated, but you should not
let the various formulas distract you from the more basic idea of evidence: hypothesis
testing is about how strong the evidence is in favor of (or against) a particular 
true/false statment about the data generating process, and confidence intervals are
about finding a range of values for a parameter that are consistent with the observed
data.

In practice, modern statistical packages automatically calculate and report confidence
intervals for most estimates, and report the result of some basic hypothesis tests
as well. When you need something more complicated, it is usually just a matter of 
looking up the command. I will ask you to do these calculations yourself so you get used
to them, but it is more important that you can correctly interpret the results.

This is the last primarily theoretical chapter. The remaining chapters will be oriented towards
data and applications.

## Practice problems {-#problems-inference}

Answers can be found in the [appendix](#answers-inference).

**SKILL #1: Identify parameter, null and alternative**

1. Suppose we have a research study of the effect of the minimum wage on employment.  Let $\beta$ 
   be the parameter defining that effect. Formally state a null hypothesis corresponding to the
   idea that the minimum wage has no effect on employment, and state the alternative hypothesis 
   as well.

**SKILL #2: Classify test statistics as valid or invalid**

2. Which of the following characteristics do test statistics need to possess?
   a. The distribution of the test statistic is known under the null.
   b. The distribution of the test statistic is known under the alternative.
   c. The test statistic has the same distribution whether the null is true or false.
   d. The test statistic has a different distribution when the null is true versus when
      the null is false.
   e. The test statistic needs to be a number that can be calculated from the data.
   f. The test statistic needs to have a normal distribution.
   g. The test statstic's value depends on the true value of the parameter.

**SKILL #3: Find distribution of test statistic under the null**

3. Suppose we have a random sample of size $n$ on the random variable $x_i$ with unknown
   mean $\mu$ and unknown variance $\sigma^2$. The conventional T-statistic for the mean 
   is defined as
    $$t = \frac{\bar{x}-\mu_0}{\hat{\sigma}_x/\sqrt{n}}$$
   where $\bar{x}$ is the sample average, $\mu_0$ is the value of $\mu$ under
   the null, and $\hat{\sigma}_x$ is the sample standard deviation.
   a. What needs to be true in order for $t$ to have the $T_{n-1}$ distribution under
      the null?
   b. What is the asymptotic distribution of $t$?

**SKILL #4: Describe distribution of test statistic under the alternative**

4. Consider the setting from problem 3 above, and suppose that the true value of 
   $\mu$ is some number $\mu_1 \neq \mu_0$.  Write an expression describing $t$ 
   as the sum of (a) a random variable that has the $T_{n-1}$ distribution and (b) a 
   random variable that is proportional to $\mu_1-\mu_0$.

**SKILL #5: Find the size of a test**

5. Suppose that we have a random sample of size $n=14$ on the random variable
   $x_i \sim N(\mu,\sigma^2)$.  We wish to test the null hypothesis 
   $H_0: \mu = 0$.  Suppose we use the standard t-statistic:
    $$t = \frac{\bar{x}-\mu_0}{\hat{\sigma}_x/\sqrt{n}}$$
   a. Suppose we use critical values $c_L = -1.96$ and $c_H = 1.96$.  Use Excel to calculate 
      the exact size of this test.
   b. Suppose we use critical values $c_L = -1.96$ and $c_H = 1.96$.  Use Excel to calculate 
      the asymptotic size of this test.
   c. Suppose we use critical values $c_L = -3$ and $c_H = 2$.  Use Excel to calculate 
      the exact size of this test.
   d. Suppose we use critical values $c_L = -3$ and $c_H = 2$.  Use Excel to calculate 
      the asymptotic size of this test.
   e. Suppose we use critical values $c_L = -\infty$ and $c_H = 1.96$.  Use Excel to calculate 
      the exact size of this test.
   f. Suppose we use critical values $c_L = -\infty$ and $c_H = 1.96$.  Use Excel to calculate 
      the asymptotic size of this test.

**SKILL #6: Find critical values**

6. Suppose that we have a random sample of size $n=18$ on the random variable
   $x_i \sim N(\mu,\sigma^2)$.  We wish to test the null hypothesis 
   $H_0: \mu = 0$.  Suppose we use the standard t-statistic:
    $$t = \frac{\bar{x}-\mu_0}{\hat{\sigma}_x/\sqrt{n}}$$
   a. Use Excel to calculate the (two-tailed) critical values that produce an exact size of 1\%.
   b. Use Excel to calculate the (two-tailed) critical values that produce an exact size of 5\%.
   c. Use Excel to calculate the (two-tailed) critical values that produce an exact size of 10\%.
   d. Use Excel to calculate the (two-tailed) critical values that produce an asymptotic size of 1\%.
   e. Use Excel to calculate the (two-tailed) critical values that produce an asymptotic size of 5\%.
   f. Use Excel to calculate the (two-tailed) critical values that produce an asymptotic size of 10\%.

**SKILL #7: Construct and interpret a confidence interval**

7. Suppose we have a random sample of size $n = 16$ on the random variable 
   $x_i \sim N(\mu,\sigma^2)$, and we calculate the sample average $\bar{x} = 4$
   and the sample standard deviation $\hat{\sigma} = 0.3$.
   a. Use Excel to calculate the 95\% (exact) confidence interval for $\mu$.
   b. Use Excel to calculate the 90\% (exact) confidence interval for $\mu$.
   c. Use Excel to calculate the 99\% (exact) confidence interval for $\mu$.
   d. Use Excel to calculate the 95\% asymptotic confidence interval for $\mu$.
   e. Use Excel to calculate the 90\% asymptotic confidence interval for $\mu$.
   f. Use Excel to calculate the 99\% asymptotic confidence interval for $\mu$.

**SKILL #8: Interpret test results and confidence intervals**

8. Suppose you estimate the effect of a university degree on earnings at age 30, and 
   you test the null hypothesis that this effect is zero.  You conduct a test at the
   5\% level of significance, and reject the null.  Based on this information, classify 
   each of these statements as "probably true", 'possibly true", or "probably false":
   a. A university degree has no effect on earnings.
   b. A university degree has some effect on earnings.
   c. A university degree has a large effect on earnings.
9. Suppose you estimate the effect of a university degree on earnings at age 30, and 
   you test the null hypothesis that this effect is zero.  You conduct a test at the
   5\% level of significance, and fail to reject the null.  Based on this information, 
   classify each of these statements as "probably true", 'possibly true", or "probably false":
   a. A university degree has no effect on earnings.
   b. A university degree has some effect on earnings.
   c. A university degree has a large effect on earnings.
10. Suppose you estimate the effect of a university degree on earnings at age 30, and 
    your 95\% confidence inteval for the effect is $(0.10,0.40)$, where an effect of 0.10 
    means a degree increases earnings by 10\% and an effect of 0.40 means that a degree
    increases earnings by 40\%. Based on this information, 
    classify each of these statements as "probably true", 'possibly true", or "probably false":
    a. A university degree has no effect on earnings.
    b. A university degree has some effect on earnings.
    c. A university degree has a large effect on earnings, where "large" means at least 10\%.
    d. A university degree has a very large effect on earnings, where "very large" means 
      at least 50\%.