Bug in WildBootTests.jl 0.9.7 when demeaning in the bootstrap (using the fe argument of boottest)

[s3alfisc]

• Unit tests fail against WildBootTests 0.9.7. See FYI, small changes to Julia wildboottest() interface #103
• tests fail due to inequalities in lower number digits -> false positives
• solutions: one or more of {increase tolerance, increase sample size, change seed, increase number of bootstrap iterations}

[s3alfisc]

Actually, I start to think that this might be a bug in WildBootTests.jl?

Summary:

• p-values are different for fixed effects models when engine = "WildBootTests.jl"
• this is idependent of the model (i.e. felm or fixest)
• the error does not occur if engine = "R"

Code:

library(fwildclusterboot)
library(fixest)
library(lfe)

voters <-
  fwildclusterboot:::create_data(
    N = 10000,
    N_G1 = 20,
    icc1 = 0.01,
    N_G2 = 10,
    icc2 = 0.01,
    numb_fe1 = 10,
    numb_fe2 = 10,
    seed = 71986045
  )

# without "fe"
fit_lm <- lm(proposition_vote ~ treatment + Q1_immigration + Q2_defense, data = voters)
fit_feols <- feols(proposition_vote ~ treatment + Q1_immigration  + Q2_defense, data = voters)
fit_felm <- felm(proposition_vote ~ treatment + Q1_immigration + Q2_defense, data = voters)

# R algo
dqrng::dqset.seed(123)
boot_lm <- boottest(fit_lm, param = ~treatment, clustid = ~group_id1, B = 999)
dqrng::dqset.seed(123)
boot_feols <- boottest(fit_feols, param = ~treatment, clustid = ~group_id1, B = 999)
dqrng::dqset.seed(123)
boot_felm <- boottest(fit_felm, param = ~treatment, clustid = ~group_id1, B = 999)

res <- list(boot_lm, boot_feols, boot_felm)
lapply(res, pval) |> unlist()
# [1] 0.3453453 0.3453453 0.3453453

# without fe, WildBootTests.jl
dqrng::dqset.seed(123); set.seed(123)
boot_lm <- boottest(fit_lm, param = ~treatment, clustid = ~group_id1, B = 999, engine = "WildBootTests.jl")
dqrng::dqset.seed(123); set.seed(123)
boot_feols <- boottest(fit_feols, param = ~treatment, clustid = ~group_id1, B = 999, engine = "WildBootTests.jl")
dqrng::dqset.seed(123); set.seed(123)
boot_felm <- boottest(fit_felm, param = ~treatment, clustid = ~group_id1, B = 999, engine = "WildBootTests.jl")

res <- list(boot_lm, boot_feols, boot_felm)
lapply(res, pval) |> unlist()
# [1] 0.3113113 0.3113113 0.3113113


# with one "fe"
fit_feols <- feols(proposition_vote ~ treatment + Q1_immigration | Q2_defense, data = voters)
fit_felm <- felm(proposition_vote ~ treatment + Q1_immigration | Q2_defense, data = voters)

# R algo
dqrng::dqset.seed(123)
boot_feols <- boottest(fit_feols, param = ~treatment, clustid = ~group_id1, B = 999)
dqrng::dqset.seed(123)
boot_felm <- boottest(fit_felm, param = ~treatment, clustid = ~group_id1, B = 999)

res <- list(boot_feols, boot_felm)
lapply(res, pval) |> unlist()
# [1] 0.3453453 0.3453453 - same as above

# with fe, WildBootTests.jl
dqrng::dqset.seed(123); set.seed(123)
boot_feols <- boottest(fit_feols, param = ~treatment, clustid = ~group_id1, B = 999, engine = "WildBootTests.jl")
dqrng::dqset.seed(123); set.seed(123)
boot_felm <- boottest(fit_felm, param = ~treatment, clustid = ~group_id1, B = 999, engine = "WildBootTests.jl")

res <- list(boot_feols, boot_felm)
lapply(res, pval) |> unlist()
# [1] 0.3113113 0.3113113

# with one fe, use fe argument
dqrng::dqset.seed(123)
boot_feols <- boottest(fit_feols, param = ~treatment, clustid = ~group_id1, B = 999, fe = ~Q2_defense)
dqrng::dqset.seed(123)
boot_felm <- boottest(fit_felm, param = ~treatment, clustid = ~group_id1, B = 999, fe = ~Q2_defense)

res <- list(boot_feols, boot_felm)
lapply(res, pval) |> unlist()
# [1] 0.3453453 0.3453453 - same as above

# with fe, WildBootTests.jl
dqrng::dqset.seed(123); set.seed(123)
boot_feols <- boottest(fit_feols, param = ~treatment, clustid = ~group_id1, B = 999, engine = "WildBootTests.jl", fe = ~Q2_defense)
dqrng::dqset.seed(123); set.seed(123)
boot_felm <- boottest(fit_felm, param = ~treatment, clustid = ~group_id1, B = 999, engine = "WildBootTests.jl",, fe = ~Q2_defense)

res <- list(boot_feols, boot_felm)
lapply(res, pval) |> unlist()
# this is different!
# [1] 0.3123123 0.3123123

[s3alfisc]

Fixed with the most recent version of WildBootTests.jl.