Robert C. Luskin, Ariel Helfer, and Gaurav Sood
How well does the gain in knowledge scores measure what people learn from campaigns and deliberative forums?
Read the manuscript · Editable source
- Post-process knowledge can track learning when observed gain does not. The general model guarantees nonnegative covariance under its monotonicity and independence assumptions, including conditional on true initial knowledge. Conditioning on noisy observed knowledge does not inherit that guarantee.
- In the retained proportional-learning simulations, observed gain's median correlation with true learning is about .01 and post-process knowledge's about .49. The filter retains 1,212 proportional, 240 additive and eight catch-up worlds out of 4,000 of each. These counts depend on the simulation grid; they do not rule out catch-up learning. Simulations
- The ordering is not general. Initial knowledge alone explains much of the association under proportional learning; gain can perform better under catch-up learning. In the poll item checks, observed gain on one randomly selected question set more often correlates more strongly with observed gain on the other than follow-up knowledge does. These checks target observed item gain, not topic-wide learning.
- Education comparisons depend on adjustment. For participants above their poll's education median, the pooled observed-gain difference is .017 [−.005, .041]; the post-process difference controlling for baseline is .066 [.049, .085]. These are associations with observed scores, with 95% posterior intervals. Estimates
- Attitude associations depend on the proxy. Across 129 indices, observed gain has positive coefficients on 50%, post-process knowledge on 59%, and post-process knowledge controlling for baseline on 53%. These descriptive summaries do not establish an absence of learning effects. Summary
- The candidate estimator is conditional mean gain given both waves' item responses. A joint IRT implementation estimates item parameters and the learning distribution, then integrates gain over each person's posterior. A separate simulation study evaluates it on new respondents. Its gains depend on the learning model, item difficulty range, and measurement invariance; the comparison reports failures as well as successes. The empirical polls remain comparisons of observed scores; the joint IRT model is not yet fitted to those polls.
The revised manuscript includes the earlier project's distinction between response-production error and item-sampling error, a bounded piecewise example, and selective learning across domains.
Paired simulations change questionnaire length and difficulty coverage while holding respondents and learning fixed. A two-domain experiment compares balanced and single-domain coverage. Baseline knowledge is included alongside observed gain and follow-up knowledge in the score and estimator comparisons. Questionnaire results, paired differences, and domain results record correlations, rank correlations and ceilings.
The empirical validation compares correlations with observed gain on a separate set of questionnaire items.
Item selection and residualization use training respondents only; discussion
groups stay together in respondent folds. Reported ranges describe variation
across partitions, not confidence intervals or validation against true learning.
Results and sample accounting
are regenerated by make analysis.
Participant-level knowledge scores and attitude indices for 21 Deliberative Polls come from the corrected dp_data export, pinned to commit f88da7fa904cde55044dd3dda9fd337ca26cb8bd. Inputs are read directly from the exports in ../dp_data; SHA-256 checks verify their contents on every analysis run. Data instructions describe the fixed within-poll median education rule.
Clone dp_data alongside this repository, or set DP_DATA_ROOT to its location. Check out the revision recorded in data/sources.csv. Missing or modified exports stop the build. R packages are pinned in renv.lock; the paper also needs Pandoc and XeLaTeX.
make restore
make checkmake check runs the simulations and analyses, draws the figures, renders the manuscript, and runs linting and tests. It reads the exports directly from dp_data without copying them. No network access is needed after the source clone and dependencies are available. make ci-docker runs the same checks in the standard Rocker R image, mounting dp_data read-only. GitHub Actions checks out the pinned dp_data revision separately.
The joint IRT comparison uses 80 training/test replications across eight scenarios. make rebuilds these results when the model, comparison code, or lockfile changes; Rscript R/04_run_all.R reruns every stage. Run IRT_CORES=3 make -B estimation to force a full rerun with three worker processes on macOS or Linux. Each fit uses 300 training respondents; predictions are evaluated on 1,000 independent respondents. The positive knowledge trait is exp(theta), with the baseline log-trait mean fixed at zero and variance at one. It is not a proportion of all knowable facts. Guessing probabilities are supplied by the simulation design; item slopes, intercepts, and change-distribution parameters are estimated. Posterior intervals condition on those fitted parameters.
To score new respondents with the fitted model:
source("R/00_common.R")
source("R/01_simulations.R")
model <- fit_joint_irt(training_pre, training_post, guess = item_guess)
stopifnot(model$converged, length(model$warnings) == 0)
gain <- posterior_irt_learning(model, new_pre, new_post)Each response matrix has one row per respondent and one column per matched item, coded 0/1 with missing responses as NA. Rows must identify the same respondents across waves, and item columns must have the same order in training and scoring. item_guess supplies known guessing probabilities. The returned estimate, lower, and upper describe gain on the positive-trait scale; the limits condition on the fitted parameters. They are not percentages of topic-wide knowledge.
Set DP_DATA_ROOT to use another local clone. Changing the upstream revision requires reviewing new checksums and regenerating the analyses.
| Folder | Contents |
|---|---|
data/sources.csv |
Upstream paths, revision and SHA-256 checksums |
R/ |
Numbered analysis stages and supporting functions |
tabs/, figs/ |
Generated tables and figures |
ms/ |
Manuscript source, bibliography and PDF |
tests/ |
Checks on the model and the outputs |
The five R files have one role each:
00_common.R: paths, input checks, table helpers, number formatting and figure styles.01_simulations.R: learning simulations and estimator comparisons.02_empirical.R: poll analyses and comparisons across question sets.03_figures.R: figure generation.04_run_all.R: run the workflow and compile the manuscript.
Run Rscript R/04_run_all.R from the repository root, or use make paper.
To run selected stages, pass their names, for example
Rscript R/04_run_all.R figures manuscript. make check also runs linting and tests.
- finite-sample/know_pgap_format — How do (biased) guessing encouraging features and guessing agnostic coding techniques affect the partisan gap?
- finite-sample/guess — Adjust naive estimates of learning for guessing
- finite-sample/know_guess_forget — Replication and modern reanalysis of Cor and Sood (2016)
- finite-sample/know_hidden — Mis-measuring Political Knowledge? Do People Know More—or Even Less—about Politics than Commonly Thought?
- finite-sample/know_pgap_review — Replication Materials For "A Gap in Our Understanding? Reconsidering the Evidence for Partisan Knowledge Gaps"
Powered by Adjacent