Note

Do monetary policy shocks affect output, employment, and prices? Meta-analyses on the effects of conventional monetary policy

About:

This repository contains raw and processed data, replication files and documentation for our meta-study on the macroeconomic effects of conventional monetary policy. Below, we provide guidance on our research process and the documentation of our data collection, data processing and transformation, and the scripts for the data analyses.

🏆 Winner of the SaxFDM Open Data Award 2025

Pre-registration:

The pre-registration and pre-analysis plan for our meta-study can be found at https://osf.io/cduq4.

Research papers:

Enzinger, M., Gechert, S., Heimberger, P., Prante, F., & Romero, D. F. (2025). The overstated effects of conventional monetary policy on output and prices. [OSF Preprint] [Replication files]

Abstract: We build a dataset of output and price effects of conventional monetary policy containing 146,463 point estimates and confidence bands from 4,871 impulse-response functions in 409 primary studies. Simple average responses suggest that interest rate hikes substantially dampen output and prices. However, we find robust evidence for publication bias. Bias corrections reduce effect sizes by half or more: in response to a 100 basis points rate hike, output and prices are unlikely to fall by more than 0.5 and 0.25 percent, respectively. Shock identification choices and publication characteristics correlate with effect sizes but are quantitatively less important than publication bias.

Project:

This repository is part of the research project Monetary Policy and Energy Prices funded by the European Macro Policy Network (EMPN).

---

Data collection

Literature data base search

We conducted a comprehensive search for literature that econometrically estimates effects of conventional monetary policy shocks on output, (un)employment or the price level. We detailed our search strategy for relevant literature in Sec. 3.3 and 3.4 of our pre-registration. We used the EconLit and the Google Scholar databases for our search of primary studies. Due to differences in their search behaviour¹, we decided to use one comprehensive query for EconLit and multiple simpler search queries for Google Scholar.

• The raw results and procedural details of our EconLit search can be accessed here.
• The raw results and procedural details of our Google Scholar search can be accessed here.

This yielded 7455 bibliographic entries from our EconLit search and 10810 bibliographic entries from our Google scholar search. After de-duplication, checks for the availability of abstracts and the inclusion of additional studies from related existing meta-studies (see here for detailed documentation of these steps and related files), our consolidated dataset of primary studies with available abstracts totaled at 10714 entries.

Abstract screening

This dataset of 10714 studies then entered into the artificial intelligence-supported abstract screening to exclude clearly ineligible studies according to our eligibility criteria as defined in Sec. 3.5 of our pre-registration. The title and abstract screening was conducted independently by two researchers using ASReview. Sec. 3.6.1 of our pre-registration presents the details of the abstract screening process. Sec. 3.7.1 of our pre-registration defines the stopping rules for the abstract screening phase. See here for further documentation and related files of the abstract screening.

After the abstract screening we conducted some validity tests as well as agreement and overlap analysis on the merged abstract screening data of both screeners. We then randomized the order of the potentially relevant studies and prepared files to assist and document the full text download. See here for the documentation and here for the R code for these steps.

Full text screening

PDF retrieval

Following the abstract screening, we proceeded to download full texts of the potentially relevant studies (i.e. relevant according to at least one screener). The full text download was conducted by research assistants using standardized procedures as documented here. To facilitate their work, the studies were divided into 26 sets² of about 100 bibliographic entries each. For each entry, research assistants attempted to access PDFs via the provided URLs or DOIs, or by searching Google Scholar when direct links were unavailable or not working. All PDFs were saved using their BibtexKey as filename in a centralized folder (which we cannot share due to copyright). During the download process, assistants verified that each PDF matched the bibliographic information, checked for the most recent version of working papers, and documented various attributes in the study sets, including availability status, whether the paper was retracted, duplicates, and whether it was a master's/bachelor's thesis (which we defined as non-eligible). Special attention was paid to accessing the most current versions of working papers that may have been subsequently published in journals. The data set was then subjected to a final duplicate check.

Full text assessment and coding

After retrieving the full texts, we conducted a systematic assessment and coding of each study following standardized procedures documented in our coding guidelines and our codebook. Each study was independently reviewed by one of five researchers who first assessed whether the study met our inclusion criteria. Importantly, studies were marked for exclusion if they lacked proper identification strategies (e.g., simple OLS without shock identification) or did not report confidence intervals or comparable effect size estimates. Reasons for exclusions were documented in the study set files.

For eligible studies, we developed a custom R package (MetaExtractR) to facilitate systematic data extraction using individual JSON files for each study,³ enabling version control and transparent documentation of all coding decisions and revisions through Git and GitHub. Researchers coded a comprehensive set of study characteristics and study-internal moderator variables, including identification strategies, estimation methods, sample characteristics, control variables, and publication characteristics. The JSON-based workflow allowed us to handle multiple models per study efficiently while maintaining the single-point-of-truth principle. When necessary, coding decisions were discussed among team members to ensure consistency and accuracy across the dataset. We also extensively double checked coding decisions, with a first round of full-study double checks on a subsample of our dataset (>10%)⁴ to identify systematic deviations between screeners and multiple further rounds to ensure consistency of these cases. For more difficult variables, 100% of the coding decisions were double checked.⁵

Graphical effect size data extraction

Since the effect sizes in our eligible studies were overwhelmingly reported as impulse response functions (IRFs) in graphical form, we implemented a systematic graphical extraction process using WebPlotDigitizer. Following our standardized extraction guidelines, researchers first captured high-quality screenshots of all relevant IRFs from each study, carefully documenting which model specification and outcome variable each graph represented. These screenshots were then processed using WebPlotDigitizer's semi-automatic extraction tools, where researchers aligned axes, traced the response curves (including point estimates and confidence bounds), and extracted the underlying data points.

For each IRF, we extracted three separate datasets: the point estimate, upper confidence bound, and lower confidence bound, saving them as individual CSV files alongside the original screenshots and WebPlotDigitizer project files (.tar) to ensure full reproducibility. The .tar-files can be opened using WebPlotDigitizer to compare the digitized values against the original graphs. The complete set of screenshots, extraction project files, and resulting data for all eligible studies is available in our effect sizes repository, organized by study identifier (key), model identifier (model_id) and response variable. This transparent approach allows for verification and potential corrections of any extraction, supporting the reproducibility of our meta-analysis results.

Snowballing

To complement our systematic database search, we conducted backward snowballing following the approach outlined in Section 3.4.2 of our pre-registration. After completing the full text screening, we identified the ten most recent eligible studies published in our dataset and systematically screened their reference lists for additional relevant studies. From these ten studies, we extracted a total of 406 references (see get_most_recent.R for the steps of this process). After removing duplicates and screening for relevance based on titles and abstracts, we identified 59 potentially relevant studies that had not been captured in our original database search. These underwent the same full text screening process as our main sample, resulting in 20 additional eligible studies⁶ being included in our meta-analysis dataset. Scripts from the snowballing process, including a final duplicate check against our existing dataset, can be found here.

External data

Citations

We collected citation counts of eligible studies from Google Scholar. This data collection was conducted systematically on July 15-16, 2024 to ensure temporal consistency across all measurements. For each included study, a research assistant searched Google Scholar using the study title and verified that the first search result matched our study's metadata (same authors, publication venue, and year). We then recorded the citation count and the search date. The complete citations data, including Google Scholar links and search dates, is stored in citations_for_included_studies.xlsx. Studies that could not be found on Google Scholar were noted accordingly.

Journal ranking

We use the 2022 SCImago Journal Rank (SJR) indicator to classify publications into top journals and other publications. Top journals belong to a list of top-50 economics journals according to the SJR. The 13 top journals for our sample are (in alphabetic order): American Economic Journal: Macroeconomics, American Economic Review, Brookings Papers on Economic Activity, Economic Journal, Journal of Business & Economic Statistics, Journal of Finance, Journal of Financial Economics, Journal of International Economics, Journal of Monetary Economics, Journal of the European Economic Association, Review of Economics and Statistics, Quantitative Economics, Review of Economic Studies.

World Bank income group classifications

For country income classifications, we utilized the World Bank income group classifications for fiscal year 2025, accessed on December 23, 2024 and stored as world_bank_country_goups_2025_fiscal_year.xlsx. These classifications divide countries into four income groups: high income, upper middle income, lower middle income, and low income. We matched each country in our dataset to its corresponding income group. Our sample only included countries from the World Bank's high-income and upper-middle-income groups, which we have labelled ‘advanced’ and ‘emerging’ respectively. Studies were then classified based on their country composition: those examining only countries from a single income group were labeled accordingly, while studies spanning multiple income groups were classified as "mixed_or_unclear".

---

Data processing and effect size standardization

After completing the data collection phase, we needed to standardize and transform the heterogeneous effect sizes from different studies to enable meaningful meta-analysis. This process involved three main components:

Data integration and standardization pipeline

We developed a custom R package, MetaExtractR, to systematically merge and standardize the study data. The package integrates:

• Study metadata and coding: Stored in individual JSON files for each study, containing all coded variables
• Effect size data: Extracted impulse response functions (IRFs) stored as CSV files, with separate files for point estimates and confidence bounds

The standardization process follows a multi-step approach including:

• Data merging: The MetaExtracR::final_join() function matches JSON metadata with corresponding IRF data for each study and model
• Effect size transformation: Based on the specific characteristics of each study (e.g., variable definitions and transformations, shock sizes, data frequency), the package applies the appropriate standardization formula to ensure comparability across studies
• Confidence interval and standard error calculation: The package approximates standard errors from confidence bands, accounting for different confidence levels

Our standardization approach handles several cases based on how variables are measured and transformed in the original studies (e.g., log levels vs. growth rates, cumulative vs. non-cumulative IRFs). The transformations for each case are detailed in our effect size transformation guide.

The final data processing after full text screening was implemented in final_join.R, resulting in a unified dataset final_join_json_irf_data.RData that was prepared for further analysis in final_data_preparation_working_paper_1.R. Importantly, we merged the external data to each observation and consolidated some moderator variables. For example, we consolidated coding categories like identification strategies into broader categories.

---

Data analyses

Folders with replication files for the data analyses in our research papers are linked above.

Footnotes

1. Check Sec. 3.3 of the pre-registration for details. ↩

2. The study_set_27.xlsx contains additional studies from the snowballing process (see below). ↩

3. The files are named after the unique study identifier (Key). ↩

4. See full_random_check entries in reason for doublecheck column in the study set files. ↩

5. See recoding_check entries in reason for doublecheck column in the study set files. See here for further documentation of these cases. ↩

6. In the linked file, the column inclusion indicates the eligibility of 20 studies. ↩