Replication materials for the systematic literature review (SLR) on Multi-Agent Debate (MAD). The study follows Kitchenham & Charters (2007) SLR guidelines and Wohlin (2014) snowballing, as described in the paper's Method section.
The review proceeds in five stages:
| Stage | Description |
|---|---|
| (i) | Definition and scoping of Multi-Agent Debate |
| (ii) | Seed search string in Scopus |
| (iii) | Independent screening (title → abstract → full text) |
| (iv) | Backward and forward snowballing (single iteration each) |
| (v) | Structured data extraction and inductive coding |
MAD is scoped to natural-language tasks among AI agents only (no human debate participants). A study qualifies when it involves:
- A discussion between two or more AI agents
- Agents that can put forward opposing arguments across multiple rounds
- Agents that collaboratively discuss the same task (not mere subtask hand-offs)
See 00-scoping/Characteristiscs that define what is MAD.docx for the working definition document used during the review.
| ID | Criterion |
|---|---|
| IC1 | Proposes or uses debate strategies satisfying the operational MAD definition |
| EC1 | Insufficient detail on the debate strategy (debate not the focus) |
| EC2 | Debates involve both AI agents and humans |
| EC3 | Debates not applied to natural language tasks |
| EC4 | Paper not in English |
| EC5 | Full text not accessible |
| EC6 | Duplicate or superseded version |
When multiple arXiv versions existed, the most recent was kept. When both arXiv and a peer-reviewed venue existed, the conference/journal version was preferred.
Search fields: title, abstract, keywords. No domain or date restriction.
(LLM OR "Large Language Model" OR LLMS OR "Large Language Models" OR "AI Agent" OR "AI Agents") AND (MAD OR "Multi-Agent Debate")
All candidate papers were reviewed independently by two authors; disagreements were resolved by discussion until consensus.
| Phase | Retrieved | After dedup / filter | Included (cumulative) |
|---|---|---|---|
| Scopus seed search | 29 | — | 12 seed papers |
| Backward snowballing | 527 (+169 duplicates removed → 358 screened) | — | +13 → 25 |
| Forward snowballing | 929 (+40 duplicates → 889) | Pre-filter removed 367 clearly out-of-scope → 522 screened | +146 → 171 |
Forward snowballing used an extra pre-filtering step (one author) before pairwise review by three author pairs.
Figure: Overview of the search and snowballing process (also available as assets/figures/selection-process.pdf).
Publication breakdown among selected studies (per paper): 51 conferences, 7 journals, 113 arXiv preprints; earliest included work from 2023.
MAD-CSUR/
├── README.md
├── LICENSE
├── 00-scoping/ ← Stage (i): MAD definition & scope
├── 01-initial-scopus-search/ ← Stage (ii–iii): seed search & screening
├── 02-backward-snowballing/ ← Stage (iv): backward snowballing
├── 03-forward-snowballing/ ← Stage (iv): forward snowballing
├── 04-data-extraction/ ← Stage (v): final selection & coding
├── assets/figures/ ← Selection flow diagram
└── scripts/snowballing/ ← Scripts and CSVs for snowballing
Full-text PDFs are not included in this package. Primary studies remain available through their original publishers and bibliographic records listed in 04-data-extraction/replication_package.xlsx.
| File | Role |
|---|---|
00-scoping/Characteristiscs that define what is MAD.docx |
Working document operationalizing MAD criteria (Table mad_criteria in the paper) |
Two independent reviewer spreadsheets (29 Scopus hits each; title/abstract/full-text decisions):
| File | Reviewer |
|---|---|
01-initial-scopus-search/reviewer-mo/1a. selection_initial_search_mo.xlsx |
MO — includes sheets Search string and DB, Selection Filtering 1st round |
01-initial-scopus-search/reviewer-jm/1b. selection_initial_serach_jm.xlsx |
JM — includes sheets Search string, DB, criteria, Selection Filtering 1st round |
Outcome: 12 seed papers (17 excluded). No reviewer disagreements at this stage.
References from seed papers were collected and screened (title → abstract → full text).
| File | Role |
|---|---|
02-backward-snowballing/reviewer-jm/2a. backwardsnowballing_jm.xlsx |
JM screening workbook (sheets: All, without duplicates, filter by title, filter by abstract) |
02-backward-snowballing/reviewer-mo/2b. backwardsnowballing_mo.xlsx |
MO screening workbook |
02-backward-snowballing/consensus/2c. discussion backward snowballing jm+mo.xlsx |
Consensus resolution of reviewer disagreements |
Outcome: 527 references found → 358 unique after deduplication → 13 additional included → 25 cumulative.
Citations to seed + backward-included papers were collected. A pre-filter removed clearly out-of-scope items before pairwise review.
| File | Role |
|---|---|
03-forward-snowballing/pre-filter/3a. forwardsnowballing_first_filter_mo.xlsx |
Initial forward citation set, deduplication, and pre-filter (367 removed) |
03-forward-snowballing/merged-candidates/4. forwardsnowballing_all.xlsx |
Merged candidate list after pre-filtering |
03-forward-snowballing/reviewer-mo/4a. forwardsnowballing_mo.xlsx |
MO pairwise screening |
03-forward-snowballing/reviewer-jm/4b. forwardsnowballing_jm.xlsx |
JM pairwise screening |
03-forward-snowballing/reviewer-qm/4c. forwardsnowballing_qm.xlsx |
QM pairwise screening |
03-forward-snowballing/consensus/4d. fowardsnowballing - discussion.xlsx |
Consensus resolution |
Outcome: 929 forward citations → 889 unique → 522 after pre-filter → 146 additional included → 171 final primary studies.
| File | Role |
|---|---|
04-data-extraction/replication_package.xlsx |
Final consolidated study list and coded taxonomy |
| Sheet | Contents |
|---|---|
selected_papers |
Bibliographic metadata for included primary studies (ID, Authors, Title, Year, Source title, …) |
data_extraction |
Inductive coding schema across three taxonomy dimensions: Participants, Interactions, Agreement (plus memory, model family, task/domain columns). A single paper may appear multiple times when it reports several MAD variants. |
Coding dimensions align with the paper taxonomy (participants, interactions, agreement strategies).
| File | Role |
|---|---|
assets/figures/selection-process.png |
Search & selection flow diagram (PNG, embedded above) |
assets/figures/selection-process.pdf |
Search & selection flow diagram (vector source) |
| File | Role |
|---|---|
scripts/snowballing/fetch_abstracts.py |
Fetches missing abstracts from the Semantic Scholar API for snowballing CSVs |
scripts/snowballing/input_csvs/citations.csv |
Forward-snowballing citation export (input) |
scripts/snowballing/input_csvs/citations - original.csv |
Citation export before manual edits |
scripts/snowballing/output_csvs/citations.csv |
Citations enriched with fetched abstracts |
Run from scripts/snowballing/:
pip install requests
python fetch_abstracts.py| Question | Where to look |
|---|---|
| What counts as MAD? | 00-scoping/ + IC/EC tables above |
| Scopus query and seed screening? | 01-initial-scopus-search/ |
| Backward snowballing decisions? | 02-backward-snowballing/ |
| Forward snowballing + pre-filter? | 03-forward-snowballing/ |
| Final included papers & codes? | 04-data-extraction/replication_package.xlsx |
| Selection flow numbers? | assets/figures/selection-process.pdf |
| How were abstracts obtained? | scripts/snowballing/ |
If you use this replication package, please cite the corresponding publication (citation to be added upon publication).
See LICENSE. Review artifacts (spreadsheets, scripts, figures) are shared under CC BY 4.0.
Primary study full texts are not redistributed in this package and must be obtained from their respective publishers.