Skip to content

scripts

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

scripts

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 
arbiter-batch.ts
arbiter-batch.ts
 
 
arbiter-brave.ts
arbiter-brave.ts
 
 
arbiter-claim-groups.ts
arbiter-claim-groups.ts
 
 
arbiter-collect.ts
arbiter-collect.ts
 
 
arbiter-details.ts
arbiter-details.ts
 
 
arbiter-explore.ts
arbiter-explore.ts
 
 
arbiter-extract-sources.ts
arbiter-extract-sources.ts
 
 
arbiter-find-studies.ts
arbiter-find-studies.ts
 
 
arbiter-intercept-sources.ts
arbiter-intercept-sources.ts
 
 
arbiter-recollect.ts
arbiter-recollect.ts
 
 
arbiter-run.ts
arbiter-run.ts
 
 
arbiter-scout.ts
arbiter-scout.ts
 
 
arbiter-step.ts
arbiter-step.ts
 
 
collect-arbiter-claims.ts
collect-arbiter-claims.ts
 
 
fetch-articles.ts
fetch-articles.ts
 
 
generate-diff-questions.ts
generate-diff-questions.ts
 
 
generate-dummy-data.js
generate-dummy-data.js
 
 
generate-dummy-data.ts
generate-dummy-data.ts
 
 
generate-questions.ts
generate-questions.ts
 
 
group-claims.ts
group-claims.ts
 
 
parse-arbiter-claims.ts
parse-arbiter-claims.ts
 
 
parse-convex-claims.ts
parse-convex-claims.ts
 
 
regroup-all-claims.ts
regroup-all-claims.ts
 
 
run-gpt54mini-judge.ts
run-gpt54mini-judge.ts
 
 
run-judge-benchmark.ts
run-judge-benchmark.ts
 
 
test-2users-final.ts
test-2users-final.ts
 
 
test-5users.ts
test-5users.ts
 
 
test-production.ts
test-production.ts
 
 
test-walkthrough.ts
test-walkthrough.ts
 
 

README.md

Scripts

fetch-articles.ts

Fetches Wikipedia article revisions via the MediaWiki API and converts them into the structured JSON format used by the experiment platform. For each article it pulls two versions: a pinned past revision (by revision ID) and whatever the current revision happens to be. The output lands in public/data/articles/ as {articleId}-past.json and {articleId}-current.json.

Running it

From the app/ directory:

npx tsx scripts/fetch-articles.ts

Requires Node 18+ (uses the built-in fetch API). No extra npm dependencies.

The script waits one second between API calls to stay within Wikipedia's rate-limit expectations. A full run (8 articles, 16 API calls) takes roughly 20 seconds.

What it produces

Each JSON file contains an Article object matching the type defined in src/types/index.ts: a title, revision date, revision ID, and an array of sections. Every section has its heading, heading level, cleaned plain-text content, and an array of extracted citations (with URLs when available).

The wikitext-to-plaintext conversion strips templates, internal/external links, ref tags, HTML, and formatting markup. Citations are extracted from <ref> tags before the ref tags are removed from the body text.

Adding articles

Edit the ARTICLES array at the top of the script. Each entry needs:

  • id — short slug used for filenames and cross-referencing
  • title — exact Wikipedia article title
  • pastRevId — the specific revision ID for the "past" snapshot
  • pastDate — ISO date string for that revision

You can find revision IDs on any article's history page.