[WaylandYang]

C1 of 0016: an answer key that does not come from us.

truth/*.json has been hand-filled, which means an answer can be written too narrow, and once written it is a record of what the system said last time — a miss no longer tells you whether the key or the system is wrong. The corpora that come from Wikipedia have an answer nobody here wrote: every article's subject, and every article it links to, is an item on Wikidata with a P31 (instance of).

Three files, and the human judgement lives in exactly one of them.

• scripts/bench/fetch-wikidata-truth.mjs — for a Wikipedia-derived corpus, resolves each article and the pages it links to (kept only when the name actually appears in the article text: navboxes contribute hundreds of unrelated links) to Wikidata items, pulls their P31 classes over SPARQL, and records for each class which of ~36 anchor classes it falls under via P279*. Output truth/<corpus>.wikidata.raw.json: refetchable, diffable, untouched by hand.
• scripts/bench/p31-to-classes.json — anchor class → acceptable schema.org keys (as key_from_iri spells them). Anchors belong to families with precedence, because Wikidata's subclass chains cross what an ontology keeps apart: a consolidated city-county is a local government is an organization, so without the rule San Francisco would accept corporation. Person beats everything, a place is not an organization, a work or an organization is not an event, anything concrete beats the abstract families. Empty list keeps the existing semantics — the ontology has nothing for it, leave the entity alone; an entity under no anchor at all is omitted rather than guessed.
• scripts/bench/make-truth.mjs — raw + mapping → truth/<corpus>.wikidata.json, same shape as the hand-filled keys. run.mjs --truth <file> scores a run against it (and reports which key it used), so one run can be scored against both keys and the differences are the first thing to read.

Generated for ai-timeline (15 articles: 610 entities with an answer, 33 to leave alone, 46 omitted) and ai-timeline-ends (6 snapshots: 184 / 7 / 11). Spot checks: OpenAI → corporation|organization; GPT-4 → software_application|product|…; Sam Altman → person; San Francisco → city|place|administrative_area; The New York Times → newspaper|periodical; Removal of Sam Altman from OpenAI → event; AI safety → [].

Not in this cut: running the benchmark itself (it needs a live server and a model; nothing changes in extraction or resolution here), and answer keys for the non-Wikipedia corpora (there is no external answer for pharma / tech; they stay hand-filled).

🤖 Generated with Claude Code