Run the derivation contract once and read the vocabulary

[antejavor]

Part of #344

Question

Does #353's derivation contract produce a usable vocabulary when it actually runs? Nobody has run it. Whether an LLM proposes a sensible domain vocabulary from session text, and prunes a GLiNER2 observation table sensibly, is the same kind of question #350 answered for extraction: by reading real output, not by argument.

What to build (throwaway, branch only, same convention as #350)

The #353 pipeline, end to end, once:

1. Propose. Whole sessions, time-stratified, up to a prompt budget, go to an LLM. It returns domain entity types (label plus description), relation names, and any corpus-specific value types, on top of the fixed core (User, Person, Duration, Quantity, Money, Date, TimeWindow). The prompt states that relation names are all the extractor sees (Near-synonymous relation labels all fire on the same pair #360: relation descriptions are dropped by joint_ie/compiler.py:27).
2. Observe. One permissive GLiNER2 pass over a larger sample: per-relation endpoint type tables with example edges, and per-type mention stats. Hold every compiled schema for the whole run (unstructured2graph: gliner2's compiled-schema cache is keyed on a memory address #365).
3. Prune. The LLM keeps sensible endpoint pairs, sets identity per derived type, and drops zero-instance types and relations. It doesn't rename or add.
4. Validate. Run hygm.validation's hard gate, including Which User mentions are the user #358's rule that User in a range requires Person there too.
5. Stability. Derive a second time from a disjoint propose sample, then report behavioural agreement: both vocabularies over one shared sample, types aligned by span co-assignment.

Run it on the eval corpus (LongMemEval) and, if feasible, on a sample of real Claude Code sessions. #347 said each corpus gets its own vocabulary, and production is coding sessions.

What to read, and judge by human reading of sampled output (the map's decision criterion)

• Is the derived vocabulary better or worse than Wire a constrained ontology and read the edges #350's hand-written stand-in on the same 10 sessions? Compare with the committed baseline (1004 raw edges; prototype/gliner2-constrained-edges).
• Does prune recover visited -> Organization (How wide should a relation's declared range be #359's MoMA edge) and drop knows -> Topic?
• Are the per-type identity calls right? Person/Organization-like types should be global, generic ones chunk.
• Does the proposer produce near-overlapping relation names (Near-synonymous relation labels all fire on the same pair #360), and if so, what's the co-firing rate?
• How much do the two derivations agree behaviourally, and does the disagreement land where a human would expect ambiguity?
• Cost: LLM tokens per derivation, and wall clock for the observe pass.

Not in scope

The evolution loop (revision, versioning, migration). This ticket runs v1 once, plus the stability derivation.

[antejavor]

From #352: observe with one turn per window (turns over 384 words split within the turn), since that's what production extraction will run. Word windows mix speakers and produce edges #358's gate can't catch.

[antejavor]

Resolution

The contract works once its input is clean, but one ~10-session propose sample decides which domains exist. Evidence: prototype/gliner2-constrained-edges @ a2f0b6b, README section J (derive.py, read_derived.py, observe_diagnosis.py, LLM outputs with provenance in derivation/).

What ran

Two derivations (A, B) from disjoint time-stratified propose samples (9-10 whole sessions, ~115k chars each), one shared 60-session observe sample, one turn per window. The 10 evidence sessions were in no sample. LLM: claude -p (claude-opus-5-5), no user settings or tools. Both passed the hard validation gate.

Read on the 10 evidence sessions (constrained, one turn per window)

vocabulary	raw	distinct	answer-bearing	MoMA visited	co-firing	User endpoints not the user*
hand (#361)	895	393	8	0.66-0.78	55/315 pairs	39%
derived A	433	186	0	none	15/162	19%
derived B	842	313	9	0.94-0.97	41/250	25%

*removed at write time by #358's gate either way.

• Coverage is the whole difference. A's sample had no travel: no Location, no visited, zero answers. B's did and roughly matches the hand vocabulary. Prune cannot add, so this is unrecoverable downstream.
• Prune is good once observation is clean. A kept 9/12 relations, B 16/20; the drops were junk (has_player_on_roster, favorite_of_user, friend_of_user). Identity calls were right: Organization/Location/CreativeWork global; Device/Dish/Activity/JobRole chunk. knows -> Topic (How wide should a relation's declared range be #359) never arose: no derivation proposed it.
• No relation into a core value type was proposed by either, so 25:50 is lost.
• Near-synonyms across samples are the same spans: JobRole->Occupation 31/32, Device->Possession 89/117.
• Stability: 84% agreement on the 1213 spans both typed, while 876 spans are typed only by B. Agreement alone called a broken derivation stable.
• Tense variants co-fire and contradict: plans_to_visit(user -> Museum of Modern Art) on the same visit as visited_location; visited_location co-fires with a variant on 21/30 pairs.
• Cost: ~$0.55 of LLM per derivation (propose ~40k input tokens, prune 15-22k). Observe was ~1 h for 60 sessions at the raised caps, the dominant cost.

Found on the way: #371

The first run observed 245 edges over 709k chars and prune kept 1 relation of 12: every user fact came out typed Activity/Date. gliner2 cuts relation candidates at relation_pair_cap/max_edges_per_type with ties broken alphabetically by type name; a permissive endpoint makes types² tied copies of each span pair, so past 11 types User, sorting last, never heads an edge (0 of 355 at the defaults; 821 of 1827 with both caps at 4096). Constrained extraction also loses ~4% of User edges (895 -> 928), because #358's rule puts Person beside User. Filed as #371.

Decisions (revising #353; comment there)

1. Batched propose: K disjoint time-stratified batches, unioned.
2. One consolidate LLM call before observe merges synonyms across batches; renames are free there. Prune stays no-rename.
3. The proposer must propose relations into the core value types; a core value type with no relation into it is reported (reporting-only).
4. No modal/tense variants (plans_to_X beside X); exception to Near-synonymous relation labels all fire on the same pair #360 noted there.
5. Stability = agreement plus coverage (spans only one vocabulary types; answer-bearing shape). Reporting-only.
6. Observe sets relation_pair_cap = max_edges_per_type = 4096; unstructured2graph: gliner2 cuts relation candidates alphabetically by type on ties #371 recommends the same for production extraction.
7. The revised contract's run is Run the revised derivation contract: batched propose and consolidate #372 (accept: every seed reaches B's coverage, 25:50 lands, no tense pair).