Skip to content

Type resolution runs itself after extraction, and applies only what stays in the subtree - #297

Merged
WaylandYang merged 3 commits into
devfrom
feat/type-resolution-queued
Sep 4, 2026
Merged

WaylandYang merged 3 commits into
devfrom
feat/type-resolution-queued

Conversation

@WaylandYang

@WaylandYang WaylandYang commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

C2 of 0016, the last open item of 0001 P3a: type resolution ran only when someone clicked. A document was extracted, its entities sat on a coarse class, and nothing happened until somebody opened the Ontology page and pressed Run.

What changes

  • A per-base switch, auto_type_resolution (Settings → knowledge base, on by default — see the measurement below). When it is on, the end of every extraction queues one resolve_types job for the base — deduplicated, so a batch of documents queues one job, not one per document (jobs::enqueue_unless_queued).
  • The job runs the same judgement as the manual round — candidates from the class index, one adjudication call per sixty entities — with the same tiering: a refinement inside the current class's subtree is applied on its own; a re-classification across the tree is left for a person. Nothing about the tiers moved.
  • Two things differ from the manual round. The engine marks every entity it judged (entities.type_resolved_at) and never looks at it again on its own: the candidate query orders by fact count and takes sixty, so without the mark every round would be the same sixty. And the review tier is not persisted by the job — those entities stay candidates, and the manual round (which ignores the mark) shows them the next time someone presses Run. A job does at most ten rounds; anything left waits for the next extraction.
  • Human decisions stay untouchable on both paths (type_source = 'human'), as before.

Measured before deciding the default (ai-timeline × schema.org pack loaded at creation × the Wikidata key from #289; 2,567 entities, 2,933 facts after extraction):

judged by the key hit miss not in the key
manual round, auto tier (60 candidates) 10 10 0 4
manual round, review tier if applied blindly 7 2 5 2
the job, 10 rounds, auto tier (94 retypes) 32 30 2 62

The review tier is where the risk sits (OpenAI → research_organization, ChatGPT Plus → service), which is what the tiering is for. The auto tier scored 39/41 over both runs; of the two misses, LinkedIn → corporation is the key being narrower than reality, and the Delaware court → courthouse is a real one (the institution taken for its building). Every batch is listed on the Ontology page and can be undone. So the switch defaults to on; existing bases get it on the next start.

Two scoring fixes fell out of this and are in the PR, because the first pass scored the same rounds at 50%: the generated key now matches names exactly (match: "exact" — substring matching had handed OpenAI's answer to "OpenAI Foundation" and "OpenAI for Government"), and a class counts as a hit when it is an accepted class or a descendant of one in the base's hierarchy (research_organization for organization, city for administrative_area — a refinement is what the auto tier is supposed to produce). run.mjs reports score.tiers.

The job was exercised end to end on the bench base: switch on, resolve_types queued, 10 rounds, 600 entities marked as judged, 94 retyped in 9 batches, audit ontology.types_resolved with via: job and no actor.

Tests: a_judged_entity_waits_its_turn (store, DB): the unattended candidate query skips judged entities and the manual one does not, human-typed entities are skipped by both, a second job for the same base does not queue while one is queued. Existing type-decision tests unchanged apart from the new parameter.

🤖 Generated with Claude Code

WaylandYang and others added 3 commits September 4, 2026 11:40
…tays in the subtree

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
…the bench scores refinements as hits

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
@WaylandYang
WaylandYang merged commit ed3e336 into dev Sep 4, 2026
@WaylandYang
WaylandYang deleted the feat/type-resolution-queued branch September 4, 2026 05:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant