Repository navigation
Type resolution runs itself after extraction, and applies only what stays in the subtree - #297
Merged
Merged
Conversation
…tays in the subtree Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: WaylandYang <wayland0916@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: WaylandYang <wayland0916@gmail.com>
…the bench scores refinements as hits Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: WaylandYang <wayland0916@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
C2 of 0016, the last open item of 0001 P3a: type resolution ran only when someone clicked. A document was extracted, its entities sat on a coarse class, and nothing happened until somebody opened the Ontology page and pressed Run.
What changes
auto_type_resolution(Settings → knowledge base, on by default — see the measurement below). When it is on, the end of every extraction queues oneresolve_typesjob for the base — deduplicated, so a batch of documents queues one job, not one per document (jobs::enqueue_unless_queued).entities.type_resolved_at) and never looks at it again on its own: the candidate query orders by fact count and takes sixty, so without the mark every round would be the same sixty. And the review tier is not persisted by the job — those entities stay candidates, and the manual round (which ignores the mark) shows them the next time someone presses Run. A job does at most ten rounds; anything left waits for the next extraction.type_source = 'human'), as before.Measured before deciding the default (
ai-timeline× schema.org pack loaded at creation × the Wikidata key from #289; 2,567 entities, 2,933 facts after extraction):The review tier is where the risk sits (OpenAI → research_organization, ChatGPT Plus → service), which is what the tiering is for. The auto tier scored 39/41 over both runs; of the two misses, LinkedIn → corporation is the key being narrower than reality, and the Delaware court → courthouse is a real one (the institution taken for its building). Every batch is listed on the Ontology page and can be undone. So the switch defaults to on; existing bases get it on the next start.
Two scoring fixes fell out of this and are in the PR, because the first pass scored the same rounds at 50%: the generated key now matches names exactly (
match: "exact"— substring matching had handed OpenAI's answer to "OpenAI Foundation" and "OpenAI for Government"), and a class counts as a hit when it is an accepted class or a descendant of one in the base's hierarchy (research_organization for organization, city for administrative_area — a refinement is what the auto tier is supposed to produce).run.mjsreportsscore.tiers.The job was exercised end to end on the bench base: switch on,
resolve_typesqueued, 10 rounds, 600 entities marked as judged, 94 retyped in 9 batches, auditontology.types_resolvedwithvia: joband no actor.Tests:
a_judged_entity_waits_its_turn(store, DB): the unattended candidate query skips judged entities and the manual one does not, human-typed entities are skipped by both, a second job for the same base does not queue while one is queued. Existing type-decision tests unchanged apart from the new parameter.🤖 Generated with Claude Code