You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: unify competency questions across ontology design and coverage evaluation #73
Competency Questions (CQs) are a single artifact that serves both directions of ontology work: they scope the ontology at design time (requirements) AND validate it at eval time (test suite). This is the classic Grüninger & Fox (1995) framing — the questions you write to define an ontology are the same questions you use to verify it.
CB today does not use CQs. evals.py generates eval questions from four sources — seed, doc, persona, work-item — tagged with layer hints. That's source-driven coverage testing, related to but distinct from ontology-competency-driven testing.
Opportunity: make CQs a first-class per-block artifact used for both (a) scoping/generating the ontology and (b) measuring coverage — where coverage = % of the block's competency questions the agent can answer. One artifact = design tool + eval suite + the concrete definition of the "context coverage" thesis.
Background / references (so the lineage isn't lost)
Grüninger & Fox (1995) — origin of competency questions; CQs as both requirements and tests.
METHONTOLOGY, On-To-Knowledge, NeOn methodology — established ontology engineering methodologies that formalize CQ-driven design.
Ontology learning (Maedche & Staab) — automatic ontology induction from text; the technique behind "AI generates the ontology when the user can't define one."
Competency-question-driven ontology learning — the hybrid: seed → competency questions → ontology (more rigorous + publishable than seed → ontology).
CQs as ontology unit tests — using the CQ set as a regression/coverage test.
LLM-generated competency questions — emerging (2023-24); VERIFY current state before citing.
Acceptance Criteria
Decide whether CQs become a first-class per-block artifact (e.g. competency-questions.yaml)
Activity