feat: RAG Context Kind — NAIL as intermediate format for retrieval-augmented generation

[watari-ai]

Background

NAIL currently defines three top-level kind values: fn, module, tool_spec.
A new use case has emerged: using NAIL as the intermediate format between a RAG database and AI agents (Devin, Claude Code, Codex, etc.).

The problem with current RAG pipelines

RAG DB → [raw natural language chunks] → AI agent

Natural language chunks are ambiguous — different AI agents interpret the same text differently.
NAIL can solve this by providing a machine-unambiguous intermediate representation.

Proposed architecture

RAG DB → [retrieval] → NAIL context chunks → [injection] → AI agent

---

What NAIL is missing for RAG use

1. context kind (new top-level kind)

A dedicated kind for retrieved knowledge chunks:

{
  "nail": "1.0",
  "kind": "context",
  "id": "auth_flow_rules",
  "source": {
    "type": "rag",
    "db": "internal-docs-v2",
    "chunk_id": "auth-003",
    "retrieved_at": "2026-03-02T10:00:00Z"
  },
  "valid_until": "2026-06-01T00:00:00Z",
  "confidence": 0.92,
  "facts": [
    {"key": "jwt_expiry_action", "value": "refresh_token()", "type": "rule"},
    {"key": "allowed_roles", "value": ["admin", "editor"], "type": "constraint"}
  ],
  "relations": ["auth_error_handling", "user_model"]
}

2. Provenance tracking (source field)

Retrieved knowledge must carry its origin:

• db: which RAG database
• chunk_id: original document chunk identifier
• retrieved_at: timestamp of retrieval

This allows AI agents to reason about knowledge freshness and cite sources.

3. Temporal validity (valid_until field)

Knowledge expires. A rule valid today may be outdated next month.
The valid_until field lets agents reject stale context without re-querying.

4. Confidence score (confidence field)

RAG retrieval has similarity scores. Propagating confidence to the AI lets it weight facts appropriately (e.g., skip facts below 0.7 threshold).

5. Cross-chunk relations (relations field)

Retrieved chunks often depend on each other.
Explicit relations lets the AI request related chunks without another full retrieval pass.

6. Semantic keyword hints (keywords field)

For embedding-based systems, optional keywords help routing:

• Which chunks to retrieve for a given query
• How to index NAIL context documents in a vector store

---

Why not use existing kinds?

Option	Problem
fn	Expresses computation, not declarative knowledge
module	Container for code, not retrieved facts
tool_spec	Describes tool interface, not domain knowledge

---

Compatibility

This is a new optional kind — fully additive. Existing NAIL documents are unaffected.
The frozen L0–L3 core is not modified.

---

Proposed deliverables

• Design spec: designs/v1.1/rag-context-kind.md
• JSON Schema extension for context kind
• L0 checker support for context kind validation
• Example: examples/rag/auth_flow_context.nail
• Example: examples/rag/transpiler_sketch.py — natural language → NAIL context converter sketch

---

Related

• Multi-layer LLM interface contracts — locality declarations and cross-layer delegation #110 Multi-layer LLM interface contracts
• Closes part of the "AI-to-AI knowledge transfer" use case space

[watari-ai]

Implementation notes (Watari, 2026-03-06)

On confidence field semantics

The confidence: 0.92 field is promising, but we should define whether it represents:

1. Retrieval similarity score (cosine distance from query) — set by RAG system
2. Factual confidence (how reliable this fact is) — requires human/model annotation

Recommend: split into two optional fields:

"retrieval_score": 0.92,
"fact_confidence": "high"

This avoids conflating retrieval quality with knowledge reliability.

---

On valid_until enforcement

Who enforces staleness? Options:

• Agent-side: agent rejects context if valid_until < now — simplest
• NAIL validator: L0 checker emits a warning (not error) for expired context

Recommend: agent-side enforcement (validators shouldn't need system time access), with a lint warning in nail check.

---

Minimal first PR scope

To avoid scope creep, suggest shipping #111 in two phases:

Phase A (can ship now):

- Design spec: designs/v1.1/rag-context-kind.md
- JSON Schema for context kind
- example: examples/rag/auth_flow_context.nail

Phase B (after community feedback):

- L0 checker support
- transpiler_sketch.py
- cross-chunk relations loader

Phase A gives us something to show without requiring checker changes.

---

Relation to #110

context kind is the payload that multi-layer contracts (#110) would route and scope. They compose well: #111 defines what knowledge looks like, #110 defines who can access it. Recommend implementing #111 first.