Spec/ml readiness wave

.github/PULL_REQUEST_TEMPLATE.md

@@ -29,5 +29,5 @@
 ## Checklist
 
 - [ ] Specs updated before implementation
-- [ ] `docs/specs/tasks.md` updated
+- [ ] contracts/docs updated when behavior changed
 - [ ] Tests added/updated for behavior changes

.github/workflows/ci.yml

@@ -49,6 +49,30 @@ jobs:
       - name: Unit and integration tests
         run: pytest -q
 
+  ml-extras-smoke:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install ML extras
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install -e .[ml]
+
+      - name: Verify optional ML imports
+        run: |
+          python - <<'PY'
+          import fasttext
+          import sentence_transformers
+          print("ml-extra-imports: ok")
+          PY
+
   postgres-integration:
     runs-on: ubuntu-latest
     services:

.gitignore

@@ -12,3 +12,4 @@ htmlcov/
 pip-wheel-metadata/
 dist/
 build/
+dev_docs/

AGENTS.md

@@ -6,8 +6,8 @@
 - `tests/`: Unit and integration tests (`test_api.py`, lexicon/release workflow tests, Postgres integration tests).
 - `scripts/`: Operational tooling (contract checks, migrations, seed sync, release management).
 - `migrations/`: Ordered SQL migrations (`0001_...sql`, `0002_...sql`).
-- `docs/master.md`: Strategic project blueprint.
-- `docs/specs/`: Spec-first source of truth (RFCs, ADRs, OpenAPI, JSON schemas, task board).
+- `contracts/`: API/spec contract artifacts used by checks and tests.
+- `templates/go-live/`: Go-live bundle template used by readiness validator.
 - `data/lexicon_seed.json`: Local fallback lexicon seed.
 
 ## Build, Test, and Development Commands
@@ -27,7 +27,7 @@
 - Prefer small, focused modules; keep business logic out of route handlers.
 - File names: `snake_case.py`; tests: `test_*.py`.
 - Migration files must be ordered (`000N_descriptive_name.sql`).
-- Keep response contracts deterministic and aligned with `docs/specs/api/openapi.yaml`.
+- Keep response contracts deterministic and aligned with `contracts/api/openapi.yaml`.
 
 ## Testing Guidelines
 
@@ -40,7 +40,7 @@
 
 - This workspace may not include full Git history; use Conventional Commit style (e.g., `feat:`, `fix:`, `docs:`).
 - PRs should include:
-  - linked task ID from `docs/specs/tasks.md`,
+  - linked issue/task ID,
   - spec references (RFC/ADR/OpenAPI/schema),
   - test evidence (`pytest` + contract check),
   - migration notes when schema changes are included.

CONTRIBUTING.md

@@ -1,16 +1,14 @@
 # Contributing to Sentinel
 
-Thank you for contributing. This repository follows a spec-driven workflow: update specs first, then implementation.
+Thank you for contributing.
 
 ## Contribution Flow
 
 1. Open or link an issue with problem statement and scope.
-2. Update relevant specs before code changes:
-   - RFC: `docs/specs/rfcs/`
-   - ADR: `docs/specs/adr/`
-   - API/schema: `docs/specs/api/`, `docs/specs/schemas/`
-3. Implement code and tests.
-4. Update `docs/specs/tasks.md` status in the same PR.
+2. Implement code and tests.
+3. Update contracts/docs when behavior changes:
+   - API: `contracts/api/openapi.yaml`
+   - Schemas: `contracts/schemas/`
 
 ## Local Setup
 
@@ -57,4 +55,4 @@ make test-db
 ## Governance and Conduct
 
 - Follow `CODE_OF_CONDUCT.md` for all interactions.
-- For moderation-outcome changes, review `docs/specs/governance.md`.
+- For moderation-outcome changes, include rationale and risk notes in your PR.

Dockerfile

@@ -6,7 +6,7 @@ COPY pyproject.toml README.md ./
 COPY src ./src
 COPY data ./data
 COPY config ./config
-COPY docs/specs ./docs/specs
+COPY contracts ./contracts
 
 RUN pip install --no-cache-dir .
 

Makefile

@@ -4,7 +4,7 @@ LIMIT ?= 20
 ITERATIONS ?= 300
 WARMUP ?= 30
 P95_BUDGET_MS ?= 150
-BUNDLE_DIR ?= docs/releases/go-live/template
+BUNDLE_DIR ?= templates/go-live
 
 run:
 	python -m uvicorn sentinel_api.main:app --reload

README.md

@@ -1,116 +1,43 @@
 # Sentinel
 
-**Multilingual political-safety infrastructure for the 2027 Kenyan General Election.**
+Sentinel is a multilingual moderation API for election-risk and civic discourse safety.
+It is designed for products that need deterministic moderation decisions with audit evidence, especially in code-switched East African contexts.
 
-Sentinel detects ethnic incitement, hate speech, and election-related disinformation across Kenya's multilingual digital landscape — where Swahili, Sheng, English, Luhya, Kikuyu, Luo, Kalenjin, and other languages mix freely in a single post.
+## Who this is for
 
-Built for newsrooms, digital publishers, civil society platforms, and fact-check partners who need fast, auditable moderation decisions during high-stakes election cycles.
+- Community forums
+- News platforms
+- Civil society reporting tools
+- Fact-check and trust-and-safety teams
 
-## Why Sentinel?
+## What Sentinel returns
 
-Generic moderation tools fail on Kenyan political content. Harmful rhetoric appears in code-switched text, local slang, and coded dog whistles that shift meaning by region, language, and electoral phase. Sentinel is purpose-built for this reality:
+For each text input, Sentinel returns:
 
-- **Code-switching first** — language identification at the span level, not the post level.
-- **Deterministic decisions** — every ALLOW, REVIEW, or BLOCK comes with reason codes, evidence traces, and artifact versions. No black boxes.
-- **Election-aware policy** — moderation posture adapts automatically across campaign, silence, voting, and results periods.
-- **Human-in-the-loop by design** — ambiguity escalates to reviewers. Humans remain accountable.
-- **Governed and auditable** — versioned lexicons, appeals workflows, transparency exports, and tamper-evident audit trails.
+- `action`: `ALLOW`, `REVIEW`, or `BLOCK`
+- `labels` and `reason_codes`
+- `evidence` used for the decision
+- provenance fields (`model_version`, `lexicon_version`, `policy_version`)
 
-## How it works
-
-```
-Input text
-  -> Normalize and route language spans (sw, en, sheng, kik, Luh, luo, kal)
-  -> Fast lexical triggers (Redis, O(1) lookup)
-  -> Semantic similarity (Postgres + pgvector)
-  -> Deterministic policy logic
-  -> Action: ALLOW / REVIEW / BLOCK
-  -> Structured output (labels, evidence, reason codes, versions, latency)
-```
-
-Every response follows a strict contract:
-
-```json
-{
-  "toxicity": 0.87,
-  "labels": ["INCITEMENT_VIOLENCE", "ETHNIC_CONTEMPT"],
-  "action": "BLOCK",
-  "reason_codes": ["R_INCITE_CALL_TO_HARM", "R_ETHNIC_SLUR_MATCH"],
-  "evidence": [
-    {"type": "lexicon", "match": "****", "severity": 3, "lang": "sw"},
-    {"type": "vector_match", "match_id": "lex_10293", "similarity": 0.89}
-  ],
-  "language_spans": [
-    {"start": 0, "end": 12, "lang": "sw"},
-    {"start": 13, "end": 40, "lang": "kik"}
-  ],
-  "model_version": "sentinel-multi-v2",
-  "lexicon_version": "hatelex-v2.1",
-  "policy_version": "policy-2026.10",
-  "latency_ms": 94
-}
-```
-
-## Features
-
-| Capability | Description |
-|---|---|
-| Real-time Moderation API | `/v1/moderate` endpoint with P95 < 150ms target |
-| Multilingual language routing | Span-level detection for Tier-1 and Tier-2 Kenyan languages |
-| Hate-Lex knowledge base | Versioned lexicon of slurs, dog whistles, violence idioms, and disinfo templates |
-| Async monitoring pipeline | Ingests partner signals, clusters emerging threats, generates governed update proposals |
-| Electoral phase modes | Policy posture shifts across pre-campaign, campaign, silence, voting day, and results period |
-| Deployment stages | Graduated rollout: shadow -> advisory -> supervised |
-| Appeals and transparency | Full case reconstruction, state-machine appeals, and privacy-controlled transparency exports |
-| Partner connectors | Fact-check integrations with retry, backoff, and circuit breaker resilience |
-
-## Project status
-
-**Active development** — core implementation phases are complete through I-407. Final go-live hardening (I-408 through I-412) is in progress. See the [task board](docs/specs/tasks.md) for current status.
-
-Not yet in production. Contributions and feedback are welcome.
-
-## Getting started
-
-### Prerequisites
-
-- Python 3.12+
-- Docker and Docker Compose (for full stack with Postgres and Redis)
-
-### Quick start with Docker Compose
-
-```bash
-git clone https://github.com/Thelastpoet/sentinel.git
-cd sentinel
-
-# Set a strong API key (do not use the default in production)
-export SENTINEL_API_KEY='your-strong-api-key-here'
-
-docker compose up --build
-make apply-migrations
-make seed-lexicon
-```
-
-Verify the service is running:
-
-```bash
-curl -sS http://localhost:8000/health
-# {"status":"ok"}
-```
-
-### Local development (without Docker)
+## Quickstart
 
 ```bash
 python -m venv .venv
 source .venv/bin/activate
-pip install --upgrade pip
-pip install -e .[dev,ops]
+python -m pip install --upgrade pip
+python -m pip install -e .[dev,ops]
 
-export SENTINEL_API_KEY='your-strong-api-key-here'
-uvicorn sentinel_api.main:app --reload
+# optional ML extras
+python -m pip install -e .[ml]
+
+docker compose up -d --build
+export SENTINEL_API_KEY='replace-with-strong-key'
+python scripts/apply_migrations.py --database-url postgresql://sentinel:sentinel@localhost:5432/sentinel
+python scripts/sync_lexicon_seed.py --database-url postgresql://sentinel:sentinel@localhost:5432/sentinel --activate-if-none
+uvicorn sentinel_api.main:app --host 0.0.0.0 --port 8000
 ```
 
-### Try a moderation request
+Test a request:
 
 ```bash
 curl -sS -X POST http://localhost:8000/v1/moderate \
@@ -119,94 +46,27 @@ curl -sS -X POST http://localhost:8000/v1/moderate \
   -d '{"text":"They should kill them now."}'
 ```
 
-## Running tests
-
-```bash
-# Unit tests
-python -m pytest -q
-
-# Contract validation
-python scripts/check_contract.py
-
-# Hot-path latency benchmark
-python scripts/benchmark_hot_path.py --iterations 300 --warmup 30 --p95-budget-ms 150
-
-# Tier-2 language pack verification
-python scripts/verify_tier2_wave1.py --registry-path data/langpacks/registry.json --pretty
-```
-
-### Pre-commit hooks
-
-```bash
-pip install pre-commit
-pre-commit install
-pre-commit run --all-files
-```
-
-## Repository layout
-
-```
-src/
-  sentinel_core/       shared models, state machines, policy config
-  sentinel_router/     span-level language identification
-  sentinel_lexicon/    lexicon matching, hot triggers, vector search
-  sentinel_langpack/   language pack registry and normalization
-  sentinel_api/        FastAPI application, auth, workers, admin endpoints
-scripts/               operational and verification tooling
-config/                policy configuration files
-data/                  lexicon seeds, eval sets, language-pack artifacts
-migrations/            Postgres schema migrations (Alembic)
-infra/                 database init scripts
-docs/                  master plan, specs, governance, operations
-tests/                 unit and integration tests
-```
-
-## Architecture and specs
-
-Sentinel follows a **spec-first workflow**: specs are written and approved before implementation.
-
-| Document | Purpose |
-|---|---|
-| [Master plan](docs/master.md) | Product direction and system blueprint |
-| [Task board](docs/specs/tasks.md) | Implementation progress tracker |
-| [OpenAPI spec](docs/specs/api/openapi.yaml) | Public API contract |
-| [JSON schemas](docs/specs/schemas/) | Request/response payload schemas |
-| [RFCs](docs/specs/rfcs/) | Feature-level behavioral specifications |
-| [ADRs](docs/specs/adr/) | Architecture decision records |
-
-## Tech stack
-
-| Layer | Technology |
-|---|---|
-| Language | Python 3.12+ |
-| API framework | FastAPI |
-| Database | PostgreSQL 16 + pgvector |
-| Cache | Redis 7 |
-| Packaging | uv workspace |
-| Testing | pytest |
-| Container runtime | Docker Compose (dev), Kubernetes (production target) |
-
-## Contributing
-
-Contributions are welcome, especially for:
-
-- **Language packs** — normalization rules, lexicon entries, and evaluation sets for Kenyan languages
-- **Lexicon proposals** — new terms, dog whistles, or coded rhetoric patterns
-- **Bug reports and fixes** — especially around false positives on legitimate political speech
-- **Documentation** — translations, onboarding guides, and operational runbooks
-
-Please read [CONTRIBUTING.md](CONTRIBUTING.md) before submitting a pull request. All interactions are governed by the [Code of Conduct](CODE_OF_CONDUCT.md).
+## Integration model
 
-For changes that affect moderation outcomes, review the [governance spec](docs/specs/governance.md). These require at least two maintainer approvals.
+Your backend calls Sentinel before publish:
 
-## Security
+1. Send user text to `POST /v1/moderate`
+2. Apply action:
+   - `ALLOW` -> publish
+   - `REVIEW` -> moderation queue
+   - `BLOCK` -> reject
+3. Store decision metadata for audit and appeals
 
-This project handles election-safety workflows. Deployments, credentials, and incident procedures should be treated as high-risk operational assets.
+## Documentation
 
-- **Never commit secrets or production credentials.** Use environment variables.
-- **Follow staged rollout controls** documented in the [go-live readiness gate](docs/specs/phase4/i408-go-live-readiness-gate.md).
-- **Report security vulnerabilities** responsibly by emailing the maintainers directly. Do not open public issues for security bugs.
+- [Docs index](docs/README.md)
+- [Quickstart](docs/quickstart.md)
+- [Integration Guide](docs/integration-guide.md)
+- [Deployment Guide](docs/deployment.md)
+- [API Reference](docs/api-reference.md)
+- [Security Notes](docs/security.md)
+- [FAQ](docs/faq.md)
 
 ## License
 
-Apache-2.0. See [LICENSE](LICENSE) for details.
+Apache-2.0. See [LICENSE](LICENSE).

alembic/versions/0012_model_artifact_lifecycle.py

@@ -0,0 +1,25 @@
+from __future__ import annotations
+
+from pathlib import Path
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "s0012"
+down_revision = "s0011"
+branch_labels = None
+depends_on = None
+
+
+def _read_sql(filename: str) -> str:
+    root = Path(__file__).resolve().parents[2]
+    return (root / "migrations" / filename).read_text(encoding="utf-8")
+
+
+def upgrade() -> None:
+    op.execute(sa.text(_read_sql("0012_model_artifact_lifecycle.sql")))
+
+
+def downgrade() -> None:
+    raise NotImplementedError("Irreversible raw SQL migration")