Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions .github/workflows/opencode-sidecar.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
name: OpenCode sidecar

# Deploy/CI sidecar only. Does not run on pull_request so app tests and the
# required Security workflow stay secret-free (see tests.yml / security.yml).
on:
workflow_dispatch:
workflow_call:
secrets:
NVIDIA_NIM_API_KEY:
required: false
NVIDIA_NIM_API_KEY_SUB:
required: false
OPENAI_API_KEY:
required: false
OPENROUTER_API_KEY:
required: false
BYTEZ_API_KEY:
required: false
push:
branches: [main]

permissions:
contents: read

concurrency:
group: opencode-sidecar-${{ github.ref }}
cancel-in-progress: true

jobs:
seed_and_serve:
name: Seed KV and serve loopback sidecar
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # actions/checkout@v7
with:
persist-credentials: false

- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # actions/setup-python@v6
with:
python-version: "3.12"

- name: Install package
run: |
python -m pip install --require-hashes -r requirements.lock
python -m pip install --no-deps -e .

- name: Seed org credentials into this-job report
env:
NVIDIA_NIM_API_KEY: ${{ secrets.NVIDIA_NIM_API_KEY }}
NVIDIA_NIM_API_KEY_SUB: ${{ secrets.NVIDIA_NIM_API_KEY_SUB }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
BYTEZ_API_KEY: ${{ secrets.BYTEZ_API_KEY }}
run: |
python -m contextual_orchestrator seed-provider-catalog \
--from-env --skip-missing \
--agents examples/agents.production.json \
--agents-db "$RUNNER_TEMP/agents.db" \
--discover-models | tee "$RUNNER_TEMP/seed-report.json"

- name: Serve loopback OpenAI-compatible API and smoke curl
env:
NVIDIA_NIM_API_KEY: ${{ secrets.NVIDIA_NIM_API_KEY }}
NVIDIA_NIM_API_KEY_SUB: ${{ secrets.NVIDIA_NIM_API_KEY_SUB }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
BYTEZ_API_KEY: ${{ secrets.BYTEZ_API_KEY }}
run: |
set -euo pipefail
CONTEXTUAL_ORCHESTRATOR_TOKEN="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
export CONTEXTUAL_ORCHESTRATOR_TOKEN
python -m contextual_orchestrator --serve \
--seed-from-env \
--agents examples/agents.production.json \
--agents-db "$RUNNER_TEMP/agents.db" \
--host 127.0.0.1 \
--port 8000 \
--auth-token "$CONTEXTUAL_ORCHESTRATOR_TOKEN" &
server_pid=$!
trap 'kill "$server_pid" 2>/dev/null || true' EXIT
ready=0
for _ in 1 2 3 4 5 6 7 8 9 10; do
if curl -sf http://127.0.0.1:8000/healthz >/dev/null; then
ready=1
break
fi
sleep 1
done
if [ "$ready" -ne 1 ]; then
echo "sidecar did not become healthy"
exit 1
fi
registered="$(python -c 'import json,sys; print(len(json.load(open(sys.argv[1]))["registered_credentials"]))' "$RUNNER_TEMP/seed-report.json")"
if [ "$registered" = "0" ]; then
echo "no provider secrets in this job; skip live chat smoke (fail-closed, no GitHub Models fallback)"
exit 0
fi
curl -sS --fail http://127.0.0.1:8000/v1/chat/completions \
-H "authorization: Bearer $CONTEXTUAL_ORCHESTRATOR_TOKEN" \
-H "content-type: application/json" \
-d '{"model":"contextual-orchestrator","messages":[{"role":"user","content":"Write one sentence."}]}'
29 changes: 18 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,23 +56,30 @@ push or open a PR.
- The reference implementation is xtrmLLMBatchPython's pgcrypto-encrypted
Postgres credential registry (`get_credential(name)`); reuse that pattern (a
DB-backed KV is fine) unless a dedicated KV is adopted.
- **Known deviation to migrate:** this repo currently resolves provider API
keys from env — `ModelClient` reads `os.environ.get(agent.api_key_env)` in
`contextual_orchestrator/orchestrator.py` (and `CONTEXTUAL_ORCHESTRATOR_*`
tokens in `__main__.py`). Move these to KV-backed reads; keep env only as the
bootstrap path that seeds the KV.
- Runtime provider keys resolve through `get_credential` / the KV registry.
Env is bootstrap transport only (`register-credential --from-env` /
`seed-provider-catalog --from-env` / `--seed-from-env` on serve). The five
org Actions secrets are `NVIDIA_NIM_API_KEY`, `NVIDIA_NIM_API_KEY_SUB`,
`OPENAI_API_KEY`, `OPENROUTER_API_KEY`, and `BYTEZ_API_KEY`. A missing secret
skips that upstream; it must not crash the pool.
- ContextualWisdomLab **no longer uses GitHub Models**. Do not add
`COPILOT_GITHUB_TOKEN`, `models.github.ai`, `gpt-5.6-luna`, or `gpt-5.6-terra`.

### This repo: the org LLM gateway

- `contextual-orchestrator` is the org's **LLM-communication hub** — the
OpenAI-compatible front door consumed by **gyeot** and **scopeweave**.
OpenAI-compatible front door consumed by **gyeot**, **scopeweave**,
**OpenCode**, and **Strix**.
- **Direction:** grow it toward a **LiteLLM-class multi-provider gateway**. The
org is open to a **Rust/Python hybrid** to cut overhead.
- Its `ModelClient` currently reads `os.environ.get(agent.api_key_env)` — this
is the KV-principle deviation above. Resolve the API key (including the org
`OPENAI_API_KEY`) from the **KV / credential registry**, not env.
- The **OpenCode review pipeline is separate** and stays on **GitHub Models** —
do not change it.
- `ModelClient` resolves the API key from the **KV / credential registry** via
`get_credential` (including the org `OPENAI_API_KEY` and NIM / OpenRouter /
Bytez keys). Env is never the request-time source.
- OpenCode/Strix should call this process as **one** OpenAI-compatible
provider: `baseURL http://127.0.0.1:8000/v1`, model `contextual-orchestrator`
(see `docs/opencode-sidecar.md`). The org-central review workflow lives in
`ContextualWisdomLab/.github` and should consume this sidecar — do not
reintroduce GitHub Models there from this repo.

### This repo's role in the ecosystem

Expand Down
39 changes: 39 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Changelog

## Unreleased

### Added

- Production agent catalog (`examples/agents.production.json`) for NVIDIA NIM
(primary + secondary Nemotron Super 49B / 120B), OpenAI, OpenRouter, and
Bytez. Capability tags cover coding, review, and reasoning so Fugu route vs
Conductor/TRINITY conduct can pick workers. GitHub Models, Copilot tokens,
`gpt-5.6-luna`, and `gpt-5.6-terra` are rejected.
- `seed-provider-catalog` CLI and `--seed-from-env` serve flag register the five
org Actions secrets (`NVIDIA_NIM_API_KEY`, `NVIDIA_NIM_API_KEY_SUB`,
`OPENAI_API_KEY`, `OPENROUTER_API_KEY`, `BYTEZ_API_KEY`) into the KV. A
missing secret skips that upstream. Optional `GET /v1/models` discovery
appends chat models; providers without a list API keep the static seed
(`docs/doctoring/provider-catalog.md`).
- OpenCode/Strix sidecar contract: loopback `http://127.0.0.1:8000/v1`, model
`contextual-orchestrator` (`.github/workflows/opencode-sidecar.yml`,
`docs/opencode-sidecar.md`). App tests and Security stay secret-free.

### Changed

- Fast-path routing is a cost-performance choose (quality per unit operator
cost), not deterministic keyword scoring and not a walk down the seed JSON.
429 / 5xx / timeout re-runs the same chooser on the remaining healthy pool.
Missing credentials drop that worker from the candidate set. An empty
healthy pool fail-closes (no GitHub Models). Deep `conduct` stays
Conductor-style and still requires a workflow hint.
- Unconfigured remote workers are skipped at select/re-selection time. When every
provider credential is missing, routing raises `NotConfigured` and does not
fall back to GitHub Models.
- Malformed upstream chat.completion bodies raise `ProviderResponseError` so
the gateway failovers or returns a JSON error instead of crashing.
- Catalog `GET /v1/models` discovery uses the same public-HTTPS egress policy
as chat (no private/loopback/redirect leak of the KV Bearer token). HTTP is
opt-in via `allow_insecure_discovery`. Reseed replaces the sqlite agent pool
so stale discovered workers cannot linger. Sidecar CI fails when `/healthz`
never comes up.
8 changes: 5 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

- **Security gate**: every PR to `main` runs the required Security workflow. A failing Trivy or pip-audit job is a real finding — remediate by bumping the dependency and regenerating `requirements.lock`; never weaken, `continue-on-error`, or disable the gate.
- **KV, not env**: runtime config and provider secrets are resolved from the KV credential registry (`get_credential`), never `os.getenv` at request time. Env is only bootstrap transport into the KV (see `docs/kv-credentials.md`).
- **Org role**: this repo is the org's LLM gateway (cost optimizer + sync/batch routing + upstream load balancing, LiteLLM-plus scope), consumed by `gyeot` and `scopeweave`. The OpenCode review pipeline is separate, stays on GitHub Models, and must not be changed.
- **Org role**: this repo is the org's LLM gateway (cost optimizer + sync/batch routing + upstream load balancing, LiteLLM-plus scope), consumed by `gyeot`, `scopeweave`, OpenCode, and Strix. ContextualWisdomLab no longer uses GitHub Models; OpenCode should call `http://127.0.0.1:8000/v1` model `contextual-orchestrator` (see `docs/opencode-sidecar.md`).
- **Research grounding**: substantive feature/process PRs should attach the relevant papers (PDF when redistribution is permissible, otherwise cite + link + summary) under `docs/papers/` with full citations.

This file complements AGENTS.md with commands and architecture; where they differ, AGENTS.md wins.
Expand Down Expand Up @@ -55,6 +55,8 @@ python -m contextual_orchestrator --eval "prompt one" "prompt two"

# Seed a provider credential into the KV at bootstrap
echo "$OPENAI_API_KEY" | python -m contextual_orchestrator register-credential --name OPENAI_API_KEY --value-stdin
python -m contextual_orchestrator seed-provider-catalog --from-env --skip-missing \
--agents examples/agents.production.json --agents-db /tmp/agents.db

# Reproduce the Trivy security gate locally (against the merge result)
trivy --download-db-only
Expand All @@ -80,7 +82,7 @@ A stdlib-Python lab implementing a single OpenAI-compatible API that routes, del
2. `TaskOrchestrator.complete()` in `orchestrator.py` picks one of two paths:
- **Fast path (`route`)**: select a single worker for simple or latency-sensitive requests.
- **Deep path (`conduct`)**: build a natural-language workflow of `thinker → worker → verifier → synthesizer` steps. Each `WorkflowStep` carries an **access list** so a worker sees only the prior outputs deliberately exposed to it.
3. `ModelClient` (infrastructure adapter) executes each step against `mock://` agents (offline, used by tests) or OpenAI-compatible HTTPS providers, with jittered retries for transient errors, failover to the next capability-matched agent, and a per-agent circuit breaker. Provider keys come from the KV via `get_credential`; egress to loopback/private/reserved addresses is blocked.
3. `ModelClient` (infrastructure adapter) executes each step against `mock://` agents (offline, used by tests) or OpenAI-compatible HTTPS providers, with jittered retries for transient errors. Worker selection is a cost-performance choose (quality per unit cost); a 429/5xx/timeout **re-runs that chooser** on the remaining healthy pool rather than walking the seed list. A per-agent circuit breaker excludes a persistently failing provider until it cools down. Provider keys come from the KV via `get_credential`; egress to loopback/private/reserved addresses is blocked.
4. The answer is framed as an OpenAI `chat.completion` (or SSE `chat.completion.chunk` stream). Full orchestration traces are only returned to trusted callers.

### Modules (`contextual_orchestrator/`)
Expand All @@ -91,7 +93,7 @@ A stdlib-Python lab implementing a single OpenAI-compatible API that routes, del
- `credentials.py` / `kv_config.py` — the KV seam: `get_credential`/`register_credential` over pluggable backends (`InMemoryCredentialBackend` default; pgcrypto-encrypted `PostgresCredentialBackend`, selected via `CONTEXTUAL_ORCHESTRATOR_KV_BACKEND`).
- `cost_ledger.py` / `cost_router.py` / `batch_routing.py` / `token_counting.py` — the cost-review + routing hub: prompt-safe usage ledger with seven attribution dimensions, `RoutingPolicy` (sync vs batch from request hints + KV thresholds), and the [pg-llm-batch](https://github.com/ContextualWisdomLab/pg-llm-batch) batch/embeddings backends (a local in-process backend keeps the standalone path working with no external service).
- `api_contract.py` / `conventions.py` — API-shape and naming-rule enforcement helpers.
- `__main__.py` — the single entry point: CLI completion, `--serve`, `--eval`, and the `register-credential` bootstrap subcommand.
- `__main__.py` — the single entry point: CLI completion, `--serve`, `--eval`, `register-credential`, and `seed-provider-catalog` / `--seed-from-env`.

Agent pools are **data, not code**: `examples/agents.mock.json` and `examples/agents.openai.json`. State is in-memory by default; `--state-db PATH` persists runs/audit/analytics to sqlite.

Expand Down
14 changes: 13 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,13 @@ HTTP serving is hardened for local lab use:
- Response caching is off by default. Pass `--cache-ttl SECONDS` to serve identical requests (same messages + mode) from an in-memory TTL+LRU cache and skip the provider calls; `0` disables it.
- `ModelClient.batch_chat(agent, {custom_id: messages})` runs many requests through the provider's Batch API (async, 24h completion window, typically ~50% cheaper) — suited to evaluation/benchmark workloads, not latency-sensitive chat. The mock path answers synchronously.

Production (not mock-only) seed: [examples/agents.production.json](examples/agents.production.json) — NVIDIA NIM primary/secondary (Nemotron Super 49B / 120B), OpenAI, OpenRouter, and Bytez. OpenCode/Strix should call `http://127.0.0.1:8000/v1` with model `contextual-orchestrator` ([docs/opencode-sidecar.md](docs/opencode-sidecar.md)). GitHub Models are not in the catalog.

```bash
python -m contextual_orchestrator seed-provider-catalog --from-env --skip-missing \
--agents examples/agents.production.json --agents-db /tmp/agents.db
```

Use real workers by replacing `mock://` agents with OpenAI-compatible endpoints. Provider secrets are resolved from a KV credential registry via `get_credential`, never from `os.getenv` at request time (see [docs/kv-credentials.md](docs/kv-credentials.md)):

```json
Expand Down Expand Up @@ -115,7 +122,7 @@ One fused orchestration loop:
- Deep path: a natural-language workflow is built with planner, worker, verifier, and synthesizer steps.
- Each step has an access list, so workers see only the prior outputs intentionally exposed to them.
- Agent definitions are data, so provider preference, exclusions, privacy constraints, and mock testing do not require code changes.
- Provider calls are resilient: transient failures (timeouts, 429, 5xx) retry with full-jitter exponential backoff, while caller errors (4xx) fail fast. If an agent still fails, the request fails over to the next capability-matched agent in the pool, and a per-agent circuit breaker skips a persistently failing provider until it cools down. Failover is recorded in the trace (`served_agent_id`, `failover_from`).
- Fast-path selection is a cost-performance choose: one worker from the live pool that maximizes expected quality per unit cost (operator `price_per_million`, TRINITY role tags, measured circuit/latency when present). Seed JSON order and prompt keywords do not pick the winner. Transient failures (timeouts, 429, 5xx) retry with full-jitter exponential backoff on that worker; if it still fails, the **same chooser** runs again on the remaining healthy pool (circuit-open agents excluded). An empty healthy pool fail-closes. Re-selection is recorded in the trace (`served_agent_id`, `failover_from`).

See [docs/architecture.md](docs/architecture.md) for the source-backed analysis.

Expand Down Expand Up @@ -252,6 +259,11 @@ python -m pip install --require-hashes -r requirements.lock
python -m pip install --no-deps -e .
python tests/test_self_check.py
python tests/test_paper_contracts.py
python tests/test_provider_catalog.py
python tests/test_catalog_bootstrap.py
python tests/test_provider_catalog_robustness.py
python tests/test_cost_performance_chooser.py
python tests/test_opencode_sidecar_contract.py
python tests/test_admin_contract.py
python tests/test_conventions.py
python tests/test_api_contract.py
Expand Down
2 changes: 2 additions & 0 deletions conductor/product.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,5 +39,7 @@ Provide one API and one domain model:
- Training a learned coordinator.
- Claiming compatibility with or ownership of any vendor model.
- Adding provider SDKs before stdlib HTTP proves insufficient.
- Reintroducing GitHub Models (`COPILOT_GITHUB_TOKEN`, `gpt-5.6-luna` / `terra`).
OpenCode and Strix call this process as one OpenAI-compatible provider.

See `docs/product_planning.md` for the paper-grounded product plan.
1 change: 1 addition & 0 deletions conductor/tracks.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@
|---|---|---|
| 001-paper-grounded-orchestrator | active | Implement the source-backed orchestration contract with TDD, DDD, and CDD |
| 002-enterprise-design-foundation | active | Add paper-grounded screen design, user stories, REST API, code/DB conventions, and i18n |
| 003-org-provider-catalog | active | Seed NIM/OpenAI/OpenRouter/Bytez into the KV, OpenCode sidecar, cost-performance choose + re-selection |
Loading
Loading