Skip to content

README.md

Latest commit

 

History

History
165 lines (125 loc) · 8.03 KB

README.md

File metadata and controls

165 lines (125 loc) · 8.03 KB
Raw

agent-learning-lab

Hands-on workbook for the measurement-first AI coding agent curriculum. GitHub Copilot · Claude Code · OpenAI Codex.

This repo is where you play. It holds the labs, the verified reading, the predictions you make before each run, and the exit gate you have to pass before moving on. It does not hold the instrument — that lives next door.

Repo Role
agent-learning-lab (here) Curriculum, labs, verified sources, findings, exit gates
agent-observatory The instrument — runner, evaluator, analyzer, OTel → Tempo → Grafana
agent-observatory-benchmarks BE-001, BE-002, acceptance suites, evaluators

The rule

A feature is not learned when the file exists. It is learned when you can explain what changed in the agent, prove that it happened, measure its effect, and identify its failure mode.

Every phase runs the same loop. Skipping PREDICT is the most common way to learn nothing: if you did not write down what you expected, any result feels like it confirms something.

LEARN THE MECHANISM → READ VERIFIED DOCS → PREDICT → BUILD THE SMALLEST VERSION
        → RUN A CONTROLLED TASK → OBSERVE → RUN DETERMINISTIC EVALS
        → DELIBERATELY BREAK IT → EXPLAIN THE FAILURE → COMPARE WITH BASELINE
        → COMMIT THE ARTIFACT → PASS EXIT GATE

How to use this repo

  1. Open the phase folder. Read its README.md top to bottom before touching a tool.
  2. Work the Verified reading list. Links are checked — see SOURCES.md.
  3. Write your predictions into a copy of templates/experiment.md before running anything.
  4. Run the labs. Record with templates/run-record.yaml.
  5. Tick the exit gate. If you cannot answer a gate question out loud, you are not done.
  6. Each phase has a tracking issue. Close it only when the gate passes.
./tools/check-links.sh     # re-verify every source before a cohort

Start here
LEARNING-PATH.md The single plan. Merges the four competing roadmaps (this curriculum + the three in businesscase/) into two interleaved tracks
build/ Track B — build a custom backend agent. Thirteen steps, B1→B13, plain prompt to governed self-learning
GUARDRAILS.md The layer model every phase points back to. Read before Phase 5A
SOURCES.md Every source, verified, with what to take from each

Two tracks. Understand (the phases below) teaches why a primitive behaves as it does. Build (build/) applies it to a real agent. They interleave — learn the primitive, then build with it, so no build step is a guess.

Progress

Phase Topic Layer Status
0A Agent mechanics + governance all three ⬜ skipped — should have been first
0B Observatory + evaluation baseline L0 ✅ Built in agent-observatory
1 Custom instructions L3 ✅ Result — see phase
2 Prompt files / reusable workflows L3 ⬜
3 Agent Skills L3 ⬜
4A Custom agents + permissions L2 ⬜
4B 🆕 Agent orchestration + multi-layer design L3/L1 ⬜
5A 🆕 Guardrails: hooks, policies, enforcement L2 ⬜
5B 🆕 Verification, bounded self-healing, completion L2 ⬜
6A Code intelligence: LSP → MCP L2 ⬜
6B 🆕 Knowledge retrieval: router → hybrid → vector L3 ⬜
7 Plugins + distribution L2 ⬜
8 Unattended agents (gh-aw) L1 ⬜
9 Memory & governed self-learning L3 ⬜
10 Production observability + impact L0 ⬜

Five of fifteen phases operate at Layer 3 only — instructions, prompt files, skills, knowledge retrieval, memory. That is where most customization effort goes, and none of it can stop anything. See GUARDRAILS.md.

Zero experiments have produced a defensible result yet. That is not a failure of the project; it is what the first honest pass through a measurement problem looks like. See Findings.

Findings so far

These are ours, earned on real runs. They are not in CURRICULUM.md and they are the most valuable thing in this repo.

Seven harness bugs made the instrument measure something other than the agent

# Bug Direction
1 AGENTS.md, installed by the runner, counted as an agent scope violation pessimistic
2 An exhausted Copilot quota recorded as F03, incorrect code pessimistic
3 A background daemon's .memdb/ files counted as unrelated production changes pessimistic
4 An acceptance suite's compiled classes in gitignored target/ ran as "the existing tests" pessimistic
5 The answer key shipped in the agent's worktree flattering
6 The answer key stayed in git history after 5 was "fixed" flattering
7 A permission block recorded as incorrect code — 7/10 sonnet runs changed no file pessimistic

The lesson is in the direction column. The five pessimistic bugs were all caught quickly, because a bad result invites investigation. The two flattering ones survived every review the project had. Nobody investigates a pass.

An explained symptom stops being evidence

A 35% drop in tool calls was written up as proof of the worktree leak. It was still there after the leak was fixed — and nobody looked, because the drop already had a name. The real cause was mundane: the tree got smaller, so exploring it took fewer calls.

A fix is not confirmed by the story that motivated it. It is confirmed by the symptom disappearing, or by an assertion that fails loudly when the leak returns.

The file being present is not the treatment being loaded

Phase 1 ran 10 + 10 clean runs comparing "with AGENTS.md" against "without", and concluded INCONCLUSIVE. Claude Code reads CLAUDE.md, not AGENTS.md. The treatment arm probably never received its treatment. Twenty runs, roughly $4, measuring a file sitting on disk.

Before an experiment, assert that the independent variable reached the agent. Not that you wrote it — that it arrived.

Alias ≠ model, and environment is a variable

Runs invoked as --model haiku loaded ~21 hooks, 2 plugins and 3–4 MCP connections from the local user environment, varying run to run, while the protocol claimed only the model varied. Pin exact model IDs. Use --setting-sources project. Fingerprint what actually loaded.

Local environment

Ports overridden in agent-observatory/infra/.env (gitignored — a fresh clone will not have these):

GRAFANA_PORT=3001    API_PORT=8081    WEB_PORT=5174
  • Copilot quota exhausted 2026-08-08. Use Claude Code. Cheapest Copilot model is gpt-5.4-mini (0 premium requests, verified by probing — the CLI will not list models).
  • BE-002 on Claude/haiku, clean tree: median $0.190, 82–167 s. Budget **$4.10** per 10 + 10 experiment.
  • gh issue view fails here with a Projects-classic GraphQL error. Use gh api repos/OWNER/REPO/issues/N --jq '.title, .body'.

Layout

phases/          one folder per curriculum phase — read the README first
templates/       experiment pre-registration, run record, lab notes
tools/           check-links.sh
SOURCES.md       every source, verified, with what to take from each
CURRICULUM.md    the source curriculum, unmodified