Skip to content

feat(contract): add API schema drift detection#401

Open
sorlen008 wants to merge 2 commits intojackwener:mainfrom
sorlen008:feat/contract-testing
Open

feat(contract): add API schema drift detection#401
sorlen008 wants to merge 2 commits intojackwener:mainfrom
sorlen008:feat/contract-testing

Conversation

@sorlen008
Copy link
Copy Markdown
Contributor

Closes #50

What this adds

A contract subcommand that detects when upstream APIs change their response shape — the #1 cause of silent adapter breakage.

Commands

  • opencli contract snapshot <site> <command> [args...] — run a command and save its response schema as a baseline
  • opencli contract check <site> <command> [args...] — run the command again and diff against the baseline
  • opencli contract list — list all saved baselines

How it works

  1. Schema capture — recursively traverses JSON responses and extracts structure (keys, types, nested objects, array element types). Array schemas merge all elements into a union.
  2. Schema diff — compares baseline vs current: fields added (+), removed (-), type changed (~). Recurses into nested objects and array items. Paths use dot notation (data.items[].slug).
  3. Storage — baselines saved at ~/.opencli/contracts/<site>/<command>.json
  4. Exit codescheck exits with code 1 if drift is detected, making it CI-friendly.

Example

# Save baseline
opencli contract snapshot hackernews top

# Later, check for drift
opencli contract check hackernews top
# ✓ No schema drift detected

# If the API changed:
# ~ data[].score: number → string
# - data[].by
# + data[].author
# ✗ 3 schema differences detected

Files

  • src/contract.ts (new) — schema capture, diff engine, storage, formatting
  • src/contract.test.ts (new) — 25 unit tests covering all core functions
  • src/cli.ts — wired up contract subcommand group

Testing

  • npx tsc --noEmit — zero type errors
  • npm test — all 25 new tests pass. 3 pre-existing failures unrelated.

@sorlen008
Copy link
Copy Markdown
Contributor Author

UAT Results (Windows 11)

Test Result
Unit tests (25/25) PASS
Type check (tsc --noEmit) PASS
Schema capture + diff (manual) PASS — correctly detects field removal, addition, type changes
Contract storage (save/load round-trip) PASS
CLI wiring (contract list, --help) PASS

All 5 tests pass. No bugs found — clean implementation.

@Astro-Han
Copy link
Copy Markdown
Contributor

Note: PR #338 also addresses issue #50 with a different approach — automated CI daily checks instead of manual CLI commands. The two could coexist (CI monitoring + manual debugging tool), but the maintainer may want to decide which direction to prioritize before merging either.

@sorlen008
Copy link
Copy Markdown
Contributor Author

Good callout @Astro-Han — just looked at #338. The two approaches are complementary:

They solve different sides of the same problem. #338 is the alarm bell, #401 is the diagnostic tool. Happy to defer to the maintainer on which to land first, or both if they see value in the combo.

@sorlen008 sorlen008 force-pushed the feat/contract-testing branch 2 times, most recently from c0fe73b to 0619fc5 Compare March 31, 2026 06:16
@sorlen008 sorlen008 force-pushed the feat/contract-testing branch from 0619fc5 to e7cb24f Compare April 7, 2026 06:33
sorlen008 and others added 2 commits April 9, 2026 06:42
Add contract testing module that captures JSON response schemas,
compares them against saved baselines, and reports structural drift
(fields added, removed, or type-changed).

New CLI subcommands:
- `opencli contract snapshot <site> <command>` — save baseline schema
- `opencli contract check <site> <command>` — diff against baseline
- `opencli contract list` — list saved contracts

Schema storage at ~/.opencli/contracts/<site>/<command>.json.
Includes 25 unit tests covering schema capture, diffing, formatting,
and end-to-end drift detection.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@sorlen008 sorlen008 force-pushed the feat/contract-testing branch from e7cb24f to 29b070a Compare April 9, 2026 06:43
@hiSandog
Copy link
Copy Markdown
Contributor

hiSandog commented Apr 9, 2026

One functional gap I noticed in the CLI wiring: contract snapshot/check reparses [args...] manually as --key value pairs, but OpenCLIs command model is broader than that.

A lot of commands in this repo rely on positional args and valueless booleans from cmd.args metadata, and the normal path already handles that in src/commanderAdapter.ts. For example, commands like gemini ask <prompt> / yollomi generate <prompt> need a positional prompt, and flags like --no-download are often used as presence-only booleans. With the current loop in src/cli.ts, the positional value gets dropped entirely and a bare boolean flag is ignored unless it has an explicit following value.

That means opencli contract snapshot gemini ask hello would never forward hello, and opencli contract snapshot yollomi generate cat --no-download would silently lose the boolean override. I think this subcommand should reuse cmd.args for parsing/normalization instead of treating everything as ad hoc --key value pairs, otherwise a lot of real adapters wont be snapshot-able.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: daily contract check for API schema drift with test accounts

3 participants