Skip to content

Add fuzz harnesses for grammar parsers and untrusted-input loaders (tool_id, path, catalog/graph deserialization) #527

Description

@dgenio

Summary

Add coverage-guided fuzz harnesses for the components that parse untrusted input: the tool_id grammar, the tool_browse path grammar, catalog loaders (JSON/YAML/MCP snapshot shapes), graph deserialization, and serde round-trips — strengthening robustness against malformed and adversarial inputs.

Priority: P2 · Confidence: Medium

Why this matters

The gateway accepts model- and upstream-controlled strings into these parsers at runtime (docs/gateway_spec.md §1 and §3 define the grammars precisely because they sit on the trust boundary). Hand-written tests cover expected shapes; fuzzing finds the unexpected ones — pathological nesting, surrogate garbage, boundary lengths — before users do. This complements property-based testing (#440, structured invariants) and static scanning (#443) with input-space exploration.

Current evidence

External context

atheris (libFuzzer for Python) is the standard Python fuzzing engine, and OSS-Fuzz accepts Python projects — an optional later step. Harnesses are small: a target function consuming bytes/str plus a seed corpus.

Proposed implementation

  1. Add fuzz/ with one harness per target: fuzz_tool_id.py, fuzz_browse_path.py, fuzz_keep_path.py, fuzz_catalog_load.py, fuzz_graph_load.py, fuzz_serde_roundtrip.py. Each defines the property: parser either succeeds or raises a ContextWeaverError subclass — never RecursionError, MemoryError, hangs, or non-hierarchy exceptions; round-trippers additionally assert parse(format(x)) == x where applicable.
  2. Seed corpora from existing fixtures (examples/sample_catalog.json, gateway real_catalogs/, tests/fixtures/).
  3. Keep atheris in a new dev-only extra or a fuzz/requirements.txt (not core deps); add a short-budget smoke run (e.g., 60s per target) as a non-gating CI job or a scheduled weekly job alongside deps-latest-weekly.yml.
  4. Triage findings into typed-exception fixes (likely candidates: depth limits in path/graph parsing — coordinate with Harden validation of untrusted tool schemas: well-formedness checks, depth and size limits, validator caching #484's size/depth limits).

AI-agent execution notes

  • Inspect first: routing/tool_id.py, routing/path.py, summarize/structured.py, routing/graph_io.py, serde.py, docs/gateway_spec.md §1/§3.
  • Run: each harness locally for a few minutes before committing; record and minimize any crashers into regular pytest regression cases under tests/.
  • Edge cases the harnesses should reach: deeply nested JSON, huge strings, NUL bytes, invalid UTF-8 (bytes mode), boundary-length names per the §2.1 caps.
  • Preserve: parser semantics — fixes should reject more gracefully, not accept more.
  • Do not gate merges on long fuzz runs; CI smoke budget only.

Acceptance criteria

  • Harnesses exist for all six targets with seed corpora and a documented invocation (make fuzz-smoke or doc'd commands).
  • Any crashes found are fixed with typed exceptions and pinned by pytest regression tests.
  • A scheduled or non-gating CI job runs the smoke budget.

Test plan

Per-harness smoke runs in CI; regression tests for every minimized crasher; full make ci unaffected.

Documentation plan

A short fuzz/README.md (how to run, how to add a target); note in SECURITY.md that fuzzing is part of the testing posture; CHANGELOG.

Migration and compatibility notes

Not expected to require migration. If fixes tighten parser behavior (e.g., new depth limits), each gets its own CHANGELOG note.

Risks and tradeoffs

atheris has platform/Python-version constraints (pin the fuzz job to one matrix cell); fuzzing CPU time costs CI minutes (bounded smoke budget). Alternative: rely solely on Hypothesis (#440) — good for invariants, weaker at coverage-guided exploration; the two are complementary.

Suggested labels

security, testing, area/routing, area/gateway

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/routingRouting engine: catalog, graph, router, cardscomplexity:complexpriority: mediumImportant, clear value, not blockingsecuritySecurity-related: data leakage, access control, redactiontesting

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions