Skip to content

Bugs, Coverage, and SDK Wave 1 Part 2: Python servers: extensive unit tests + per-file 90% coverage gate #4855

Description

@cliffhall

Part of the 2026-07-28 Spec Refactor tracker #4857. This is the first step, and has no dependencies. It can run in parallel with Part 1 (#4854).

Goal

Before any SDK or spec changes, build an extensive pytest suite that pins down the current behavior of every Python server (fetch, git, time) on mcp 1.x. The suite must pass the coverage quality gate from #4474.

This suite is the regression net for Part 4 (#4851). The SDK v2 migration is supposed to be transparent on the wire, and there is no codemod for Python, so the test suite is the only safety net for a port done by hand. Existing tests may be deleted where the new suite covers them better.

Test design

  • Test through the protocol, in-process.
    • Connect an mcp ClientSession to each server over in-memory streams, and assert on the wire-level results: tool lists, call results and errors.
    • Refactor server construction so it can run in-process, if it can't already. Keep at most a thin smoke test that runs the uvx/console-script entry point as a subprocess.
  • Stay migration-proof.
    • Keep assertions independent of SDK internals.
    • v2 renames type fields to snake_case (e.g. inputSchema → input_schema) and moves types to mcp_types. Where practical, compare against the wire JSON (e.g. model_dump(by_alias=True, mode="json")) rather than Python attribute names, so Part 4 doesn't have to rewrite assertions.
    • Any assertion that has to change during Part 4 must be justified in that PR.
  • Coverage by server:
  • Use pytest-asyncio consistently. Standardize the test directory name: time currently uses test/, the others use tests/.

Quality gate (from #4474, modeled on the inspector's v2/main AGENTS.md)

  • Threshold: ≥ 90 per file on lines, statements, functions and branches, using the closest equivalent coverage.py can measure. Enable branch coverage.
  • Per-file enforcement: add pytest-cov/coverage.py. coverage.py's fail_under is global, not per-file, so it isn't enough on its own. Implement per-file enforcement, e.g. coverage json plus a small check script with the same semantics as the TS gate.
  • Unreachable code: don't lower the gate for it. Mark it with a justified inline # pragma: no cover # <reason> instead.
  • Separate coverage command: coverage is its own documented command per server, not part of the fast uv run pytest loop. CI posture (amended): CI enforces the per-file gate in a parallel coverage job, matching Part 1 (Bugs, Coverage, and SDK Wave 1 Part 1: TypeScript servers: extensive unit tests + per-file 90% coverage gate #4854) as amended. Maintainers decided this on PR docs: agentic software factory inception (docs/agent-guidance-inception.md) #4861. This issue wires it in (changed from the earlier plan, which left the wiring to Add local:gate, pre-push-gate skill #4871): besides the command and the per-file check script, add the parallel coverage job to python.yml and a Python coverage stage to local:gate, and record the stage in docs/quality-gate.md and the pre-push-gate skill. Add local:gate, pre-push-gate skill #4871 builds local:gate first, without coverage, so that it does not have to wait for this suite.

Acceptance criteria

  • Every Python server passes the per-file 90% gate, enforced by config and script rather than convention.
  • A PR that drops any Python file below 90 fails CI, and npm run local:gate fails on it too.
  • Every pragma carries a reason.
  • pyright and ruff are clean.
  • The suite passes on current main (mcp 1.x) in CI.
  • Removed tests are listed in the PR with the reason.

Carried over from #4474, which this issue supersedes

Activity

  1. added this to the v2.0.0 milestone on Sep 26, 2026
  2. self-assigned this
    on Sep 26, 2026
  3. changed the title [-]2026-07-28 Spec Refactor Part 5: Extensive new-spec unit tests for TypeScript servers[/-] [+]2026-07-28 Spec Refactor Part 6: Extensive new-spec unit tests for Python servers[/+] on Sep 26, 2026
  4. changed the title [-]2026-07-28 Spec Refactor Part 6: Extensive new-spec unit tests for Python servers[/-] [+]2026-07-28 Spec Refactor Part 2: Python servers: extensive unit tests + per-file 90% coverage gate[/+] on Sep 26, 2026
  5. cliffhall commented on Sep 28, 2026

    @cliffhall
    MemberAuthor

    Amended: the CI posture is now "CI enforces the per-file coverage gate in a parallel job", replacing "coverage stays local". Maintainers decided this on PR #4861, matching the Inspector's CI (inspector#2159). Wiring it into CI and local:gate is #4871; this issue still provides the coverage command.

  6. olaservo commented on Sep 29, 2026

    @olaservo
    Member

    Here is a coverage baseline for the three Python servers to size this work, from main at f46d957 on 2026-09-28. Every file in every server is below the 90% gate. The gap in each server.py is serve(), and the pure functions underneath are well tested. serve() can be driven in-process today with no refactor. The git suite fails on Windows, and its fixture fix should go first.

    Baseline

    Server Tests passing server.py lines / funcs / branches Uncovered in serve()
    fetch 20 52.3 / 50.0 / 50.0 193-288
    git 47 (40 error at teardown on Windows) 71.7 / 75.0 / 58.5 442-581 plus the dead list_repos
    time 38 69.7 / 62.5 / 42.9 124-131, 182-220
    Per-file tables (uv run --with pytest-cov pytest --cov=src --cov-branch, Python 3.10 and 3.11, coverage.py per-function regions for Funcs)

    Bold is below the gate.

    fetch

    File Lines Funcs Branches
    server.py 52.3 50.0 50.0
    __init__.py 25.0 50.0 50.0
    __main__.py 0 0 100

    git

    File Lines Funcs Branches
    server.py 71.7 75.0 58.5
    __init__.py 52.6 50.0 16.7
    __main__.py 0 0 100

    time

    File Lines Funcs Branches
    server.py 69.7 62.5 42.9
    __init__.py 30.0 50.0 50.0
    __main__.py 0 0 100

    git's 442-581 is list_tools and call_tool. The well-tested pure functions are fetch_url, the git_* helpers with their flag-injection guards, and TimeServer.

    Fix first: git fixture on Windows

    test_repository calls shutil.rmtree while GitPython's git cat-file processes still have their cwd inside the repo. 40 of 47 tests then error at teardown with PermissionError: [WinError 32]. Calling test_repo.close() before the rmtree fixes it (verified). This also unblocks #1149.

    Running serve() in-process

    Patch mcp_server_<name>.server.stdio_server with an asynccontextmanager that yields the server side of mcp.shared.memory.create_client_server_memory_streams(). Then run serve(...) in an anyio task group and open ClientSession on the client side. This works for all three servers today. The create_server(...) -> Server split this issue asks for can follow without changing those tests.

    More on the in-process route
    • The git suite already patches stdio_server in test_serve_run_does_not_raise_exceptions.
    • create_server(...) is the cleaner shape for create_connected_server_and_client_session.
    • The low-level call_tool wrapper validates arguments against inputSchema before the handler runs and returns isError "Input validation error".
    • It also folds any handler exception into an isError result.
    • get_prompt lets McpError through as a JSON-RPC error.

    What the new suites should pin

    All verified against the source on main, most by driving serve() in-process.

    fetch

    fetch details

    git

    git details

    time

    time details
    • Error branches: timezone: "" and a bad timezone or time string. The SDK catches an absent key.
    • convert_time's missing-key branch at 197-201 is unreachable through the protocol.

    all three

    Not gaps after all

    The scan is scripted and can be rerun against v2/main as slices land, with per-file deltas against this baseline.


    Scan and verification done with help from Claude Code. The coverage numbers come from a real run; the per-issue verdicts were checked against the source but are a starting point, not a review.

  7. 4 remaining items

  8. changed the title [-]2026-07-28 Spec Refactor Part 2: Python servers: extensive unit tests + per-file 90% coverage gate[/-] [+]2026-07-28 Spec Refactor Wave 1 Part 2: Python servers: extensive unit tests + per-file 90% coverage gate[/+] on Oct 5, 2026
  9. changed the title [-]2026-07-28 Spec Refactor Wave 1 Part 2: Python servers: extensive unit tests + per-file 90% coverage gate[/-] [+]Bugs, Coverage, and SDK Wave 1 Part 2: Python servers: extensive unit tests + per-file 90% coverage gate[/+] on Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions