Conformance Testing #684
Description
Rust SDK Conformance
Doing some work for https://github.com/modelcontextprotocol/conformance/blob/main/SDK_INTEGRATION.md so we can tier ourselves as an SDK and evaluate
My initial conformance server and client are here: https://github.com/modelcontextprotocol/rust-sdk/tree/alexhancock/conformance and below is some data from the first run
Server Conformance Tests
Command:
cd ~/Development/conformance
npx tsx src/index.ts server --url http://localhost:8001/mcp --suite active
Output:
=== SUMMARY ===
✓ server-initialize: 1 passed, 0 failed
✓ logging-set-level: 1 passed, 0 failed
✓ ping: 1 passed, 0 failed
✓ completion-complete: 1 passed, 0 failed
✓ tools-list: 1 passed, 0 failed
✓ tools-call-simple-text: 1 passed, 0 failed
✓ tools-call-image: 1 passed, 0 failed
✓ tools-call-audio: 1 passed, 0 failed
✓ tools-call-embedded-resource: 1 passed, 0 failed
✓ tools-call-mixed-content: 1 passed, 0 failed
✓ tools-call-with-logging: 1 passed, 0 failed
✓ tools-call-error: 1 passed, 0 failed
✓ tools-call-with-progress: 1 passed, 0 failed
✓ tools-call-sampling: 1 passed, 0 failed
✓ tools-call-elicitation: 1 passed, 0 failed
✗ elicitation-sep1034-defaults: 2 passed, 3 failed
✓ server-sse-multiple-streams: 2 passed, 0 failed
✗ elicitation-sep1330-enums: 4 passed, 1 failed
✓ resources-list: 1 passed, 0 failed
✓ resources-read-text: 1 passed, 0 failed
✓ resources-read-binary: 1 passed, 0 failed
✓ resources-templates-read: 1 passed, 0 failed
✓ resources-subscribe: 1 passed, 0 failed
✓ resources-unsubscribe: 1 passed, 0 failed
✓ prompts-list: 1 passed, 0 failed
✓ prompts-get-simple: 1 passed, 0 failed
✗ prompts-get-with-args: 0 passed, 1 failed
✗ prompts-get-embedded-resource: 0 passed, 1 failed
✓ prompts-get-with-image: 1 passed, 0 failed
✗ dns-rebinding-protection: 1 passed, 1 failed
Total: 33 passed, 7 failed
```
# Client Conformance Tests
Command:
```
npx tsx src/index.ts client \
--command "$HOME/Development/rust-sdk/target/release/conformance-client" \
--suite all
Output:
=== SUITE SUMMARY ===
✓ initialize: 1 passed, 0 failed
✓ tools_call: 1 passed, 0 failed
✓ elicitation-sep1034-client-defaults: 5 passed, 0 failed
✓ sse-retry: 3 passed, 0 failed
✓ auth/metadata-default: 14 passed, 0 failed
✓ auth/metadata-var1: 14 passed, 0 failed
✓ auth/metadata-var2: 14 passed, 0 failed
✗ auth/metadata-var3: 0 passed, 4 failed
✓ auth/basic-cimd: 14 passed, 0 failed
✗ auth/scope-from-www-authenticate: 11 passed, 0 failed, 1 warnings
✓ auth/scope-from-scopes-supported: 15 passed, 0 failed
✓ auth/scope-omitted-when-undefined: 15 passed, 0 failed
✗ auth/scope-step-up: 13 passed, 1 failed
✓ auth/scope-retry-limit: 11 passed, 0 failed
✓ auth/token-endpoint-auth-basic: 19 passed, 0 failed
✓ auth/token-endpoint-auth-post: 19 passed, 0 failed
✓ auth/token-endpoint-auth-none: 19 passed, 0 failed
✗ auth/resource-mismatch: 14 passed, 1 failed
✓ auth/pre-registration: 14 passed, 0 failed
✓ auth/2025-03-26-oauth-metadata-backcompat: 12 passed, 0 failed
✗ auth/2025-03-26-oauth-endpoint-fallback: 0 passed, 3 failed
✗ auth/client-credentials-jwt: 4 passed, 1 failed
✓ auth/client-credentials-basic: 9 passed, 0 failed
✗ auth/cross-app-access-complete-flow: 10 passed, 2 failed
Total: 251 passed, 12 failed, 1 warnings