Skip to content

Commit b0e1423

Browse files
KehrlannKehrlann
committed
Conformance testing: add app to test auth suite
Signed-off-by: Daniel Garnier-Moiroux <git@garnier.wf>
1 parent 86cde46 commit b0e1423

File tree

14 files changed

+726
-6
lines changed

14 files changed

+726
-6
lines changed

‎.github/workflows/conformance.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,3 +60,45 @@ jobs:
6060
command: 'java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-1.0.0-SNAPSHOT.jar'
6161
scenario: ${{ matrix.scenario }}
6262
expected-failures: ./conformance-tests/conformance-baseline.yml
63+
64+
auth:
65+
name: Auth Conformance
66+
runs-on: ubuntu-latest
67+
strategy:
68+
matrix:
69+
scenario:
70+
- auth/metadata-default
71+
- auth/metadata-var1
72+
- auth/metadata-var2
73+
- auth/metadata-var3
74+
- auth/basic-cimd
75+
- auth/scope-from-www-authenticate
76+
- auth/scope-from-scopes-supported
77+
- auth/scope-omitted-when-undefined
78+
- auth/scope-step-up
79+
- auth/scope-retry-limit
80+
- auth/token-endpoint-auth-basic
81+
- auth/token-endpoint-auth-post
82+
- auth/token-endpoint-auth-none
83+
- auth/pre-registration
84+
steps:
85+
- uses: actions/checkout@v4
86+
87+
- name: Set up JDK 17
88+
uses: actions/setup-java@v4
89+
with:
90+
java-version: '17'
91+
distribution: 'temurin'
92+
cache: 'maven'
93+
94+
- name: Build client
95+
run: mvn clean install -DskipTests
96+
97+
- name: Run conformance test
98+
uses: modelcontextprotocol/conformance@v0.1.15
99+
with:
100+
node-version: '22' # see https://github.com/modelcontextprotocol/conformance/pull/162
101+
mode: client
102+
command: 'java -jar conformance-tests/client-spring-http-client/target/client-spring-http-client-1.0.0-SNAPSHOT.jar'
103+
scenario: ${{ matrix.scenario }}
104+
expected-failures: ./conformance-tests/conformance-baseline.yml

‎conformance-tests/VALIDATION_RESULTS.md‎

Lines changed: 50 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,9 @@
22

33
## Summary
44

5-
**Server Tests:** 37/40 passed (92.5%)
5+
**Server Tests:** 37/40 passed (92.5%)
66
**Client Tests:** 3/4 scenarios passed (9/10 checks passed)
7+
**Auth Tests:** 12/14 scenarios fully passing (178 passed, 1 failed, 1 warning, 85.7% scenarios, 98.9% checks)
78

89
## Server Test Results
910

@@ -20,7 +21,7 @@
2021
### Failing (3/40)
2122

2223
1. **resources-subscribe** - Not implemented in SDK
23-
2. **resources-unsubscribe** - Not implemented in SDK
24+
2. **resources-unsubscribe** - Not implemented in SDK
2425

2526
## Client Test Results
2627

@@ -32,17 +33,45 @@
3233

3334
### Partially Passing (1/4 scenarios, 1/2 checks)
3435

35-
- **sse-retry (1/2 + 1 warning):**
36+
- **sse-retry (1/2 + 1 warning):**
3637
- ✅ Reconnects after stream closure
3738
- ❌ Does not respect retry timing
3839
- ⚠️ Does not send Last-Event-ID header (SHOULD requirement)
3940

4041
**Issue:** Client treats `retry:` SSE field as invalid instead of parsing it for reconnection timing.
4142

43+
## Auth Test Results (Spring HTTP Client)
44+
45+
**Status: 178 passed, 1 failed, 1 warning across 14 scenarios**
46+
47+
Uses the `client-spring-http-client` module with Spring Security OAuth2 and the [mcp-client-security](https://github.com/springaicommunity/mcp-client-security) library.
48+
49+
### Fully Passing (12/14 scenarios)
50+
51+
- **auth/metadata-default (12/12):** Default metadata discovery
52+
- **auth/metadata-var1 (12/12):** Metadata discovery variant 1
53+
- **auth/metadata-var2 (12/12):** Metadata discovery variant 2
54+
- **auth/metadata-var3 (12/12):** Metadata discovery variant 3
55+
- **auth/scope-from-www-authenticate (13/13):** Scope extraction from WWW-Authenticate header
56+
- **auth/scope-from-scopes-supported (13/13):** Scope extraction from scopes_supported
57+
- **auth/scope-omitted-when-undefined (13/13):** Scope omitted when not defined
58+
- **auth/scope-retry-limit (11/11):** Scope retry limit handling
59+
- **auth/token-endpoint-auth-basic (17/17):** Token endpoint with HTTP Basic auth
60+
- **auth/token-endpoint-auth-post (17/17):** Token endpoint with POST body auth
61+
- **auth/token-endpoint-auth-none (17/17):** Token endpoint with no client auth
62+
- **auth/pre-registration (6/6):** Pre-registered client credentials flow
63+
64+
### Partially Passing (2/14 scenarios)
65+
66+
- **auth/basic-cimd (12/12 + 1 warning):** Basic Client-Initiated Metadata Discovery — all checks pass, minor warning
67+
- **auth/scope-step-up (11/12):** Scope step-up challenge — 1 failure, client does not fully handle scope escalation after initial authorization
68+
4269
## Known Limitations
4370

4471
1. **Resource Subscriptions:** SDK doesn't implement `resources/subscribe` and `resources/unsubscribe` handlers
4572
2. **Client SSE Retry:** Client doesn't parse or respect the `retry:` field, reconnects immediately, and doesn't send Last-Event-ID header
73+
3. **Auth Scope Step-Up:** Client does not fully handle scope step-up challenges where the server requests additional scopes after initial authorization
74+
4. **Auth Basic CIMD:** Minor conformance warning in the basic Client-Initiated Metadata Discovery flow
4675

4776
## Running Tests
4877

@@ -70,11 +99,26 @@ for scenario in initialize tools_call elicitation-sep1034-client-defaults sse-re
7099
done
71100
```
72101

102+
### Auth (Spring HTTP Client)
103+
104+
Ensure you run with the conformance testing suite `0.1.15` or higher.
105+
106+
```bash
107+
# Build
108+
cd conformance-tests/client-spring-http-client
109+
../../mvnw clean package -DskipTests
110+
111+
# Run auth suite
112+
npx @modelcontextprotocol/conformance@0.1.15 client \
113+
--spec-version 2025-11-25 \
114+
--command "java -jar target/client-spring-http-client-0.18.0-SNAPSHOT.jar" \
115+
--suite auth
116+
```
117+
73118
## Recommendations
74119

75120
### High Priority
76121
1. Fix client SSE retry field handling in `HttpClientStreamableHttpTransport`
77122
2. Implement resource subscription handlers in `McpStatelessAsyncServer`
78-
79-
### Medium Priority
80-
3. Add Host/Origin validation in `HttpServletStreamableServerTransportProvider` for DNS rebinding protection
123+
3. Implement CIMD
124+
4. Implement scope step up

‎conformance-tests/client-spring-http-client/README.md‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# MCP Conformance Tests - Spring HTTP Client (Auth Suite)
2+
3+
This module provides a conformance test client implementation for the Java MCP SDK's **auth** suite.
4+
5+
OAuth2 support is not implemented in the SDK itself, but we provide hooks to implement the Authorization section of the specification. One such implementation is done in Spring, with Sprign AI and the [mcp-client-security](https://github.com/springaicommunity/mcp-client-security) library.
6+
7+
## Overview
8+
9+
The conformance test client is designed to work with the [MCP Conformance Test Framework](https://github.com/modelcontextprotocol/conformance). It validates that the Java MCP SDK client, combined with Spring Security's OAuth2 support, properly implements the MCP authorization specification.
10+
11+
Test with @modelcontextprotocol/conformance@0.1.15.
12+
13+
## Conformance Test Results
14+
15+
**Status: 178 passed, 1 failed, 1 warning across 14 scenarios**
16+
17+
| Scenario | Result | Details |
18+
|---|---|---|
19+
| auth/metadata-default | ✅ Pass | 12/12 |
20+
| auth/metadata-var1 | ✅ Pass | 12/12 |
21+
| auth/metadata-var2 | ✅ Pass | 12/12 |
22+
| auth/metadata-var3 | ✅ Pass | 12/12 |
23+
| auth/basic-cimd | ⚠️ Warning | 12/12 passed, 1 warning |
24+
| auth/scope-from-www-authenticate | ✅ Pass | 13/13 |
25+
| auth/scope-from-scopes-supported | ✅ Pass | 13/13 |
26+
| auth/scope-omitted-when-undefined | ✅ Pass | 13/13 |
27+
| auth/scope-step-up | ❌ Fail | 11/12 (1 failed) |
28+
| auth/scope-retry-limit | ✅ Pass | 11/11 |
29+
| auth/token-endpoint-auth-basic | ✅ Pass | 17/17 |
30+
| auth/token-endpoint-auth-post | ✅ Pass | 17/17 |
31+
| auth/token-endpoint-auth-none | ✅ Pass | 17/17 |
32+
| auth/pre-registration | ✅ Pass | 6/6 |
33+
34+
See [VALIDATION_RESULTS.md](../VALIDATION_RESULTS.md) for the full project validation results.
35+
36+
## Architecture
37+
38+
The client is a Spring Boot application that reads test scenarios from environment variables and accepts the server URL as a command-line argument, following the conformance framework's conventions:
39+
40+
- **MCP_CONFORMANCE_SCENARIO**: Environment variable specifying which test scenario to run
41+
- **MCP_CONFORMANCE_CONTEXT**: Environment variable with JSON context (used by `auth/pre-registration`)
42+
- **Server URL**: Passed as the last command-line argument
43+
44+
### Scenario Routing
45+
46+
The application uses Spring's conditional configuration to select the appropriate scenario at startup:
47+
48+
- **`DefaultConfiguration`** — Activated for all scenarios except `auth/pre-registration`. Uses the OAuth2 Authorization Code flow with dynamic client registration via `McpClientOAuth2Configurer`.
49+
- **`PreRegistrationConfiguration`** — Activated only for `auth/pre-registration`. Uses the Client Credentials flow with pre-registered client credentials read from `MCP_CONFORMANCE_CONTEXT`.
50+
51+
### Key Dependencies
52+
53+
- **Spring Boot 4.0** with Spring Security OAuth2 Client
54+
- **Spring AI MCP Client** (`spring-ai-starter-mcp-client`)
55+
- **mcp-client-security** — Community library providing MCP-specific OAuth2 integration (metadata discovery, dynamic client registration, transport context)
56+
57+
## Building
58+
59+
Build the executable JAR:
60+
61+
```bash
62+
cd conformance-tests/client-spring-http-client
63+
../../mvnw clean package -DskipTests
64+
```
65+
66+
This creates an executable JAR at:
67+
```
68+
target/client-spring-http-client-0.18.0-SNAPSHOT.jar
69+
```
70+
71+
## Running Tests
72+
73+
### Using the Conformance Framework
74+
75+
Run the full auth suite:
76+
77+
```bash
78+
npx @modelcontextprotocol/conformance@0.1.15 client \
79+
--spec-version 2025-11-25 \
80+
--command "java -jar conformance-tests/client-spring-http-client/target/client-spring-http-client-0.18.0-SNAPSHOT.jar" \
81+
--suite auth
82+
```
83+
84+
Run a single scenario:
85+
86+
```bash
87+
npx @modelcontextprotocol/conformance@0.1.15 client \
88+
--spec-version 2025-11-25 \
89+
--command "java -jar conformance-tests/client-spring-http-client/target/client-spring-http-client-0.18.0-SNAPSHOT.jar" \
90+
--scenario auth/metadata-default
91+
```
92+
93+
Run with verbose output:
94+
95+
```bash
96+
npx @modelcontextprotocol/conformance@0.1.15 client \
97+
--spec-version 2025-11-25 \
98+
--command "java -jar conformance-tests/client-spring-http-client/target/client-spring-http-client-0.18.0-SNAPSHOT.jar" \
99+
--scenario auth/metadata-default \
100+
--verbose
101+
```
102+
103+
### Manual Testing
104+
105+
You can also run the client manually if you have a test server:
106+
107+
```bash
108+
export MCP_CONFORMANCE_SCENARIO=auth/metadata-default
109+
java -jar conformance-tests/client-spring-http-client/target/client-spring-http-client-0.18.0-SNAPSHOT.jar http://localhost:3000/mcp
110+
```
111+
112+
## Known Issues
113+
114+
1. **auth/scope-step-up** (1 failure) — The client does not fully handle scope step-up challenges where the server requests additional scopes after initial authorization.
115+
2. **auth/basic-cimd** (1 warning) — Minor conformance warning in the basic Client-Initiated Metadata Discovery flow.
116+
117+
## References
118+
119+
- [MCP Specification — Authorization](https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization)
120+
- [MCP Conformance Tests](https://github.com/modelcontextprotocol/conformance)
121+
- [mcp-client-security Library](https://github.com/springaicommunity/mcp-client-security)
122+
- [SDK Integration Guide](https://github.com/modelcontextprotocol/conformance/blob/main/SDK_INTEGRATION.md)

‎conformance-tests/client-spring-http-client/pom.xml‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0"
3+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
5+
<modelVersion>4.0.0</modelVersion>
6+
<parent>
7+
<groupId>org.springframework.boot</groupId>
8+
<artifactId>spring-boot-starter-parent</artifactId>
9+
<version>4.0.2</version>
10+
<relativePath/> <!-- lookup parent from repository -->
11+
</parent>
12+
<groupId>io.modelcontextprotocol.sdk</groupId>
13+
<artifactId>client-spring-http-client</artifactId>
14+
<version>1.0.0-SNAPSHOT</version>
15+
<packaging>jar</packaging>
16+
<name>MCP Conformance Tests - Spring HTTP Client</name>
17+
<description>Spring HTTP Client conformance tests for the Java MCP SDK</description>
18+
<url>https://github.com/modelcontextprotocol/java-sdk</url>
19+
20+
<scm>
21+
<url>https://github.com/modelcontextprotocol/java-sdk</url>
22+
<connection>git://github.com/modelcontextprotocol/java-sdk.git</connection>
23+
<developerConnection>git@github.com/modelcontextprotocol/java-sdk.git</developerConnection>
24+
</scm>
25+
26+
<properties>
27+
<java.version>17</java.version>
28+
<spring-ai.version>2.0.0-M2</spring-ai.version>
29+
</properties>
30+
31+
<dependencies>
32+
<dependency>
33+
<groupId>org.springframework.boot</groupId>
34+
<artifactId>spring-boot-starter-webmvc</artifactId>
35+
</dependency>
36+
37+
<dependency>
38+
<groupId>org.springframework.boot</groupId>
39+
<artifactId>spring-boot-starter-restclient</artifactId>
40+
</dependency>
41+
42+
<dependency>
43+
<groupId>org.springframework.ai</groupId>
44+
<artifactId>spring-ai-starter-mcp-client</artifactId>
45+
<version>${spring-ai.version}</version>
46+
</dependency>
47+
48+
<dependency>
49+
<groupId>org.springframework.boot</groupId>
50+
<artifactId>spring-boot-starter-oauth2-client</artifactId>
51+
</dependency>
52+
53+
<dependency>
54+
<groupId>org.springaicommunity</groupId>
55+
<artifactId>mcp-client-security</artifactId>
56+
<version>0.1.2</version>
57+
</dependency>
58+
</dependencies>
59+
60+
<build>
61+
<plugins>
62+
<plugin>
63+
<groupId>org.springframework.boot</groupId>
64+
<artifactId>spring-boot-maven-plugin</artifactId>
65+
</plugin>
66+
</plugins>
67+
</build>
68+
69+
<repositories>
70+
<repository>
71+
<id>maven-central</id>
72+
<url>https://repo.maven.apache.org/maven2/</url>
73+
<snapshots>
74+
<enabled>false</enabled>
75+
</snapshots>
76+
<releases>
77+
<enabled>true</enabled>
78+
</releases>
79+
</repository>
80+
<repository>
81+
<id>spring-milestones</id>
82+
<name>Spring Milestones</name>
83+
<url>https://repo.spring.io/milestone</url>
84+
<snapshots>
85+
<enabled>false</enabled>
86+
</snapshots>
87+
</repository>
88+
</repositories>
89+
90+
</project>

0 commit comments

Comments
 (0)