Skip to content

Commit a27d36e

Browse files
markpollackmarkpollack
committed
feat: expand CI to test all 17 passing examples
- Update GitHub Actions workflow to run all 17 known passing tests - Increase timeout from 30 to 60 minutes for full test suite - Organize tests into logical groups (simple, agents, patterns, MCP) - Add progress indicators showing test count [n/17] - Update summary to list all 17 tests being executed - Complete Phase 3a documentation in CI plan
1 parent 4276bff commit a27d36e

File tree

2 files changed

+231
-45
lines changed

2 files changed

+231
-45
lines changed

‎.github/workflows/integration-tests-manual.yml‎

Lines changed: 127 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ on:
1717
jobs:
1818
test:
1919
runs-on: ubuntu-latest
20-
timeout-minutes: 30
20+
timeout-minutes: 60
2121

2222
steps:
2323
- name: Checkout repository
@@ -72,11 +72,12 @@ jobs:
7272
BRAVE_API_KEY: ${{ secrets.BRAVE_API_KEY }}
7373
run: |
7474
echo "🧪 Running integration tests with Spring AI ${{ inputs.spring_ai_version }}..."
75-
echo "📝 Note: Only running tests known to pass consistently"
75+
echo "📝 Testing all 17 known passing examples"
7676
echo ""
7777
7878
# Track test results
7979
test_results=0
80+
test_count=0
8081
8182
# List of known passing tests (17 total)
8283
# Excluding:
@@ -88,27 +89,116 @@ jobs:
8889
# - model-context-protocol/dynamic-tool-update/client (needs server)
8990
# - model-context-protocol/dynamic-tool-update/server (needs client)
9091
91-
# Test subset of reliably passing examples
92-
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
93-
echo "Test 1: models/chat/helloworld"
94-
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
92+
echo "════════════════════════════════════"
93+
echo " SIMPLE EXAMPLES (3 tests)"
94+
echo "════════════════════════════════════"
95+
echo ""
96+
97+
test_count=$((test_count + 1))
98+
echo "[$test_count/17] models/chat/helloworld"
9599
./integration-testing/scripts/run-integration-tests.sh "chat/helloworld" || test_results=$((test_results + 1))
100+
echo ""
96101
102+
test_count=$((test_count + 1))
103+
echo "[$test_count/17] kotlin/kotlin-hello-world"
104+
./integration-testing/scripts/run-integration-tests.sh "kotlin-hello-world" || test_results=$((test_results + 1))
105+
echo ""
106+
107+
test_count=$((test_count + 1))
108+
echo "[$test_count/17] misc/spring-ai-java-function-callback"
109+
./integration-testing/scripts/run-integration-tests.sh "spring-ai-java-function-callback" || test_results=$((test_results + 1))
97110
echo ""
98-
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
99-
echo "Test 2: agents/reflection"
100-
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
111+
112+
echo "════════════════════════════════════"
113+
echo " AGENT EXAMPLES (2 tests)"
114+
echo "════════════════════════════════════"
115+
echo ""
116+
117+
test_count=$((test_count + 1))
118+
echo "[$test_count/17] agents/reflection"
101119
./integration-testing/scripts/run-integration-tests.sh "agents/reflection" || test_results=$((test_results + 1))
120+
echo ""
102121
122+
test_count=$((test_count + 1))
123+
echo "[$test_count/17] agents/tools-and-agent-tools"
124+
./integration-testing/scripts/run-integration-tests.sh "agents/tools-and-agent-tools" || test_results=$((test_results + 1))
125+
echo ""
126+
127+
echo "════════════════════════════════════"
128+
echo " AGENTIC PATTERNS (5 tests)"
129+
echo "════════════════════════════════════"
130+
echo ""
131+
132+
test_count=$((test_count + 1))
133+
echo "[$test_count/17] agentic-patterns/chain-workflow"
134+
./integration-testing/scripts/run-integration-tests.sh "agentic-patterns/chain-workflow" || test_results=$((test_results + 1))
135+
echo ""
136+
137+
test_count=$((test_count + 1))
138+
echo "[$test_count/17] agentic-patterns/evaluator-optimizer-pattern"
139+
./integration-testing/scripts/run-integration-tests.sh "agentic-patterns/evaluator-optimizer-pattern" || test_results=$((test_results + 1))
140+
echo ""
141+
142+
test_count=$((test_count + 1))
143+
echo "[$test_count/17] agentic-patterns/orchestrator-workers"
144+
./integration-testing/scripts/run-integration-tests.sh "agentic-patterns/orchestrator-workers" || test_results=$((test_results + 1))
145+
echo ""
146+
147+
test_count=$((test_count + 1))
148+
echo "[$test_count/17] agentic-patterns/parallelization"
149+
./integration-testing/scripts/run-integration-tests.sh "agentic-patterns/parallelization" || test_results=$((test_results + 1))
103150
echo ""
104-
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
105-
echo "Test 3: misc/spring-ai-java-function-callback"
106-
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
107-
./integration-testing/scripts/run-integration-tests.sh "spring-ai-java-function-callback" || test_results=$((test_results + 1))
108151
152+
test_count=$((test_count + 1))
153+
echo "[$test_count/17] agentic-patterns/routing-agent"
154+
./integration-testing/scripts/run-integration-tests.sh "agentic-patterns/routing-agent" || test_results=$((test_results + 1))
109155
echo ""
156+
157+
echo "════════════════════════════════════"
158+
echo " MCP EXAMPLES (7 tests)"
159+
echo "════════════════════════════════════"
160+
echo ""
161+
162+
test_count=$((test_count + 1))
163+
echo "[$test_count/17] model-context-protocol/brave"
164+
./integration-testing/scripts/run-integration-tests.sh "model-context-protocol/brave" || test_results=$((test_results + 1))
165+
echo ""
166+
167+
test_count=$((test_count + 1))
168+
echo "[$test_count/17] model-context-protocol/client-starter/starter-default-client"
169+
./integration-testing/scripts/run-integration-tests.sh "model-context-protocol/client-starter/starter-default-client" || test_results=$((test_results + 1))
170+
echo ""
171+
172+
test_count=$((test_count + 1))
173+
echo "[$test_count/17] model-context-protocol/filesystem"
174+
./integration-testing/scripts/run-integration-tests.sh "model-context-protocol/filesystem" || test_results=$((test_results + 1))
175+
echo ""
176+
177+
test_count=$((test_count + 1))
178+
echo "[$test_count/17] model-context-protocol/sqlite/chatbot"
179+
./integration-testing/scripts/run-integration-tests.sh "model-context-protocol/sqlite/chatbot" || test_results=$((test_results + 1))
180+
echo ""
181+
182+
test_count=$((test_count + 1))
183+
echo "[$test_count/17] model-context-protocol/sqlite/simple"
184+
./integration-testing/scripts/run-integration-tests.sh "model-context-protocol/sqlite/simple" || test_results=$((test_results + 1))
185+
echo ""
186+
187+
test_count=$((test_count + 1))
188+
echo "[$test_count/17] model-context-protocol/web-search/brave-chatbot"
189+
./integration-testing/scripts/run-integration-tests.sh "model-context-protocol/web-search/brave-chatbot" || test_results=$((test_results + 1))
190+
echo ""
191+
192+
test_count=$((test_count + 1))
193+
echo "[$test_count/17] model-context-protocol/web-search/brave-starter"
194+
./integration-testing/scripts/run-integration-tests.sh "model-context-protocol/web-search/brave-starter" || test_results=$((test_results + 1))
195+
echo ""
196+
197+
echo "════════════════════════════════════"
198+
echo " TEST SUMMARY"
199+
echo "════════════════════════════════════"
110200
if [ $test_results -eq 0 ]; then
111-
echo "✅ All 3 examples tested successfully with Spring AI ${{ inputs.spring_ai_version }}"
201+
echo "✅ All 17 examples tested successfully with Spring AI ${{ inputs.spring_ai_version }}"
112202
else
113203
echo "❌ $test_results test(s) failed with Spring AI ${{ inputs.spring_ai_version }}"
114204
exit 1
@@ -140,13 +230,34 @@ jobs:
140230
echo "## 📊 Test Summary" >> $GITHUB_STEP_SUMMARY
141231
echo "" >> $GITHUB_STEP_SUMMARY
142232
echo "- **Spring AI Version**: ${{ inputs.spring_ai_version }}" >> $GITHUB_STEP_SUMMARY
143-
echo "- **Test Filter**: ${{ inputs.test_filter || 'None (ran 3 examples)' }}" >> $GITHUB_STEP_SUMMARY
233+
echo "- **Test Filter**: ${{ inputs.test_filter || 'None (ran all 17 passing examples)' }}" >> $GITHUB_STEP_SUMMARY
144234
echo "- **Status**: ${{ job.status }}" >> $GITHUB_STEP_SUMMARY
145235
echo "" >> $GITHUB_STEP_SUMMARY
146-
echo "### Examples Tested" >> $GITHUB_STEP_SUMMARY
236+
echo "### Examples Tested (17 total)" >> $GITHUB_STEP_SUMMARY
237+
echo "#### Simple Examples:" >> $GITHUB_STEP_SUMMARY
147238
echo "1. models/chat/helloworld" >> $GITHUB_STEP_SUMMARY
148239
echo "2. kotlin/kotlin-hello-world" >> $GITHUB_STEP_SUMMARY
149240
echo "3. misc/spring-ai-java-function-callback" >> $GITHUB_STEP_SUMMARY
241+
echo "" >> $GITHUB_STEP_SUMMARY
242+
echo "#### Agent Examples:" >> $GITHUB_STEP_SUMMARY
243+
echo "4. agents/reflection" >> $GITHUB_STEP_SUMMARY
244+
echo "5. agents/tools-and-agent-tools" >> $GITHUB_STEP_SUMMARY
245+
echo "" >> $GITHUB_STEP_SUMMARY
246+
echo "#### Agentic Patterns:" >> $GITHUB_STEP_SUMMARY
247+
echo "6. agentic-patterns/chain-workflow" >> $GITHUB_STEP_SUMMARY
248+
echo "7. agentic-patterns/evaluator-optimizer-pattern" >> $GITHUB_STEP_SUMMARY
249+
echo "8. agentic-patterns/orchestrator-workers" >> $GITHUB_STEP_SUMMARY
250+
echo "9. agentic-patterns/parallelization" >> $GITHUB_STEP_SUMMARY
251+
echo "10. agentic-patterns/routing-agent" >> $GITHUB_STEP_SUMMARY
252+
echo "" >> $GITHUB_STEP_SUMMARY
253+
echo "#### MCP Examples:" >> $GITHUB_STEP_SUMMARY
254+
echo "11. model-context-protocol/brave" >> $GITHUB_STEP_SUMMARY
255+
echo "12. model-context-protocol/client-starter/starter-default-client" >> $GITHUB_STEP_SUMMARY
256+
echo "13. model-context-protocol/filesystem" >> $GITHUB_STEP_SUMMARY
257+
echo "14. model-context-protocol/sqlite/chatbot" >> $GITHUB_STEP_SUMMARY
258+
echo "15. model-context-protocol/sqlite/simple" >> $GITHUB_STEP_SUMMARY
259+
echo "16. model-context-protocol/web-search/brave-chatbot" >> $GITHUB_STEP_SUMMARY
260+
echo "17. model-context-protocol/web-search/brave-starter" >> $GITHUB_STEP_SUMMARY
150261
if [ "${{ inputs.test_filter }}" != "" ]; then
151262
echo "" >> $GITHUB_STEP_SUMMARY
152263
echo "**Additional filtered test**: ${{ inputs.test_filter }}" >> $GITHUB_STEP_SUMMARY

‎integration-testing/plans/github-actions-ci-plan.md‎

Lines changed: 104 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -188,68 +188,143 @@ Note: Our implementation will use simpler patterns suitable for examples reposit
188188
*Critical validation phase - ensure all integration tests work locally before CI/CD*
189189

190190
### Phase 3a Pre-Conditions
191-
- [ ] Confirm version management scripts work
192-
- [ ] Have OPENAI_API_KEY configured locally
193-
- [ ] Have ANTHROPIC_API_KEY configured locally (for AI validation)
194-
- [ ] Claude Code CLI installed locally
191+
- [x] Confirm version management scripts work
192+
- [x] Have OPENAI_API_KEY configured locally
193+
- [x] Have ANTHROPIC_API_KEY configured locally (for AI validation)
194+
- [x] Claude Code CLI installed locally
195195

196196
### Phase 3a Implementation - Version Update
197-
- [ ] Update all modules to Spring AI 1.0.1:
197+
- [x] Update all modules to Spring AI 1.0.1:
198198
```bash
199199
./scripts/update-spring-ai-version.sh 1.0.1
200200
```
201-
- [ ] Verify version update:
201+
- [x] Verify version update:
202202
```bash
203203
./scripts/check-spring-ai-version.sh
204204
```
205-
- [ ] Confirm all 17 pom files show version 1.0.1
205+
- [x] Confirm all 32 pom files show version 1.0.1 (17 use property, 15 use direct BOM)
206206

207207
### Phase 3a Testing - Core Examples
208-
- [ ] Test models/chat/helloworld:
208+
- [x] Test models/chat/helloworld:
209209
```bash
210210
./integration-testing/scripts/run-integration-tests.sh "chat/helloworld"
211211
```
212-
- [ ] Test kotlin/kotlin-hello-world:
212+
- [x] Test kotlin/kotlin-hello-world:
213213
```bash
214214
./integration-testing/scripts/run-integration-tests.sh kotlin-hello-world
215215
```
216-
- [ ] Test misc/spring-ai-java-function-callback:
216+
- [x] Test misc/spring-ai-java-function-callback:
217217
```bash
218218
./integration-testing/scripts/run-integration-tests.sh spring-ai-java-function-callback
219219
```
220220

221221
### Phase 3a Testing - All Examples
222-
- [ ] Run full test suite:
222+
- [x] Run full test suite:
223223
```bash
224224
./integration-testing/scripts/run-integration-tests.sh
225225
```
226-
- [ ] Document which tests pass/fail
227-
- [ ] Identify any version-specific issues
228-
- [ ] Fix or disable failing tests
229-
- [ ] Re-run until all enabled tests pass
226+
- [x] Document which tests pass/fail (17 pass, 7 fail/timeout)
227+
- [x] Identify any version-specific issues (none found - false negative on kotlin-hello-world)
228+
- [x] Fix Docker Compose path issue in kotlin/rag-with-kotlin
229+
- [x] Re-run until all enabled tests pass
230230

231231
### Phase 3a AI Validation Testing
232-
- [ ] Verify Claude Code CLI is available:
232+
- [x] Verify Claude Code CLI is available:
233233
```bash
234234
claude --version
235235
```
236-
- [ ] Test with AI validation enabled for core examples
237-
- [ ] Document any AI validation issues
238-
- [ ] Adjust validation prompts if needed
236+
- [x] Test with AI validation enabled for core examples
237+
- [x] Document AI validation false negative for kotlin-hello-world
238+
- [x] Added TODO to investigate AI validation issues
239239

240240
### Phase 3a Troubleshooting
241-
- [ ] For build failures: Check for API breaking changes in 1.0.1
242-
- [ ] For runtime failures: Check logs in integration-testing/logs/
243-
- [ ] For AI validation failures: Check Claude API access and prompts
244-
- [ ] Document all issues and resolutions
241+
- [x] For build failures: Checked for API breaking changes in 1.0.1 (none found)
242+
- [x] For runtime failures: Checked logs in integration-testing/logs/
243+
- [x] For AI validation failures: Identified false negative with kotlin-hello-world
244+
- [x] Document all issues and resolutions in analysis-summary.md
245245

246246
### Phase 3a Completion & Commit Point
247-
- [ ] All core examples (3) pass with Spring AI 1.0.1
248-
- [ ] Document pass/fail status for all examples
249-
- [ ] Commit any fixes: `git commit -m "fix: ensure integration tests work with Spring AI 1.0.1"`
250-
- [ ] Create learnings document: `integration-testing/learnings/phase-3a-local-testing.md`
251-
- [ ] Update test configurations as needed
252-
- [ ] Review Phase 3b and adjust based on findings
247+
- [x] All core examples (3) pass with Spring AI 1.0.1
248+
- [x] Document pass/fail status for all examples (test-results-1.0.1.md)
249+
- [x] Commit any fixes: `git commit -m "fix: resolve Docker Compose path for kotlin RAG example"`
250+
- [x] Create learnings document: Created test-results-1.0.1.md and analysis-summary.md
251+
- [x] Update test configurations as needed (TODO.txt updated)
252+
- [x] Review Phase 3b and adjust based on findings
253+
254+
---
255+
256+
## Phase 3c: Validate All 17 Passing Tests in CI
257+
258+
*Critical validation phase - ensure all 17 known passing tests work in GitHub Actions before matrix testing*
259+
260+
### Phase 3c Pre-Conditions
261+
- [ ] Read learnings from Phase 3a (test results with 1.0.1)
262+
- [ ] Confirm which 17 tests pass locally with API keys
263+
- [ ] Ensure ANTHROPIC_API_KEY is added to GitHub Secrets
264+
- [ ] Ensure BRAVE_API_KEY is added to GitHub Secrets
265+
266+
### Phase 3c Implementation - Expand Workflow
267+
- [ ] Update workflow to run all 17 passing tests instead of just 3:
268+
```yaml
269+
# The 17 passing tests:
270+
# 1. agents/reflection
271+
# 2. agents/tools-and-agent-tools
272+
# 3. agentic-patterns/chain-workflow
273+
# 4. agentic-patterns/evaluator-optimizer-pattern
274+
# 5. agentic-patterns/orchestrator-workers
275+
# 6. agentic-patterns/parallelization
276+
# 7. agentic-patterns/routing-agent
277+
# 8. kotlin/kotlin-hello-world (with AI validation workaround)
278+
# 9. misc/spring-ai-java-function-callback
279+
# 10. model-context-protocol/brave
280+
# 11. model-context-protocol/client-starter/starter-default-client
281+
# 12. model-context-protocol/filesystem
282+
# 13. model-context-protocol/sqlite/chatbot
283+
# 14. model-context-protocol/sqlite/simple
284+
# 15. model-context-protocol/web-search/brave-chatbot
285+
# 16. model-context-protocol/web-search/brave-starter
286+
# 17. models/chat/helloworld
287+
```
288+
- [ ] Structure tests in logical groups (agents, patterns, MCP, etc.)
289+
- [ ] Add progress indicators between test groups
290+
- [ ] Increase timeout if needed (from 30 to 60 minutes)
291+
292+
### Phase 3c Testing Strategy
293+
- [ ] Run tests in order of complexity (simple → complex)
294+
- [ ] Group related tests together for better debugging
295+
- [ ] Add clear section headers in output
296+
- [ ] Track execution time per test
297+
298+
### Phase 3c GitHub Actions Execution
299+
- [ ] Commit updated workflow: `git commit -m "feat: expand CI to test all 17 passing examples"`
300+
- [ ] Push to GitHub: `git push`
301+
- [ ] Trigger workflow with Spring AI 1.0.1
302+
- [ ] Monitor execution for all 17 tests
303+
- [ ] Document execution times
304+
- [ ] Identify any CI-specific failures
305+
306+
### Phase 3c Troubleshooting
307+
- [ ] If tests fail in CI but pass locally:
308+
- Check environment variable configuration
309+
- Verify API keys are properly set
310+
- Review CI-specific resource constraints
311+
- Check for timeout issues
312+
- [ ] Document any CI-specific adjustments needed
313+
- [ ] Update workflow with fixes
314+
- [ ] Re-run until all 17 tests pass
315+
316+
### Phase 3c Performance Analysis
317+
- [ ] Calculate total execution time
318+
- [ ] Identify slowest tests
319+
- [ ] Determine if parallel execution is needed
320+
- [ ] Document resource usage
321+
322+
### Phase 3c Completion & Commit Point
323+
- [ ] All 17 tests pass in GitHub Actions with Spring AI 1.0.1
324+
- [ ] Document execution times and resource usage
325+
- [ ] Commit final workflow: `git commit -m "feat: successfully validate 17 passing tests in CI"`
326+
- [ ] Create learnings document: `integration-testing/learnings/phase-3c-full-validation.md`
327+
- [ ] Review Phase 4 and determine if matrix testing is viable with current execution times
253328

254329
---
255330

0 commit comments

Comments
 (0)