CI: add Claude NL suite workflow, prompt, and long Unity test script … #30
Conversation
…on default branch for manual runs
|
Caution Review failedThe pull request is closed. WalkthroughAdds a new Claude NL/T test prompt, a GitHub Actions workflow to run the suite and optionally compile Unity projects/packages, and a large standalone Unity MonoBehaviour script used as the primary NL test target. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant GH as GitHub Actions
participant Repo as Repository
participant MCP as Unity MCP Server
participant Claude as Claude Code Base Action
participant Reporter as Test Reporter
participant Unity as Unity Test Runner
GH->>Repo: checkout (fetch-depth: 0)
GH->>GH: setup Python/uv
GH->>MCP: install deps (pyproject/requirements)
GH->>Claude: run NL/T suite (prompt_file, tools, model, max_turns)
Claude->>MCP: structured edit ops (replace/insert/delete/anchor/edits)
MCP-->>Claude: results + evidence
Claude->>Repo: write reports (JUnit XML, Markdown)
GH->>GH: upload artifact (JUnit)
GH->>Reporter: annotate PR (java-junit)
GH->>GH: detect Unity mode (license, project/package)
alt Project mode
GH->>Unity: run EditMode tests (projectPath .)
else Package mode
GH->>Unity: run package compile (packageMode true)
end
GH->>Repo: clean working tree (restore + clean)
sequenceDiagram
autonumber
participant UnityLoop as Unity Engine
participant Script as LongUnityScriptClaudeTest
participant Anim as Animator
UnityLoop->>Script: Update()
Script->>Script: FindBestTarget()
Script->>Script: HandleTargetSwitch(prev/current, timestamps)
Script->>Script: TickBlendOnce()
Script->>Script: AccumulateBlend()
Script->>Anim: ApplyBlend(reachX, reachY)
UnityLoop->>Script: LateUpdate()
Script->>Script: Decay previousTarget if elapsed > 0.5s
Note over Script,Anim: OnObjectHeld/OnObjectPlaced adjust held list and Anim "objectsHeld"
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. 📜 Recent review detailsConfiguration used: CodeRabbit UI 💡 Knowledge Base configuration:
You can enable these sources in your CodeRabbit configuration. 📒 Files selected for processing (3)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Greptile Summary
This PR introduces a comprehensive automated testing framework for validating the Unity MCP's natural language interface capabilities with Claude Desktop. The changes add three key components that work together to create a robust CI testing environment:
Core Components Added:
-
Claude NL Test Suite (
.claude/prompts/nl-unity-suite.md) - A detailed specification document defining 20+ test scenarios that validate MCP editing functionality through natural language commands. These tests cover everything from basic code modifications to complex error handling scenarios, ensuring the Unity-Claude integration works reliably across various use cases. -
GitHub Actions Workflow (
.github/workflows/claude-nl-suite.yml) - A sophisticated CI pipeline that orchestrates the entire testing process. The workflow sets up Python dependencies for the MCP server, executes Claude NL/T tests using Anthropic's code generation actions, and optionally compiles Unity projects to validate generated code. It includes intelligent project structure detection and proper cleanup mechanisms. -
Large Unity Test Script (
ClaudeTests/longUnityScript-claudeTest.cs) - A deliberately verbose 2000+ line Unity MonoBehaviour script designed to test Claude's ability to navigate and edit large codebases. The script includes 600+ padding methods to simulate real-world complexity while maintaining a functional core with target tracking and object management systems.
Integration with Existing Codebase:
This testing framework builds upon the existing Unity MCP bridge architecture found in the UnityMcpBridge directory. The workflow is designed to work with the MCP server components and validates the communication layer between Claude Desktop and Unity Editor tools. The manual trigger mechanism (workflow_dispatch) aligns with the project's approach to controlled, expensive AI-powered testing that complements the existing automated Unity tests in .github/workflows/unity-tests.yml.
The test suite follows established patterns from the existing codebase, using proper Unity conventions and maintaining the same structure as other test assets in the TestProjects directory.
Important Files Changed
Files Modified
| Filename | Score | Overview |
|---|---|---|
.github/workflows/claude-nl-suite.yml |
2/5 | Adds CI workflow for Claude NL testing but has critical path configuration issues |
.claude/prompts/nl-unity-suite.md |
4/5 | Comprehensive test specification for Claude-Unity MCP validation with proper safety measures |
ClaudeTests/longUnityScript-claudeTest.cs |
5/5 | Large Unity test script with intentional verbosity for AI editing validation |
Confidence score: 2/5
- This PR requires careful review due to critical configuration issues that will prevent the workflow from executing successfully
- Score lowered due to incorrect MCP server path references and potential missing dependencies that could cause immediate CI failures
- Pay close attention to
.github/workflows/claude-nl-suite.ymlwhich references non-existent server paths and may have dependency resolution issues
Sequence Diagram
sequenceDiagram
participant User
participant "GitHub Actions" as GHA
participant "Claude Code Base Action" as Claude
participant "Unity MCP Server" as MCP
participant "Unity Test File" as TestFile
participant "Test Reporter" as Reporter
User->>GHA: "Trigger workflow_dispatch"
GHA->>GHA: "Checkout repository"
GHA->>GHA: "Install Python + uv"
GHA->>GHA: "Install UnityMcpServer dependencies"
GHA->>Claude: "Run Claude NL/T test suite"
Note over Claude: "Load prompt from .claude/prompts/nl-unity-suite.md"
Claude->>MCP: "Start Unity MCP Server"
Note over MCP: "python UnityMcpBridge/UnityMcpServer~/src/server.py"
loop "For each test case (NL-0 through T-J)"
Claude->>TestFile: "Read ClaudeTests/longUnityScript-claudeTest.cs"
Claude->>MCP: "mcp__unity__* tool calls"
Note over Claude,MCP: "replace_method, insert_method, delete_method, etc."
MCP->>TestFile: "Apply structured edits"
Claude->>TestFile: "Verify changes via windowed reads"
Claude->>TestFile: "Revert changes for cleanup"
end
Claude->>Claude: "Generate JUnit XML report"
Claude->>Claude: "Generate markdown summary"
Claude-->>GHA: "Return test results"
GHA->>GHA: "Upload JUnit artifacts"
GHA->>Reporter: "Annotate PR with test results"
alt "If Unity license available"
GHA->>GHA: "Run Unity compile tests"
Note over GHA: "Validate that edits don't break compilation"
end
GHA->>GHA: "Clean working tree (discard temp edits)"
3 files reviewed, 1 comment
| "mcpServers": { | ||
| "unity": { | ||
| "command": "python", | ||
| "args": ["UnityMcpBridge/UnityMcpServer~/src/server.py"] |
There was a problem hiding this comment.
logic: Path 'UnityMcpBridge/UnityMcpServer~/src/server.py' does not exist in the repository. The MCP server path needs to be corrected or the server files need to be added.
…on default branch for manual runs. This will ultimately become the suite of Claude Desktop test prompts that will talk to the Unity MCP, and to headless Unity, to validate MCP functionality.
Summary by CodeRabbit
New Features
Documentation
Tests
CI