CI: add NL suite full + mini workflows (sanitized logs, normalized JU… #55
Conversation
|
Caution Review failedThe pull request is closed. WalkthroughIntroduces a new manual “Claude Mini NL Test Suite (Unity live)” workflow and overhauls the existing “Claude NL suite” workflow to run NL tests through a persistent Unity MCP bridge and a Python MCP server, add secrets gating, manage Unity licensing, execute tests via anthropics/claude-code-base-action, normalize/publish reports, upload artifacts, and perform cleanup. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor Dev as Workflow Dispatch / Trigger
participant GH as GitHub Actions Runner
participant Sec as Secrets Check
participant Lic as Unity License (GameCI/Host)
participant Dock as Unity Bridge (Docker)
participant MCP as Python MCP Server
participant CLA as anthropics/claude-code-base-action
participant Rep as Reports/Artifacts
participant Sum as GitHub Job Summary
Dev->>GH: Start workflow (mini or main)
GH->>Sec: Detect secrets (unity_ok, anthropic_ok)
Sec-->>GH: Gate results
alt Secrets ok
GH->>Lic: Prime/verify Unity license caches
GH->>Dock: Start persistent unity-mcp container
GH->>Dock: Wait for readiness (ports/status)
GH->>MCP: Install/setup via Python (uv)
GH->>GH: Write .claude/mcp.json config
GH->>CLA: Run Claude NL suite (Unity params, model, tools)
CLA-->>GH: Produce JUnit XML + Markdown
GH->>GH: Normalize/duplicate JUnit, sanitize Markdown
GH->>Sum: Append NL/T summary
GH->>Rep: Publish JUnit for Actions
GH->>Rep: Upload artifacts (reports/*)
else Missing secrets
GH-->>Sum: Note gated/Skipped steps
end
GH->>Dock: Stop Unity container (cleanup)
GH-->>Dev: Workflow complete
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 💡 Knowledge Base configuration:
You can enable these sources in your CodeRabbit configuration. 📒 Files selected for processing (2)
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Greptile Summary
This PR introduces comprehensive CI workflows for automated testing of the Unity MCP bridge using Claude AI's natural language capabilities. The changes add two new GitHub Actions workflows that create sophisticated testing environments where Claude can perform automated code editing and validation on Unity C# scripts in real-time.
The implementation uses Docker containerization to run Unity Editor in headless mode with persistent MCP bridge connections, allowing Claude to interact with Unity projects through the Model Context Protocol. Both workflows (claude-nl-suite.yml and claude-nl-suite-mini.yml) follow the same architectural pattern but target different test suites - the full suite runs comprehensive tests while the mini version provides faster feedback for development.
Key technical components include:
- GameCI-based Unity licensing and setup within Docker containers
- Complex port discovery mechanisms to establish MCP bridge connectivity
- Python-based MCP server configuration and management
- Comprehensive logging with sensitive data sanitization
- JUnit test report generation for GitHub integration
- Robust error handling and cleanup procedures
The workflows integrate with the existing Unity MCP bridge infrastructure, leveraging the project's editor tools and runtime serialization components. This represents a significant enhancement to the project's testing capabilities, enabling automated validation of Unity-Claude interactions that would be difficult to test through traditional unit testing approaches.
Important Files Changed
Files Changed
| Filename | Score | Overview |
|---|---|---|
.github/workflows/claude-nl-suite-mini.yml |
3/5 | New mini test suite workflow with Docker Unity setup and Claude integration |
.github/workflows/claude-nl-suite.yml |
3/5 | Major rewrite of full test suite workflow with comprehensive Docker and MCP infrastructure |
Confidence score: 3/5
- This PR introduces complex CI infrastructure that may have reliability issues in production environments
- Score reflects concerns about Docker networking complexity, hardcoded paths, and references to non-existent prompt files
- Pay close attention to both workflow files for potential runtime failures and path resolution issues
Sequence Diagram
sequenceDiagram
participant User
participant GitHub Actions
participant Unity Container
participant MCP Server
participant Claude API
participant GameCI
User->>GitHub Actions: "Trigger workflow_dispatch"
GitHub Actions->>GitHub Actions: "Detect secrets (Unity & Anthropic)"
GitHub Actions->>GitHub Actions: "Checkout repository"
GitHub Actions->>GitHub Actions: "Setup Python (uv)"
GitHub Actions->>MCP Server: "Install MCP server dependencies"
GitHub Actions->>GameCI: "Prime Unity license"
GameCI->>GitHub Actions: "License activated"
GitHub Actions->>GitHub Actions: "Clean old MCP status files"
GitHub Actions->>Unity Container: "Start Unity (persistent bridge)"
Unity Container->>Unity Container: "Initialize Unity Editor in batch mode"
Unity Container->>Unity Container: "Execute StartAutoConnect method"
Unity Container->>GitHub Actions: "Bridge listening/ready signals"
GitHub Actions->>GitHub Actions: "Wait for Unity bridge (robust check)"
GitHub Actions->>Unity Container: "Probe connection on detected port"
Unity Container->>GitHub Actions: "Connection confirmed"
GitHub Actions->>GitHub Actions: "Write MCP config (.claude/mcp.json)"
GitHub Actions->>GitHub Actions: "Prepare reports directory"
GitHub Actions->>Claude API: "Run Claude NL suite (single pass)"
Claude API->>MCP Server: "Connect via stdio transport"
MCP Server->>Unity Container: "Execute Unity tools and commands"
Unity Container->>MCP Server: "Return Unity operation results"
MCP Server->>Claude API: "Provide Unity context and responses"
Claude API->>GitHub Actions: "Generate JUnit and markdown reports"
GitHub Actions->>GitHub Actions: "Normalize JUnit for consumer actions"
GitHub Actions->>GitHub Actions: "Sanitize markdown reports"
GitHub Actions->>GitHub Actions: "Publish JUnit reports"
GitHub Actions->>GitHub Actions: "Upload artifacts"
GitHub Actions->>Unity Container: "Stop Unity container"
Unity Container->>GitHub Actions: "Container stopped"
2 files reviewed, 1 comment
| if [ -f "${UNITY_CACHE_ROOT}/.local/share/unity3d/Unity_lic.ulf" ]; then | ||
| MANUAL_ARG=(-manualLicenseFile /root/.local/share/unity3d/Unity_lic.ulf) |
There was a problem hiding this comment.
logic: File path check uses hardcoded container path '/root/.local/share/unity3d/Unity_lic.ulf' but the license could be in host path '${UNITY_CACHE_ROOT}/.local/share/unity3d/Unity_lic.ulf'
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Bugbot free trial expires on August 31, 2025
Learn more in the Cursor dashboard.
| - name: NL/T details → Job Summary | ||
| if: always() | ||
| run: | | ||
| echo "## Unity NL/T Editing Suite — Full Coverage" >> $GITHUB_STEP_SUMMARY |
There was a problem hiding this comment.
Bug: Workflow Summary Mismatch
The claude-nl-suite-mini.yml workflow's job summary incorrectly displays "Full Coverage." As this workflow is for the mini test suite, this claim is misleading about the actual scope of the tests.
…nit)
Summary by CodeRabbit
Tests
Chores