-
Notifications
You must be signed in to change notification settings - Fork 1.9k
[XEXPR] Refactor test skills/agents to dispatcher pattern #33721
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com>
- Rename write-tests skill to write-ui-tests - Rename uitest-coding-agent to write-tests-agent (dispatcher) - Add write-xaml-tests skill (credit: Stephane Delcroix) - Update all references across README-AI.md, copilot-instructions.md, pr.md, etc. - Fix pr-finalize to reference write-tests-agent instead of specific skill
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Refactors the repo’s Copilot test-writing setup from a single UI-test-focused agent/skill into a dispatcher agent (write-tests-agent) plus separate skills for UI tests and XAML unit tests.
Changes:
- Adds
write-tests-agentdispatcher and introduceswrite-ui-tests+write-xaml-testsskills (removing the oldwrite-testsskill anduitest-coding-agent). - Updates documentation and guidance files to reference the new agent/skill names and usage patterns.
- Expands UI test authoring guidance in
.github/instructions/uitests.instructions.md.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/skills/write-xaml-tests/SKILL.md | New skill for creating XAML unit tests in Controls.Xaml.UnitTests. |
| .github/skills/write-ui-tests/SKILL.md | New skill for creating UI tests and verifying they reproduce issues. |
| .github/skills/write-tests/SKILL.md | Removes the old monolithic write-tests skill. |
| .github/skills/pr-finalize/SKILL.md | Updates references away from old test-writing entry points. |
| .github/instructions/uitests.instructions.md | Adds/updates UI test guidance (notably screenshot verification guidance). |
| .github/instructions/sandbox.instructions.md | Updates “when not to use” guidance to point to write-tests-agent. |
| .github/instructions/agents.instructions.md | Updates agent naming examples to write-tests-agent. |
| .github/copilot-instructions.md | Updates agent/skill catalog and delegation examples for new architecture. |
| .github/agents/write-tests-agent.md | Adds dispatcher agent that routes to UI vs XAML test-writing skills. |
| .github/agents/uitest-coding-agent.md | Removes the old UI-test-specific agent definition. |
| .github/agents/pr.md | Updates PR agent references to the new test-writing skill names. |
| .github/README-AI.md | Updates the “how to use agents” README to reflect new agent/skills. |
.github/agents/write-tests-agent.md
Outdated
| 1. Read the XAML unit test guidelines | ||
| 2. Create XAML and code-behind files | ||
| 3. Build and run the test | ||
|
|
||
| **🛑 CRITICAL:** The skill enforces that tests must FAIL before reporting success. A passing test does NOT prove it catches the bug. |
Copilot
AI
Jan 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The agent text says the invoked skill “enforces that tests must FAIL before reporting success.” That’s true for write-ui-tests, but write-xaml-tests currently allows both reproduction tests (fail before fix) and regression tests (pass) and does not include the same hard requirement. Please reword this to apply only to UI tests, or update write-xaml-tests to explicitly enforce the same failure-first rule if that’s intended.
.github/copilot-instructions.md
Outdated
| 2. **write-tests-agent** - Agent for writing tests. Determines test type and invokes appropriate skill (currently UI tests via write-ui-tests) | ||
| - **Use when**: Creating new tests for issues or PRs | ||
| - **Capabilities**: Test type determination, skill invocation, test verification |
Copilot
AI
Jan 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This description still says write-tests-agent currently supports only UI tests via write-ui-tests, but the PR adds write-xaml-tests and write-tests-agent.md claims XAML dispatch support. Please update this section to reflect current capabilities (UI + XAML) to avoid confusing users and the delegation logic.
| 2. **write-tests-agent** - Agent for writing tests. Determines test type and invokes appropriate skill (currently UI tests via write-ui-tests) | |
| - **Use when**: Creating new tests for issues or PRs | |
| - **Capabilities**: Test type determination, skill invocation, test verification | |
| 2. **write-tests-agent** - Agent for writing tests. Determines test type (UI vs XAML) and invokes the appropriate skill (`write-ui-tests`, `write-xaml-tests`) | |
| - **Use when**: Creating new tests for issues or PRs | |
| - **Capabilities**: Test type determination (UI and XAML), skill invocation, test verification |
- Add retryTimeout parameter to VerifyScreenshot (from PR #33705) - Fix regex to match both '.' and ',' decimal separators - Update write-tests-agent to clarify XAML vs UI test behavior - Update copilot-instructions to reflect XAML test support
|
/azp run maui-pr-uitests, maui-pr-devicetests |
|
Azure Pipelines successfully started running 2 pipeline(s). |
> [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ## Summary Refactors the test-writing infrastructure from a monolithic agent to a dispatcher + skills architecture for better maintainability and extensibility. ## Changes ### Skills - **Renamed** `write-tests` → `write-ui-tests` (clearer naming) - **Added** `write-xaml-tests` skill for XAML unit tests (credit: @StephaneDelcroix) ### Agents - **Renamed** `uitest-coding-agent` → `write-tests-agent` - **Simplified** from 635 lines to 79 lines (dispatcher pattern) - Agent now determines test type and invokes appropriate skill ### References Updated - README-AI.md - copilot-instructions.md - pr.md - sandbox.instructions.md - agents.instructions.md - pr-finalize/SKILL.md ## Architecture | Before | After | |--------|-------| | `uitest-coding-agent.md` (635 lines) | `write-tests-agent.md` (79 lines) - dispatcher | | `write-tests/SKILL.md` | `write-ui-tests/SKILL.md` - UI tests | | — | `write-xaml-tests/SKILL.md` - XAML unit tests | ## Future Work - Wire `write-xaml-tests` into `write-tests-agent` dispatch table - Add additional test type skills (unit tests, integration tests) --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> Co-authored-by: Shane Neuville <shneuvil@microsoft.com>
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Summary
Refactors the test-writing infrastructure from a monolithic agent to a dispatcher + skills architecture for better maintainability and extensibility.
Changes
Skills
write-tests→write-ui-tests(clearer naming)write-xaml-testsskill for XAML unit tests (credit: @StephaneDelcroix)Agents
uitest-coding-agent→write-tests-agentReferences Updated
Architecture
uitest-coding-agent.md(635 lines)write-tests-agent.md(79 lines) - dispatcherwrite-tests/SKILL.mdwrite-ui-tests/SKILL.md- UI testswrite-xaml-tests/SKILL.md- XAML unit testsFuture Work
write-xaml-testsintowrite-tests-agentdispatch table