You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
assets/hooks/skill_prehook.py — UserPromptSubmit pre-hook: keyword-matches mutation-testing prompts, injects SKILL.md (or a short directive with MUTATION_TESTING_HOOK_FULL=0), always exits 0, logs failures loudly to stderr. Triggers deliberately exclude generic test terms so ordinary testing prompts stay silent.
AGENTS.md — documents the mutation-testing-engineer role (owns test-suite quality per the "every test must fail if the real logic is wrong" rule) and the skills-src → install + pre-hook convention.
Machine-local setup (not in this diff, mirrors the other two skills): skill copied to ~/.agents/skills/mutation-testing/ and hook registered in ~/.kimi-code/config.toml.
Verification
Hook trigger tests: fires on check my mutation score / triage the surviving mutants (JSON and raw-text stdin); silent on fix the navbar and generic add a unit test for the login form; empty stdin exits 0; HOOK_FULL=0 emits the short directive.
codacy-analysis analyze on the changed files: 0 issues (Semgrep, Trivy).
npm run check: 0 errors, 0 warnings. npm run build: green. npm run test: 45 files / 363 tests pass.
Notes
No mutation runner (Stryker) is added — tooling adoption and CI thresholds are a separate, maintainer-approved step.
docs-agent-roles also inserts an Agent Split section into AGENTS.md; whichever PR merges second may need a small rebase — sections are adjacent but independent.
CodeAnt-AI Description
Add mutation-testing guidance and automatically apply it to relevant agent prompts
What Changed
Adds a mutation-testing skill covering score interpretation, survivor triage, behavior-focused test design, supported tools, and CI practices
Adds a prompt hook that injects the guidance only for mutation-testing and test-quality requests, while staying silent for ordinary testing prompts
Supports JSON, encoded-string, and plain-text prompts, with a concise fallback when the skill file cannot be read
Logs hook and skill-loading failures without interrupting the prompt flow
Documents the mutation-testing role and local skill installation process in AGENTS.md
Impact
✅ Mutation-testing guidance available during relevant agent tasks ✅ Fewer unnecessary skill injections for ordinary testing requests ✅ Prompt processing continues despite hook or skill-file errors
💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
@codeant-ai ask: Your question here
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
@codeant-ai ask: Can you suggest a safer alternative to storing this secret?
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
@codeant-ai: Your feedback here
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
@codeant-ai: Do not flag unused imports.
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
@codeant-ai: review
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.
Added mutation-testing guidance covering concepts, scoring, workflows, triage, CI strategy, and best practices.
Added support for automatically recognizing mutation-testing requests and providing relevant guidance.
Added language-specific tooling references for configuring, running, and optimizing mutation tests.
Documentation
Expanded engineering guidance for reviewing test quality and validating exact code mutations.
Added practical references for analyzing surviving mutants and using feedback loops.
Walkthrough
Adds a mutation-testing engineering skill, supporting references, a prompt-submit hook, and repository instructions for installation, activation, execution, triage, and CI use.
This PR adds a prompt hook that detects mutation-testing requests, loads the repository skill, and injects its guidance into the agent context while preserving a successful hook exit.
sequenceDiagram
participant User
participant Harness
participant Prehook
participant Skill file
participant Agent
User->>Harness: Submit prompt
Harness->>Prehook: Run with prompt
Prehook->>Prehook: Match mutation testing terms
Prehook->>Skill file: Load mutation testing skill
Skill file-->>Prehook: Return skill guidance
Prehook-->>Harness: Inject guidance into context
Harness-->>Agent: Provide enriched prompt context
Prehook-->>Harness: Exit successfully
Add mutation-testing skill source, prehook, and role documentation
✨ Enhancement📝 Documentation🕐 20-40 Minutes
AI Description
• Add a repo-local mutation-testing skill with guidance on scoring, triage, and CI ratcheting.
• Introduce a UserPromptSubmit prehook that injects the skill only on mutation-testing-specific
prompts.
• Document the mutation-testing engineer role and the skills-src → install + prehook convention.
Diagram
graph TD
U(["User prompt"]) --> H["skill_prehook.py"] --> C(["Injected context"])
H --> S["SKILL.md"] --> R["references/*.md"]
C --> D["AGENTS.md"]
Loading
High-Level Assessment
The following are alternative approaches to this PR:
1. Centralize hook logic into a shared hook utility
➕ Avoids trigger/IO/exit-code logic duplication across skills
➕ Enables consistent logging, env var conventions, and future unit tests
➖ Adds a shared dependency that all skills must track
➖ Slightly more repo structure/complexity than per-skill scripts
2. Drive triggers from SKILL.md frontmatter (single source of truth)
➕ Keeps trigger vocabulary co-located with the skill definition
➕ Reduces risk of triggers diverging from the skill description
➖ Requires parsing YAML frontmatter in the hook
➖ Harder to keep regex-safe trigger patterns without additional conventions
Recommendation: The PR’s approach (a per-skill prehook mirroring existing skills, with mutation-specific triggers and a FULL vs directive mode) is the right pragmatic choice for now: it’s self-contained, low-risk, and consistent with the established convention. If more skills are added, consider centralizing common hook logic to reduce drift and make trigger behavior easier to test.
• Introduces a mutation-testing skill document covering the mental model, score interpretation, a step-by-step workflow, mental mutation testing for small diffs, an agent feedback loop, CI strategies, and common anti-patterns. Links to supporting references for tooling and survivor triage.
agent-mutation-loop.mdDocument an agent mutation-feedback loop for improving AI-written tests+69/-0
Document an agent mutation-feedback loop for improving AI-written tests
• Adds guidance on using surviving mutants as high-signal feedback for iterative test generation, including prompt patterns, validation rules (pass on original / fail on mutant), stopping rules, and a manual adversarial mode when tools aren’t available.
surviving-mutant-triage.mdAdd survivor triage guide (coverage vs equivalent vs genuine gaps)+74/-0
Add survivor triage guide (coverage vs equivalent vs genuine gaps)
• Provides a structured triage flow for surviving mutants, heuristics for equivalent mutants, an operator-to-missing-test map, priority tiers, and a concrete kill-test recipe with both-direction verification.
tools-by-language.mdAdd cross-language mutation testing tooling and CI patterns+202/-0
Add cross-language mutation testing tooling and CI patterns
• Adds a language-to-tool selection table plus practical setup/workflow notes for StrykerJS, mutmut/Cosmic Ray, cargo-mutants, PIT, Infection, and others. Includes a universal CI pattern for PR-diff runs plus scheduled full sweeps with artifact uploads and caching guidance.
AGENTS.mdDocument mutation-testing engineer role and skill installation convention+23/-0
Document mutation-testing engineer role and skill installation convention
• Adds a dedicated role section defining responsibility for test-suite quality via mutation testing and points to the new skill source. Documents the repo convention for skills-src installation and prehook registration, including the 'always exit 0, log loudly' hook rule.
skill_prehook.pyAdd UserPromptSubmit prehook to inject mutation-testing skill on keyword match+117/-0
Add UserPromptSubmit prehook to inject mutation-testing skill on keyword match
• Adds a Python prehook that reads JSON or raw-text stdin, matches mutation-testing-specific keywords, and injects either the full SKILL.md or a short directive (configurable via MUTATION_TESTING_HOOK_FULL). The hook always exits 0 and logs unexpected failures to stderr to avoid breaking prompt flow.
The reason will be displayed to describe this comment to others. Learn more.
This PR adds well-structured mutation testing documentation and a pre-hook system. The skill documentation is comprehensive and follows repository conventions. I identified one critical issue that blocks merge:
Critical Issue:
The Python hook silently catches and suppresses OSError without logging, violating the repository's "Always fail loudly" rule. This makes debugging impossible when the skill file is missing or unreadable.
Verification Confirmation:
Per the PR description, all validation checks pass (codacy-analysis, npm run check/build/test). Hook trigger tests confirm proper behavior.
Note:
No mutation runner is installed per the PR scope - tool adoption is correctly deferred to a separate, maintainer-approved step.
You can now have the agent implement changes and create commits directly on your pull request's source branch. Simply comment with /q followed by your request in natural language to ask the agent to make changes.
The reason will be displayed to describe this comment to others. Learn more.
🛑 Silent Failure Violation: OSError is caught and suppressed without logging. This violates the repository's "Always fail loudly" rule (AGENTS.md lines 9-11). When the skill file cannot be read, users receive the path directive without knowing the file is actually missing or unreadable, making debugging impossible.
Suggested change
exceptOSError:
pass# fall through to path directive
exceptOSErrorasfile_error:
msg=f"mutation-testing prehook: skill file not readable: {skill_path} ({file_error})\n"
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
The pull request is currently not up to standards. While it provides comprehensive documentation for the mutation-testing skill, the implementation of the skill_prehook.py script has critical issues that should prevent merging. Specifically, the logic for parsing prompts from stdin fails to handle JSON string literals, which will cause valid user inputs to be ignored in certain environments.
Furthermore, the script is flagged as complex and lacks any automated unit tests to verify its keyword matching, environment variable toggles, or error handling. There is also significant code duplication between this hook and other existing skill hooks, which should be addressed by refactoring shared logic into a common utility to ensure maintainability.
About this PR
Despite the complexity of the skill_prehook.py script (handling regex triggers, JSON parsing, and environment variables), no automated unit tests have been provided. Automated verification is required to ensure keyword matching is precise and that the hook behaves correctly under different environment configurations.
Test suggestions
Verify keyword matching: triggers on specific terms (e.g., 'mutation score', 'mutmut', 'weak assertion') and remains silent on generic ones (e.g., 'test', 'coverage').
Verify stdin parsing: correctly extracts the prompt from multiple possible JSON keys ('prompt', 'user_prompt', 'message', 'text') and handles raw text fallback.
Verify toggle behavior: injects the full SKILL.md when MUTATION_TESTING_HOOK_FULL=1 (default) and only the path-based directive when set to 0.
Verify error handling: confirms the hook exits with 0 and prints to stderr if the skill file is missing or if JSON parsing fails on valid-but-incorrectly-shaped input.
Verify path resolution: confirms the hook correctly resolves the relative path to SKILL.md based on its installation directory.
Automated unit test coverage for logic in .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Verify keyword matching: triggers on specific terms (e.g., 'mutation score', 'mutmut', 'weak assertion') and remains silent on generic ones (e.g., 'test', 'coverage').
2. Verify stdin parsing: correctly extracts the prompt from multiple possible JSON keys ('prompt', 'user_prompt', 'message', 'text') and handles raw text fallback.
3. Verify toggle behavior: injects the full SKILL.md when MUTATION_TESTING_HOOK_FULL=1 (default) and only the path-based directive when set to 0.
4. Verify error handling: confirms the hook exits with 0 and prints to stderr if the skill file is missing or if JSON parsing fails on valid-but-incorrectly-shaped input.
5. Verify path resolution: confirms the hook correctly resolves the relative path to SKILL.md based on its installation directory.
6. Automated unit test coverage for logic in .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py
The reason will be displayed to describe this comment to others. Learn more.
🟡 MEDIUM RISK
The function ignores inputs that are valid JSON string literals. While it catches raw text via JSONDecodeError, a prompt passed as a quoted JSON string would result in data being a string literal, failing the current dictionary check and returning an empty string. Try running the following prompt in your coding agent: > Update the read_prompt function in .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py to return data if it is a string after JSON parsing.
The reason will be displayed to describe this comment to others. Learn more.
🟡 MEDIUM RISK
Suggestion: This logic for reading prompts and resolving skill paths is duplicated across several engineering skills. Maintaining multiple copies of this boilerplate makes it harder to update the agent hook protocol or fix bugs in environment handling. Consider moving these utilities to a shared helper module or using a template to generate these hooks.
Try running the following prompt in your IDE agent:
Refactor the skill_prehook.py files in the .agents/skills-src/ directory to use a shared logic module for read_prompt and environment variable resolution, reducing duplication across the mutation-testing, drizzle-engineering, and sqlite-engineering skills.
The reason will be displayed to describe this comment to others. Learn more.
⚪ LOW RISK
Nitpick: Try running the following prompt in your coding agent: > In .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py, add optional pluralization and trailing word boundaries (\b) to the 'killed mutant' and 'equivalent mutant' trigger regexes.
The reason will be displayed to describe this comment to others. Learn more.
⚪ LOW RISK
Nitpick: The multi-line docstring summary should start on the same line as the opening triple quotes to comply with standard Python documentation conventions.
The mutation-testing prehook includes ambiguous trigger regexes (e.g., matching any occurrence of
“mutant(s)”) that can inject the full SKILL.md into unrelated prompts, bloating context and reducing
relevance. This contradicts the hook’s own intent to avoid firing on generic testing prompts and
repeats a previously accepted “too-broad prehook triggers” bug pattern.
ⓘ Recommendations generated based on similar findings in past PRs
Evidence
The hook claims it is tuned for mutation-testing-specific vocabulary, but the trigger list contains
standalone generic patterns that will match many non-mutation prompts; this is the same failure mode
described in the historical prehook-trigger bug (PR #51).
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
The mutation-testing prehook matches overly generic terms (notably `\bmutants?\b`), which can cause the hook to fire on unrelated prompts and inject the full skill into context.
## Issue Context
The file comment explicitly says triggers are mutation-testing-specific and generic test terms are excluded to avoid injecting the full skill into ordinary testing prompts.
## Fix Focus Areas
- .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py[31-46]
### Suggested changes
- Remove or constrain `r"\bmutants?\b"` so it requires mutation-testing context (examples: require `mutation` within the same phrase, or replace with more specific phrases like `surviving mutant(s)`, `killed mutant`, `equivalent mutant`, `mutation score`, etc.).
- Consider tightening other ambiguous tool-name words that overlap common idioms (e.g., `gremlins`) by requiring surrounding mutation-testing context.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Newly added .md and .py files begin with content (front matter, headings, or a docstring) and do
not include an AGPL license header as the first non-empty, non-shebang line. This can cause
license/audit noncompliance for newly introduced source and documentation files.
+---+name: mutation-testing+description: 'Mutation testing engineering — verify that a test suite actually catches bugs, not just executes lines. Use when auditing or hardening test-suite quality, reviewing test coverage claims ("we have 90% coverage"), hunting surviving mutants, writing tests that kill specific mutants, setting up or configuring mutation tools (Stryker/StrykerJS/Stryker.NET, mutmut, Cosmic Ray, PIT/pitest, Infection, cargo-mutants, go-mutesting/Gremlins, mutant, muter, Mull), wiring mutation testing into CI (incremental PR runs, thresholds, --since/--in-diff), interpreting mutation scores, handling equivalent/timeout/no-coverage mutants, or closing the mutation-feedback loop on AI-generated tests. Triggers on: mutation testing, mutation score, surviving mutants, killed mutants, equivalent mutants, are my tests actually good, test suite quality, weak assertions, mutation coverage, mutant.'+---
Relevance
●●● Strong
AGPL header enforcement was accepted previously (added header to new migration) in PR #23.
ⓘ Recommendations generated based on similar findings in past PRs
Evidence
PR Compliance ID 2401185 requires every newly added comment-capable file to start with an AGPL
license header. The cited new files start immediately with YAML front matter / headings or a Python
docstring and contain no AGPL header at the top.
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
Newly added comment-capable files are missing the required AGPL license header at the top of the file.
## Issue Context
Compliance requires an AGPL header notice in each newly added file that supports comments (including `.md` and `.py`). These new files currently start with front matter/headings or a docstring instead.
## Fix Focus Areas
- .agents/skills-src/mutation-testing/SKILL.md[1-5]
- .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py[1-10]
- .agents/skills-src/mutation-testing/references/agent-mutation-loop.md[1-8]
- .agents/skills-src/mutation-testing/references/surviving-mutant-triage.md[1-8]
- .agents/skills-src/mutation-testing/references/tools-by-language.md[1-8]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
3. OSError swallowed without logging✓ Resolved📘 Rule violation☼ Reliability
Description
except OSError: pass silently ignores failures when reading SKILL.md, so when full injection is
enabled the hook produces only a fallback directive without any stderr diagnostics or context. This
makes install/misconfiguration problems harder to diagnose and conflicts with the requirement (and
documented convention) that caught exceptions should be surfaced via contextual logging rather than
silently ignored.
ⓘ Recommendations generated based on similar findings in past PRs
Evidence
PR Compliance ID 2401647 requires catch blocks to rethrow, return an error result, or log with
context before falling back, yet in
.agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py the hook catches OSError while
reading SKILL.md and executes pass, resulting in no logging and an otherwise silent failure
path. Separately, AGENTS.md states that hooks “log failures loudly to stderr,” but this hook
suppresses common file-read/open errors with except OSError: pass, leaving stderr empty and
creating a mismatch between documented behavior and actual implementation (with similar suppression
patterns noted in other hooks).
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
The mutation-testing prehook catches `OSError` when reading `SKILL.md` (`except OSError: pass`) and silently falls through, producing no stderr diagnostics or contextual signal before emitting its fallback path directive.
## Issue Context
Hooks must exit 0, but compliance requires caught exceptions to be surfaced via logging and/or explicit fallback behavior rather than being silently ignored. Additionally, `AGENTS.md` documents a convention that hooks “log failures loudly to stderr,” which this prehook currently violates on the common file-read/open failure path.
## Fix Focus Areas
- .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py[89-107]
- AGENTS.md[31-37]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
The new Python hook uses space-based indentation, violating the requirement to indent with tabs
only. This can cause inconsistent formatting and tooling conflicts where tabs are mandated.
ⓘ Recommendations generated based on similar findings in past PRs
Evidence
PR Compliance ID 2401137 requires indentation to be tabs, not spaces. In the new hook file, indented
blocks (e.g., inside read_prompt) are space-indented.
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
The new hook file uses spaces for indentation, but the compliance rule requires tabs only.
## Issue Context
This is a newly added source file and is fully in-scope for formatting compliance.
## Fix Focus Areas
- .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py[63-107]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
description: 'Mutation testing engineering — verify that a test suite actually catches bugs, not just executes lines. Use when auditing or hardening test-suite quality, reviewing test coverage claims ("we have 90% coverage"), hunting surviving mutants, writing tests that kill specific mutants, setting up or configuring mutation tools (Stryker/StrykerJS/Stryker.NET, mutmut, Cosmic Ray, PIT/pitest, Infection, cargo-mutants, go-mutesting/Gremlins, mutant, muter, Mull), wiring mutation testing into CI (incremental PR runs, thresholds, --since/--in-diff), interpreting mutation scores, handling equivalent/timeout/no-coverage mutants, or closing the mutation-feedback loop on AI-generated tests. Triggers on: mutation testing, mutation score, surviving mutants, killed mutants, equivalent mutants, are my tests actually good, test suite quality, weak assertions, mutation coverage, mutant.'
Newly added .md and .py files begin with content (front matter, headings, or a docstring) and do
not include an AGPL license header as the first non-empty, non-shebang line. This can cause
license/audit noncompliance for newly introduced source and documentation files.
Agent Prompt
## Issue description
Newly added comment-capable files are missing the required AGPL license header at the top of the file.
## Issue Context
Compliance requires an AGPL header notice in each newly added file that supports comments (including `.md` and `.py`). These new files currently start with front matter/headings or a docstring instead.
## Fix Focus Areas
- .agents/skills-src/mutation-testing/SKILL.md[1-5]
- .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py[1-10]
- .agents/skills-src/mutation-testing/references/agent-mutation-loop.md[1-8]
- .agents/skills-src/mutation-testing/references/surviving-mutant-triage.md[1-8]
- .agents/skills-src/mutation-testing/references/tools-by-language.md[1-8]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
@Amazon-Q-Developer[bot], @qodo-code-review[bot] (Add server-side moderation pipeline #3), @CodeAnt-AI[bot] — except OSError: pass silent fallback: valid, violates the repo's loud-failure rule. 21089e4 logs skill file not readable: <path> (<error>) to stderr before falling through to the path directive; exit still 0. Reproduced silent (0 bytes stderr) before, loud after.
@codacy-production[bot] — read_prompt drops JSON string literals (MEDIUM): valid bug. A bare JSON string (e.g. "check my mutation score") parsed to str, failed the dict check, and was silently ignored. 29465ca returns it directly; docstring updated. Reproduced 0 bytes out before, full injection after.
@qodo-code-review[bot] (Phase D: Auth and cron routes #4) — overbroad triggers \bmutants?\b, gremlins: valid, matches the accepted PR chore: drizzle-engineering skill source + pre-hook #51 narrowing precedent. Reproduced: "my favorite X-Men mutant is Wolverine" injected the full skill. 29465ca removes the bare terms; mutant now only fires in mutation phrases (surviving/killed/equivalent/kill … mutants).
@codacy-production[bot] — pluralize killed/equivalent mutant (LOW): valid, trivial. 29465ca → killed mutants? / equivalent mutants? with word boundaries.
Verification after fixes: 13 trigger/silence/stderr behavior checks all correct, codacy-analysis clean (Trivy + Semgrep, 0 issues).
Declined — with reasons
@codacy-production[bot] — shared hook module across skills: valid observation, wrong PR. The two sibling skills are already merged, each hook is self-contained by design (must survive a standalone cp install), and minimal-change rules apply. Candidate for a dedicated follow-up branch.
@qodo-code-review[bot] (Scaffold Drizzle/libSQL and adapter-node setup #1) — AGPL headers on skill files: conflicts with precedent — both merged skill PRs carry no headers and were maintainer-approved; additionally SKILL.md's YAML front matter must occupy line 1 for skill loaders. Deferring to maintainer decision.
@codacy-production[bot] — docstring-first-line nitpick: conflicts with the merged sibling hooks' identical layout; consistency wins.
@codacy-production[bot] — automated unit tests for the hook: no Python test infrastructure exists in this repo; PR chore: drizzle-engineering skill source + pre-hook #51 set the documented-manual-verification precedent for hooks. Follow-up candidate if pytest wiring is wanted.
Not actionable
@CodeAnt-AI[bot] quality gate PASSED, @sonarqubecloud[bot] quality gate PASSED (0 new issues), @coderabbitai[bot] rate-limited (no review produced), @netlify[bot] deploy preview informational.
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py:
- Around line 103-129: Update both exception handlers in skill_prehook.py’s main
flow to retain stderr logging and also write a concise failure directive to
stdout so injected-output harnesses show the error while exiting successfully.
Update AGENTS.md at the documented hook behavior section to describe both
server-side stderr logging and the user-visible stdout error directive; apply
the change at .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py
lines 103-129 and document it at AGENTS.md lines 36-37.
- Around line 48-50: Restrict the trigger pattern in the prehook’s
mutation-testing prompt detection to requests containing an explicit
mutation-testing term; remove or revise the broad r"\bare my tests\b" pattern
while preserving the existing mutation-testing and weak-assertion triggers.
In @.agents/skills-src/mutation-testing/references/agent-mutation-loop.md:
- Around line 26-37: Update the prompt code fence in the mutation-testing
reference content to use the text language tag after the opening backticks,
preserving the prompt wording and all existing test instructions.
In @.agents/skills-src/mutation-testing/references/surviving-mutant-triage.md:
- Around line 50-59: Update the ordered lists in the mutation-testing triage
guidance so numbering restarts at 1 under each heading: use 1–3 for both “Kill
when practical” and “Accept or exclude,” preserving all existing list items and
wording.
In @.agents/skills-src/mutation-testing/references/tools-by-language.md:
- Line 13: Update the Java/JVM PIT entry in tools-by-language.md to remove the
obsolete scmMutationCoverage Maven goal; reference Arcmutate’s Git integration
instead, or explicitly pin and label a PIT version older than 1.18.0 wherever
this guidance is referenced.
- Line 46: Update the StrykerJS cache references in the mutation-testing
documentation to use the actual default incremental results file,
reports/stryker-incremental.json, instead of temporary .stryker-tmp paths;
alternatively, configure the shown StrykerJS setup with an explicit
incrementalFile value and keep all references consistent.
- Around line 97-115: Update the mutmut workflow commands in the
tools-by-language reference to use mutants/ for mutation state and mutmut browse
for inspection and retesting. Replace the deprecated mutmut results and mutmut
result-ids commands, including the survivor re-test flow, with the supported CLI
equivalents; alternatively pin a mutmut version that supports the existing
commands.
- Around line 9-10: Update both mutmut configuration examples in
tools-by-language.md to use the current 3.x schema: replace paths_to_mutate with
source_paths and tests_dir with pytest_add_cli_args_test_selection, using arrays
for pyproject.toml and strings for setup.cfg. Do not leave examples using the
legacy keys unless the mutmut dependency is explicitly pinned to a compatible
legacy version.
In @.agents/skills-src/mutation-testing/SKILL.md:
- Around line 12-18: Update the mutation-testing status definitions and
mutation-score formula in the documentation around the listed statuses to be
explicitly scoped to the supported mutation-testing tool(s). Either define
repository-wide policy or provide per-tool mappings, including Stryker’s
NoCoverage, Timeout, CompileError, and RuntimeError treatment, and ensure the
score calculation matches each tool’s semantics.
- Around line 1-4: Add the approved license header using Markdown comment syntax
to .agents/skills-src/mutation-testing/SKILL.md at lines 1-4, placing it before
the YAML frontmatter, and add the same header to
.agents/skills-src/mutation-testing/references/tools-by-language.md at lines
1-3, .agents/skills-src/mutation-testing/references/agent-mutation-loop.md at
lines 1-3, and
.agents/skills-src/mutation-testing/references/surviving-mutant-triage.md at
lines 1-3.
- Line 18: Update .agents/skills-src/mutation-testing/SKILL.md:18 and :48 to
cite Yao, Harman, and Jia for the ~23% equivalent-mutant estimate, including its
basis of 18 programs and 4,181 mutants, and correct the ~50-point claim to
AdverTest’s 50% FDR drop for the w/o Iter ablation on Defects4J rather than
MuTAP. Update
.agents/skills-src/mutation-testing/references/agent-mutation-loop.md:7-9 and
:50 to scope ~89.5% MUTGEN to HumanEval-Java and 73% ACH to its reported
acceptance study, and remove or qualify the unsupported ~7× token-use claim.
Update
.agents/skills-src/mutation-testing/references/surviving-mutant-triage.md:24 to
describe 15–25% as a context-dependent estimate, not an established rate for
survivors in mature triage.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
Push a commit to this branch (recommended)
Create a new PR with the fixes
ℹ️ Review info⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 78095534-81d8-4373-bcb3-c291ebeaccb7
📥 Commits
Reviewing files that changed from the base of the PR and between 8f59d6a and 29465ca.
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Restrict this trigger to mutation-testing prompts.
r"\bare my tests\b" matches ordinary prompts such as “Are my tests passing?”. It injects the full skill without a mutation-testing request. Remove this trigger or require an explicit mutation-testing term.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py around
lines 48 - 50, Restrict the trigger pattern in the prehook’s mutation-testing
prompt detection to requests containing an explicit mutation-testing term;
remove or revise the broad r"\bare my tests\b" pattern while preserving the
existing mutation-testing and weak-assertion triggers.
Make hook failures visible in the injected output.
The hook logs failures only to stderr, and AGENTS.md documents that behavior. This does not reliably show an error to the user in stdout-injecting harnesses.
.agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py#L103-L129: Write a concise failure directive to stdout in both exception handlers.
AGENTS.md#L36-L37: Document both stderr logging and the stdout user-visible error behavior.
As per coding guidelines, fallbacks must log server-side and show an error to the user.
🧰 Tools🪛 Ruff (0.16.0)
[warning] 125-125: Do not catch blind exception: Exception
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py around
lines 103 - 129, Update both exception handlers in skill_prehook.py’s main flow
to retain stderr logging and also write a concise failure directive to stdout so
injected-output harnesses show the error while exiting successfully. Update
AGENTS.md at the documented hook behavior section to describe both server-side
stderr logging and the user-visible stdout error directive; apply the change at
.agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py lines 103-129
and document it at AGENTS.md lines 36-37.
Use text after the opening fence. This resolves the MD040 warning.
🧰 Tools🪛 markdownlint-cli2 (0.23.1)
[warning] 26-26: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agents/skills-src/mutation-testing/references/agent-mutation-loop.md around
lines 26 - 37, Update the prompt code fence in the mutation-testing reference
content to use the text language tag after the opening backticks, preserving the
prompt wording and all existing test instructions.
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agents/skills-src/mutation-testing/references/surviving-mutant-triage.md
around lines 50 - 59, Update the ordered lists in the mutation-testing triage
guidance so numbering restarts at 1 under each heading: use 1–3 for both “Kill
when practical” and “Accept or exclude,” preserving all existing list items and
wording.
mutmut current documentation configuration source_paths pytest_add_cli_args_test_selection pyproject.toml version 3
💡 Result:
In mutmut version 3, you can configure the mutation testing system using a pyproject.toml file by defining a [tool.mutmut] section [1][2]. For configuring source paths and pytest arguments in pyproject.toml, you must specify them as arrays [1][3]. The relevant configuration keys are: - source_paths: Defines the directories or files that mutmut should mutate [1][2]. - pytest_add_cli_args_test_selection: Used for passing pytest command-line arguments that are specifically responsible for selecting or deselecting tests (e.g., -m, -k) [4][5]. - pytest_add_cli_args: Used for passing other pytest command-line arguments, such as disabling plugins or overriding pytest configuration settings [4][6]. Example configuration in pyproject.toml: [tool.mutmut] source_paths = [ "src/" ] pytest_add_cli_args_test_selection = [ "-m", "not fail", "-k", "test_include" ] pytest_add_cli_args = [ "-p", "no:some_plugin" ] Changes to these configuration settings, as well as others like type_check_command or timeout settings, are automatically detected by mutmut, which will then invalidate the affected cached results [4][6]. Alternative configuration files such as setup.cfg are also supported, where these values are provided as strings rather than arrays [1][2][4].
#!/bin/bashset -e
python3 - <<'PY'import jsonimport urllib.requestfor url in ( "https://pypi.org/pypi/mutmut/json", "https://raw.githubusercontent.com/boxed/mutmut/main/README.rst",): print(f"--- {url} ---") with urllib.request.urlopen(url, timeout=15) as response: data = response.read().decode() if url.endswith("/json"): payload = json.loads(data) print("latest:", payload["info"]["version"]) print("requires_python:", payload["info"].get("requires_python")) else: lines = data.splitlines() needles = ( "source_paths", "paths_to_mutate", "pytest_add_cli_args_test_selection", "tests_dir", "runner", "[tool.mutmut]", "[mutmut]", ) for i, line in enumerate(lines): if any(needle in line for needle in needles): lo, hi = max(0, i - 4), min(len(lines), i + 5) print(f"{lo + 1}-{hi}:") print("\n".join(f"{j + 1}: {lines[j]}" for j in range(lo, hi)))PY
#!/bin/bashset -e
python3 - <<'PY'import reimport tomllibfrom pathlib import Pathlocal = Path(".agents/skills-src/mutation-testing/references/tools-by-language.md").read_text()toml_block = re.search(r"```toml\n(.*?)\n```", local, re.S).group(1)config = tomllib.loads(toml_block)["tool"]["mutmut"]current_keys = {"source_paths", "pytest_add_cli_args_test_selection"}legacy_keys = {"paths_to_mutate", "tests_dir"}print("local TOML keys:", sorted(config))print("legacy keys present:", sorted(set(config) & legacy_keys))print("current keys present:", sorted(set(config) & current_keys))print("TOML values are arrays for current path-selection keys:", all(isinstance(config[k], list) for k in current_keys if k in config))setup_block = re.search(r"```ini\n(.*?)\n```", local, re.S).group(1)setup_keys = { line.split("=", 1)[0].strip() for line in setup_block.splitlines() if "=" in line and not line.lstrip().startswith("#")}print("local setup.cfg keys:", sorted(setup_keys))print("legacy keys present in setup.cfg:", sorted(set(setup_keys) & legacy_keys))PY
Repository: Bonobo791/Moderaty
Length of output: 276
🏁 Script executed:
#!/bin/bashset -e
python3 - <<'PY'import refrom pathlib import Pathlocal = Path(".agents/skills-src/mutation-testing/references/tools-by-language.md").read_text()toml_block = re.search(r"```toml\n(.*?)\n```", local, re.S).group(1)toml_keys = { line.split("=", 1)[0].strip() for line in toml_block.splitlines() if "=" in line and not line.lstrip().startswith("#")}current_keys = {"source_paths", "pytest_add_cli_args_test_selection"}legacy_keys = {"paths_to_mutate", "tests_dir"}print("local TOML keys:", sorted(toml_keys))print("legacy keys present:", sorted(toml_keys & legacy_keys))print("current keys present:", sorted(toml_keys & current_keys))print("TOML current-key values use arrays:", all(re.search(r"=\s*\[", line) for line in toml_block.splitlines() if line.split("=", 1)[0].strip() in current_keys))setup_block = re.search(r"```ini\n(.*?)\n```", local, re.S).group(1)setup_keys = { line.split("=", 1)[0].strip() for line in setup_block.splitlines() if "=" in line and not line.lstrip().startswith("#")}print("local setup.cfg keys:", sorted(setup_keys))print("legacy keys present in setup.cfg:", sorted(setup_keys & legacy_keys))PY
Repository: Bonobo791/Moderaty
Length of output: 478
Update both mutmut configuration examples.
With an unpinned install, mutmut uses the current 3.x schema. Replace paths_to_mutate and tests_dir with source_paths and pytest_add_cli_args_test_selection. Use arrays in pyproject.toml and strings in setup.cfg, or pin a legacy mutmut version.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agents/skills-src/mutation-testing/references/tools-by-language.md around
lines 9 - 10, Update both mutmut configuration examples in tools-by-language.md
to use the current 3.x schema: replace paths_to_mutate with source_paths and
tests_dir with pytest_add_cli_args_test_selection, using arrays for
pyproject.toml and strings for setup.cfg. Do not leave examples using the legacy
keys unless the mutmut dependency is explicitly pinned to a compatible legacy
version.
StrykerJS current configuration incrementalFile default reports/stryker-incremental.json .stryker-tmp
💡 Result:
In StrykerJS, the default value for the incrementalFile configuration option is "reports/stryker-incremental.json" [1][2]. This file is used to store results when incremental mode is enabled (via --incremental or "incremental": true) [3][4]. It is important to note that.stryker-tmp is a default directory used by StrykerJS for temporary files (such as the sandbox where tests are executed), but it is distinct from the incremental report file [1][3]. The incremental report file should be treated as an artifact rather than source code [4]. If you wish to change the location of the incremental file, you can do so using the --incrementalFile command-line flag or by setting the incrementalFile property in your configuration file [1][2].
StrykerJS stores incremental results in reports/stryker-incremental.json by default. .stryker-tmp/ contains temporary files. Update the cache references at lines 55 and 202, or set incrementalFile explicitly.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agents/skills-src/mutation-testing/references/tools-by-language.md at line
46, Update the StrykerJS cache references in the mutation-testing documentation
to use the actual default incremental results file,
reports/stryker-incremental.json, instead of temporary .stryker-tmp paths;
alternatively, configure the shown StrykerJS setup with an explicit
incrementalFile value and keep all references consistent.
mutmut apply 7 # writes mutant to disk — file MUST be committed first
# write a test that fails on the mutant, then:
git checkout -- src/file.py # revert immediately — never leave applied mutants on disk
mutmut run 7 # re-test one mutant
for id in $(mutmut result-ids survived); do mutmut run $id; done # re-test all survivors
```
Quirks and controls:
- `# pragma: no mutate` — whitelist a line (version strings, logging, intentional perf trade-offs like `break`→`continue`).
- `mutmut_config.py` with `pre_mutation(context)` — skip mutants programmatically (e.g. skip all `log.*` lines, skip a file) or change the test command per mutant.
- Only needs an exit code from the runner, so any test command works; `hammett` runner is dramatically faster than pytest if adoptable.
- `mutmut html` for a browsable report.
- WARNING for agents: `mutmut apply` physically corrupts the working tree. Always verify `git status` is clean before applying, and revert in the same session.
mutmut documentation current CLI mutants directory mutmut browse mutmut results mutmut html mutmut result-ids cache
💡 Result:
Mutmut is a mutation testing system for Python that uses a file-based workflow for storing results and caches [1][2]. Key CLI components and directories: Mutants Directory: Mutmut stores data about completed mutations and the mutants themselves in a mutants/ directory [3][2]. To perform a full run from scratch, you can delete this directory [2]. Cache: Mutmut maintains a result cache, typically named.mutmut-cache [4][5]. If you need to ensure a full fresh run or if changes to the configuration file (mutmut_config.py) are not taking effect, you should manually delete this file [4][5]. CLI Commands: - mutmut run: Executes the mutation testing suite [1][6]. It automatically attempts to locate code and tests within your project [1][3]. - mutmut browse: Opens an interactive text-based user interface (TUI) where you can view mutants, retest specific functions (press f) or modules (press m), and rerun tests for selected mutants (press r) [1][2]. - mutmut html: Generates an HTML report containing the results of the mutation testing [7]. - mutmut apply: Allows you to write a specific mutant to disk [1][6]. - mutmut status: Provides a list of mutants, optionally filtered by a specific file [7]. - mutmut show : Displays the diff for a specific mutant ID [7]. The general workflow involves running mutmut, browsing the results to identify surviving mutants, and then updating your test suite to kill those mutants [1][2]. Always ensure your files are under version control before applying any mutants to your disk [1][6].
Mutmut uses a directory named mutants/ to store information about the mutants it has generated and the results of its mutation testing process [1][2][3]. While official documentation explicitly mentions the mutants/ directory for storing the data of its operations, it does not provide specific technical documentation regarding a.mutmut-cache file or a result-ids component within its file structure [1][2][3]. To perform a clean, full run of mutmut and clear existing state, you can delete the mutants/ directory [1][2][3]. Changes to your project code or configuration—such as dependency upgrades or changes to mutmut settings—are handled by mutmut's internal tracking mechanisms, which may trigger warnings or automatic cache invalidation for relevant results to ensure they reflect the current state of your code [1][2][3].
Use mutants/ for state and mutmut browse for inspection and retesting. Replace mutmut results and mutmut result-ids, or pin the mutmut version that supports them.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agents/skills-src/mutation-testing/references/tools-by-language.md around
lines 97 - 115, Update the mutmut workflow commands in the tools-by-language
reference to use mutants/ for mutation state and mutmut browse for inspection
and retesting. Replace the deprecated mutmut results and mutmut result-ids
commands, including the survivor re-test flow, with the supported CLI
equivalents; alternatively pin a mutmut version that supports the existing
commands.
description: 'Mutation testing engineering — verify that a test suite actually catches bugs, not just executes lines. Use when auditing or hardening test-suite quality, reviewing test coverage claims ("we have 90% coverage"), hunting surviving mutants, writing tests that kill specific mutants, setting up or configuring mutation tools (Stryker/StrykerJS/Stryker.NET, mutmut, Cosmic Ray, PIT/pitest, Infection, cargo-mutants, go-mutesting/Gremlins, mutant, muter, Mull), wiring mutation testing into CI (incremental PR runs, thresholds, --since/--in-diff), interpreting mutation scores, handling equivalent/timeout/no-coverage mutants, or closing the mutation-feedback loop on AI-generated tests. Triggers on: mutation testing, mutation score, surviving mutants, killed mutants, equivalent mutants, are my tests actually good, test suite quality, weak assertions, mutation coverage, mutant.'
#!/bin/bashset -eu
printf'%s\n''--- AGENTS.md license policy ---'
sed -n '270,320p' AGENTS.md
printf'%s\n''--- target file prefixes ---'forfin \
.agents/skills-src/mutation-testing/SKILL.md \
.agents/skills-src/mutation-testing/references/tools-by-language.md \
.agents/skills-src/mutation-testing/references/agent-mutation-loop.md \
.agents/skills-src/mutation-testing/references/surviving-mutant-triage.md
doecho"### $f"
sed -n '1,25p'"$f"doneprintf'%s\n''--- representative Markdown headers ---'forfin DEPLOY.md DESIGN.md docs/e2e-verification.md README.md;doecho"### $f"
sed -n '1,22p'"$f"doneprintf'%s\n''--- Markdown files under .agents with header references ---'
rg -l -U '^---\n|Copyright \(C\) 2026 Andrew Philip Weilbacher' .agents --glob '*.md'| head -80
Repository: Bonobo791/Moderaty
Length of output: 15658
🏁 Script executed:
#!/bin/bashset -eu
python3 - <<'PY'from pathlib import Pathtargets = [ Path(".agents/skills-src/mutation-testing/SKILL.md"), Path(".agents/skills-src/mutation-testing/references/tools-by-language.md"), Path(".agents/skills-src/mutation-testing/references/agent-mutation-loop.md"), Path(".agents/skills-src/mutation-testing/references/surviving-mutant-triage.md"),]required_markers = [ "# Moderaty — YouTube Comment Auto-Moderation Tool", "# Copyright (C) 2026 Andrew Philip Weilbacher", "GNU Affero General Public License", "Commercial licensing: contact@marketingprowess.simplelogin.com — see COMMERCIAL.md",]for path in targets: text = path.read_text() missing = [marker for marker in required_markers if marker not in text] first = next((line for line in text.splitlines() if line.strip()), "") print(f"{path}: first_nonempty={first!r}; missing_header_markers={missing}")skill = targets[0].read_text()print(f"SKILL.md frontmatter_starts_at_first_line={skill.startswith('---\\n')}")print(f"SKILL.md frontmatter_closes_before_heading={'\\n---\\n' in skill}")PY
Repository: Bonobo791/Moderaty
Length of output: 417
🏁 Script executed:
#!/bin/bashset -eu
python3 - <<'PY'from pathlib import Pathtargets = [ Path(".agents/skills-src/mutation-testing/SKILL.md"), Path(".agents/skills-src/mutation-testing/references/tools-by-language.md"), Path(".agents/skills-src/mutation-testing/references/agent-mutation-loop.md"), Path(".agents/skills-src/mutation-testing/references/surviving-mutant-triage.md"),]required_markers = [ "# Moderaty — YouTube Comment Auto-Moderation Tool", "# Copyright (C) 2026 Andrew Philip Weilbacher", "GNU Affero General Public License", "Commercial licensing: contact@marketingprowess.simplelogin.com — see COMMERCIAL.md",]for path in targets: text = path.read_text() missing = [marker for marker in required_markers if marker not in text] first = next((line for line in text.splitlines() if line.strip()), "") print(f"{path}: first_nonempty={first!r}; missing_header_markers={missing}")skill = targets[0].read_text()starts = skill.startswith("---\n")closes = "\n---\n" in skillprint(f"SKILL.md frontmatter_starts_at_first_line={starts}")print(f"SKILL.md frontmatter_closes_before_heading={closes}")PY
Repository: Bonobo791/Moderaty
Length of output: 1678
Add the approved license header to all four Markdown files.
Use Markdown comment syntax. Place the header before the YAML frontmatter in SKILL.md.
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agents/skills-src/mutation-testing/SKILL.md around lines 1 - 4, Add the
approved license header using Markdown comment syntax to
.agents/skills-src/mutation-testing/SKILL.md at lines 1-4, placing it before the
YAML frontmatter, and add the same header to
.agents/skills-src/mutation-testing/references/tools-by-language.md at lines
1-3, .agents/skills-src/mutation-testing/references/agent-mutation-loop.md at
lines 1-3, and
.agents/skills-src/mutation-testing/references/surviving-mutant-triage.md at
lines 1-3.
- **Killed** — at least one test fails. The suite would catch this bug. Good.
- **Survived** — all tests pass despite the change. A genuine test gap or an equivalent mutant. Act on it.
- **Timed out** — mutant caused an infinite loop. Counts as killed in most tools; investigate if frequent (flaky timing or real performance sensitivity).
- **No coverage** — no test executes the mutated line. Counts as undetected; add a behavior test or exclude the file.
- **Error/unviable** — mutant does not compile or crashes setup. Excluded from the score.
Mutation score = killed / (total − equivalent). Equivalent mutants change syntax but not behavior (`i <= n-1` vs `i < n`); they can never be killed and detecting them automatically is undecidable — expect to flag them by hand. Research puts ~23% of mutants as equivalent, so 100% is a mathematical ceiling, not a target.
In Stryker mutation testing, mutants are categorized by specific statuses that reflect the outcome of running tests against the mutated code [1][2]. Mutant Statuses: - Killed: At least one test failed while the mutant was active [1][2]. - Survived: All tests passed while the mutant was active, indicating a gap in test coverage [1][2]. - No Coverage: The original code is not covered by any tests, so the mutant survived by default [1][2]. - Timeout: The tests took too long to complete, likely due to an infinite loop caused by the mutant [1][2]. These are treated as detected (similar to "Killed") [1][2]. - RuntimeError: An error occurred while running the tests (e.g., a crash or unparsable code) [1][2]. These are typically excluded from the mutation score calculation because they do not reliably indicate test quality [1][2]. - CompileError: The mutant resulted in code that failed to compile [1]. These are excluded from the mutation score [1]. - Pending: The mutant has been generated but not yet tested [1]. - Ignored: The mutant was explicitly excluded by the user [1][3]. These are excluded from the score [1][3]. Mutation Score Formula: The standard mutation score is calculated by dividing the number of detected mutants by the total number of valid (non-ignored/non-errored) mutants [2][4]. Formula: Mutation Score = (Killed + Timeout) / (Killed + Timeout + Survived + NoCoverage) * 100 [2][5][6] Denominator: The denominator consists of all valid mutants, which includes "Total Detected" (#Killed + #Timeout) and "Total Undetected" (#Survived + #NoCoverage) [2][5]. Mutants categorized as RuntimeError, CompileError, or Ignored are excluded from the denominator as they are not considered representative of the test suite's effectiveness [1][2][4]. Note that some stricter reporting variations may include "No Coverage" mutants in the denominator, punishing the score for untested code [7].
In Stryker mutation testing, NoCoverage is a state for a mutant that is not reached by any of your tests, meaning it effectively survives because no test ever executes the code where the mutation was applied [1]. Regarding its impact on the mutation score: 1. Mutation Score: Stryker calculates the standard mutation score as the number of detected mutants divided by the total number of valid mutants [1]. Because NoCoverage mutants are considered undetected (grouped with survived mutants), they negatively impact this score [1]. 2. Mutation Score Based on Covered Code: Stryker also provides a secondary metric called the mutation score based on covered code, which is calculated as the number of detected mutants divided by the number of covered mutants [1]. NoCoverage mutants are excluded from this specific calculation, meaning they do not lower the score in this context [1]. In summary, while NoCoverage mutants reduce your primary mutation score, they are ignored when calculating the mutation score based on covered code, allowing you to measure the effectiveness of your tests specifically on the code they currently cover [1].
Status handling and mutation-score formulas vary by tool. For example, Stryker reports NoCoverage separately from Survived, counts Timeout as detected, and excludes CompileError and RuntimeError from the score. Rewrite these rules as repository policy or define them for each supported tool.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agents/skills-src/mutation-testing/SKILL.md around lines 12 - 18, Update
the mutation-testing status definitions and mutation-score formula in the
documentation around the listed statuses to be explicitly scoped to the
supported mutation-testing tool(s). Either define repository-wide policy or
provide per-tool mappings, including Stryker’s NoCoverage, Timeout,
CompileError, and RuntimeError treatment, and ensure the score calculation
matches each tool’s semantics.
- **No coverage** — no test executes the mutated line. Counts as undetected; add a behavior test or exclude the file.
- **Error/unviable** — mutant does not compile or crashes setup. Excluded from the score.
Mutation score = killed / (total − equivalent). Equivalent mutants change syntax but not behavior (`i <= n-1` vs `i < n`); they can never be killed and detecting them automatically is undecidable — expect to flag them by hand. Research puts ~23% of mutants as equivalent, so 100% is a mathematical ceiling, not a target.
MuTAP mutation feedback loop removing mutation feedback largest drop 50 points study
💡 Result:
MuTAP (Mutation Test case generation using Augmented Prompt) is a technique that uses Large Language Models (LLMs) to generate effective unit tests by leveraging mutation testing [1][2]. In the MuTAP framework, a mutation-guided feedback loop is employed where surviving mutants—those that the existing test suite fails to detect—are used to augment the prompt, instructing the LLM to generate new, targeted test cases to kill those mutants [1][3]. Regarding the specific mention of a "50 points" drop and removing mutation feedback, technical documentation and implementation guidelines for systems similar to or extending MuTAP (such as the qa-test-mutate skill) include iterative feedback loops designed to stop when diminishing returns are reached [4]. These guidelines often define stopping criteria based on the improvement delta (e.g., if the improvement in mutation score is below a certain threshold, such as 5 points) rather than a 50-point drop [4]. There is no widely cited authoritative study or standard MuTAP documentation that describes a "50 points drop" as a recognized phenomenon or a specific trigger for "removing" mutation feedback [1][2]. It is possible that the "50 points" figure originates from a specific implementation or heuristic-based threshold in an automated testing agent, where a significant drop in mutation score might trigger a fallback mechanism or a reconfiguration of the test generation process [4]. In evolutionary or iterative test generation, large drops in mutation score are typically indicators of instability or issues in the test generation process, rather than a planned removal of the feedback mechanism [5][6].
MUTGEN is a mutation-guided, Large Language Model (LLM)-based unit test generation approach that significantly improves the fault-detection capability of generated test suites by incorporating mutation feedback directly into the LLM prompts [1][2][3]. In a study evaluating the approach, MUTGEN was tested against the HumanEval-Java dataset (using 104 subjects after excluding those where both EvoSuite and MutGen achieved a 100% mutation score) and a LeetCode-Java corpus [4][5][3]. The results demonstrated that MUTGEN achieved a mutation score of 89.5% on HumanEval-Java (across 1,144 mutants) and 89.1% on the LeetCode-Java dataset [4][6][3]. This performance significantly outperformed both the state-of-the-art search-based tool EvoSuite and a vanilla LLM-based prompting strategy [4][1][3]. The effectiveness of the approach is attributed to its iterative generation mechanism and the use of mutation feedback, which directs the LLM to identify and kill surviving mutants [1][2]. The approach also includes steps for test summarization, generation, and fixing of failed tests, with ablation studies confirming that each component—including the mutation feedback—is essential for achieving these high mutation scores [7][3]. The replication package and further technical details are available in the associated project repository [8].
Meta ACH equivalent mutant filter 73% generated tests accepted engineers production scale
💡 Result:
At Meta, the Automated Compliance Hardening (ACH) system is an industrial-scale framework that uses mutation-guided, LLM-based test generation to improve software quality and compliance [1][2][3]. A central component of ACH is an LLM-based equivalent mutant detection agent, which filters out mutants that are semantically identical to the original code to prevent developer noise [1][4][2]. This filtering agent achieves a precision and recall of 0.79 and 0.47, which improves significantly to 0.95 and 0.96 when combined with simple preprocessing [2][3]. During deployments across platforms such as Facebook, Instagram, WhatsApp, and wearables, ACH generated 9,095 targeted mutants and produced 571 high-quality hardening tests [1][2]. Meta engineers accepted 73% of these generated tests into production code [1][5][3]. While 36% of these accepted tests were judged as directly relevant to privacy or compliance concerns, engineers frequently accepted the remaining tests for their broader benefits, such as improving code coverage and catching edge cases [4][5][2][3]. This 73% acceptance rate reflects the efficacy of the system in providing an automated safety net at scale [5][6].
The 23% figure commonly cited in mutation testing refers to a notable study by Yao, Harman, and Jia (2014) titled A Study of Equivalent and Stubborn Mutation Operators using Human Analysis of Equivalence [1][2]. In this study, the researchers manually analyzed 1,230 unkilled mutants across 18 programs and found that approximately 23% were equivalent mutants [1][2]. The estimate that 15% to 25% of mutants are equivalent is frequently referenced in literature as a practical range for developers to expect [3]. However, research indicates that the proportion of equivalent mutants varies widely depending on the codebase, programming language, and the specific mutation operators employed [4][5][3]. Key takeaways from the research include: 1. Significant Variability: Estimates of equivalent mutants in the literature span a broad range, from as low as 2% to as high as 50% [4][5]. 2. Context Matters: The specific "equivalent mutant rate" is highly dependent on the subject program and the test suite's quality [4][5][6]. For example, studies have shown that more robust test suites may appear to have a higher percentage of equivalent mutants among their remaining unkilled mutants because they have already successfully caught most non-equivalent ones [7][6]. 3. Impact on Mutation Testing: Because identifying equivalent mutants is an undecidable problem, this "equivalent mutant problem" is a primary obstacle to achieving a 100% mutation score [8][4][9][3]. Consequently, many practitioners consider a 100% score neither necessary nor practically achievable, often targeting lower, realistic coverage goals instead [3].
LLM mutation testing immediate per-test mutation feedback 7x token use
💡 Result:
The association of "7x token use" with LLM-based testing appears to be a conflation of two distinct concepts found in recent technical literature: agentic coordination overhead and mutation-guided test generation workflows [1][2][3]. 1. Agentic Coordination Overhead (7x Token Multiplier): Multiple sources consistently identify that using multi-agent systems—often employed for tasks like automated code review or test generation—frequently results in a token usage multiplier of approximately 7x compared to a single-agent baseline [2][4][5][3]. This "7x" figure is widely cited as the cost of coordination overhead, which includes repeated scaffolding, context replay, progress reporting, and inter-agent communication, rather than actual reasoning or generation [2][3][6]. 2. Mutation-Guided Test Generation: Mutation testing is a well-documented technique for improving LLM-generated tests [7][8][9][10]. In this workflow, an LLM generates tests, a mutation testing tool (e.g., PITest, StrykerJS) identifies "surviving" mutants (bugs the tests failed to catch), and this mutation feedback is fed back into the LLM as prompt context to strengthen the test suite [1][8][9]. While this process is iteratively expensive, there is no standardized "7x token" benchmark specifically for per-test mutation feedback [8][9]. Instead, empirical studies indicate that mutation-guided generation can improve test quality and fault detection significantly, with performance often plateauing after a few iterations (e.g., ~4 loops) [7][8][10]. Summary of the Conflation: There is no evidence of a direct technical link where "per-test mutation feedback" inherently requires 7x token usage. Rather, developers who implement agentic systems to handle mutation-guided test generation may encounter the 7x token multiplier due to the orchestration of multiple agents (e.g., a "generator" agent, a "mutation-runner" agent, and a "refiner" agent) [2][3]. To mitigate this, advanced implementations avoid redundant context passing (e.g., writing mutation reports to shared storage instead of re-injecting them into agent prompts) to reduce token waste [6].
MuTAP (Mutation Test case generation using Augmented Prompt), presented in arXiv 2308.16557 (also published as Dakhel et al., 2024), is a technique designed to improve the effectiveness of Large Language Model (LLM)-generated test cases by leveraging mutation testing [1][2][3]. Regarding the performance metrics mentioned in your query: 1. Fault Detection: MuTAP is reported to detect up to 28% more faulty human-written code snippets compared to baseline approaches [1][4][5]. Of those additional detected faults, 17% were previously undetected by both the state-of-the-art automated test generation tool (Pynguin) and standard zero-shot/few-shot LLM-based prompting techniques [1][4][3]. 2. Mutation Score: On synthetic buggy code, MuTAP achieves a Mutation Score (MS) of 93.57%, which the authors report outperforms other evaluated approaches [1][6][3]. The core mechanism of MuTAP involves using surviving mutants from mutation testing to augment the prompts provided to LLMs. These surviving mutants serve as feedback, highlighting the weaknesses in existing test cases and guiding the LLM to generate more effective "Augmented Unit Tests" (AUTs) capable of killing those mutants [2][7]. While the paper focuses on the overall effectiveness and comparative performance of the MuTAP framework, specific "ablation" studies—typically used to isolate the contribution of individual components—are standard within this research context to validate that the feedback loop (the use of surviving mutants) is the primary driver of the observed improvements in fault detection [1][2].
MuTAP (Mutation Test case generation using Augmented Prompt) is a technique for generating effective unit tests using Large Language Models (LLMs) by leveraging mutation testing as a feedback mechanism [1][2]. The core of MuTAP involves an iterative process where surviving mutants—code variations that pass existing tests—are used to identify gaps in test coverage and prompt the LLM to generate targeted tests [3][4][5]. Removing the mutation-feedback loop from the MuTAP process has a significant negative impact on the performance of the generated test suite. Research indicates that this loop is critical for maintaining high fault detection capabilities [6][7]. Specifically, studies evaluating the removal of this iterative mechanism have observed the following effects: - Substantial Drop in Fault Detection: Removing the mutation-feedback loop has been reported to cause the largest decrease in the fault detection rate (FDR), with some evaluations showing a drop of approximately 50.00% [6][7]. - Reduced Coverage and Mutation Score: In addition to lower fault detection, removing the loop also leads to significant reductions in both code coverage and the mutation score [7]. Providing the LLM with specific information about surviving mutants is essential for enabling the model to "kill" those mutants and improve overall test effectiveness [7]. - Incomplete Test Evolution: Without the feedback loop, the process loses its adversarial nature, where the LLM is guided to address specific behavioral gaps [6][7]. While the LLM might still produce test cases, it lacks the necessary context provided by the surviving mutants, resulting in lower-quality tests that fail to detect corner cases or subtle bugs [7]. In summary, the mutation-feedback loop is the defining feature of MuTAP that allows it to outperform standard zero-shot or few-shot LLM test generation. Disabling it effectively reverts the process to a standard, non-iterative approach, which is significantly less effective at creating robust, bug-revealing test suites [1][7].
In the context of LLM-based software testing, the phrase "up to 7" specifically refers to findings related to mutation-guided test generation, particularly in research from Meta [1]. Recent studies, including research on mutation-guided LLM-based test generation, have demonstrated that incorporating LLM-generated counterexamples—often derived from or validated against mutation feedback—can yield improvements of up to 7% (or in some contexts, relate to specific performance metrics like Pass@5 or mutation score improvements) in the effectiveness of generated test suites [1][2]. Mutation-guided feedback mechanisms generally function by using an LLM to generate unit tests, executing them, and then feeding information about "surviving mutants" (mutations the tests failed to catch) back into the LLM [3][4][5]. The model then uses this feedback to iteratively refine the tests to kill the remaining mutants, thereby increasing the overall mutation score [3][6][2]. While this process is highly effective at improving fault detection beyond traditional coverage metrics, it can involve significant token consumption [7][8]. Research highlights that balancing this iterative feedback is critical for efficiency, as repeated prompts with detailed mutation data can increase costs [7]. Systems like MUTGEN, TESTSMITH, and other agentic frameworks utilize these loops to optimize test quality, often targeting a higher mutation score while managing the trade-offs between generation iterations and computational overhead [3][5][2].
Yao Harman Jia 2014 1230 unkilled mutants 23% equivalent
💡 Result:
The query refers to the 2014 research paper titled A Study of Equivalent and Stubborn Mutation Operators using Human Analysis of Equivalence, authored by Xiangjuan Yao, Mark Harman, and Yue Jia [1][2][3]. In this study, the authors manually analyzed 1,230 unkilled mutants generated from 18 programs to investigate the prevalence and properties of equivalent and stubborn mutants [1][2][3]. The key findings regarding the 1,230 unkilled mutants were [3]: - Equivalent Mutants: Approximately 946 of these mutants (roughly 23% of the total 4,181 mutants generated in the study) were identified as equivalent [3]. - Stubborn Mutants: Approximately 284 of these mutants (roughly 7% of the total mutants generated) were identified as stubborn [3]. The paper defines stubborn mutants as those that remain undetected by a high-quality test suite despite not being equivalent [1][2]. The authors used these findings to highlight the uneven distribution of these mutants across different mutation operator classes and to argue that mutation testing tools should prioritize operators that generate more stubborn mutants and fewer equivalent ones [1][2][3].
"73%" "Mutation-Guided LLM-based Test Generation at Meta" accepted tests
💡 Result:
At Meta, the 73% acceptance rate by engineers for automatically generated tests has been reported in two distinct contexts involving Large Language Model (LLM)-based tools [1][2]. First, in the context of Meta’s Automated Compliance Hardening (ACH) system, which utilizes mutation-guided LLM-based test generation, engineers accepted 73% of the tests generated for privacy-related use cases during Messenger and WhatsApp test-a-thons in 2024 [1][3]. In this application, 36% of the accepted tests were judged to be specifically relevant to privacy, while others were valued for their broader utility in hardening code against regressions [1][3]. Second, an earlier deployment of a different tool, TestGen-LLM, which focuses on automatically improving existing human-written unit tests, also saw a 73% acceptance rate by software engineers during Instagram and Facebook test-a-thons [2]. This tool was designed to ensure code improvement through specific filters to mitigate hallucinations, and its recommendations were successfully landed into production [2]. These figures highlight that across different LLM-driven testing initiatives at Meta—ranging from mutation-guided test generation for compliance to unit test improvement—engineers have consistently accepted approximately 73% of the AI-generated recommendations [1][3][2].
arXiv 2602.08146 mutation feedback 50.00% fault detection ablation title
💡 Result:
The paper corresponding to arXiv 2602.08146 is titled "Test vs Mutant: Adversarial LLM Agents for Robust Unit Test Generation" [1][2][3]. In the paper's ablation study, removing the iterative loop ("w/o Iter") is reported to cause a 50.00% drop in Fault Detection Rate (FDR) [1][4]. This ablation study evaluates the contributions of the framework's core components—specifically the adversarial iteration process and mutation-guided test case enhancement—by measuring their impact on coverage and fault detection rates on the Defects4J dataset [1][4]. The findings indicate that the adversarial iteration process and mutation-guided enhancement are critical, as their removal results in performance degradation on fault detection rates by 25.93%–50.00% [1][4].
“Test vs Mutant: Adversarial LLM Agents for Robust Unit Test Generation”
by Pengyu Chang, Yixiong Fang, Silin Chen, Yuling Shi, Beijun Shen, and Xiaodong Gu.
The paper introduces AdverTest, a two-agent framework where one LLM generates tests and another generates mutants designed to expose weaknesses in those tests. On Defects4J, it reports an 8.56% improvement in fault detection over existing LLM-based methods and 63.30% over EvoSuite, while also improving coverage. [1] (arxiv.org)
"Mutation Test case generation using Augmented Prompt" ablation 50.00
💡 Result:
The term "Mutation Test case generation using Augmented Prompt" refers to a technique, often embodied by the framework MuTAP, designed to improve the effectiveness of unit tests generated by Large Language Models (LLMs) [1][2]. In this approach, mutation testing is leveraged to identify weaknesses in LLM-generated test cases. When a test suite fails to kill a specific mutant (a "surviving mutant"), that information—representing the limitation of the current test suite—is used to augment the prompt provided to the LLM [3][1][4]. This iterative, feedback-driven process forces the LLM to generate new or refined test cases specifically targeted at killing those surviving mutants [3][2]. Regarding "ablation 50.00," this likely refers to specific ablation study results or a parameter setting within research evaluating these frameworks. While the number "50.00" does not appear as a globally standardized constant in this field, ablation studies are a standard component of this research to isolate the impact of different elements—such as mutation feedback, code summarization, or iterative fixing steps—on the overall mutation score [5][6][7]. For example, studies on similar systems like MutGen conduct ablation experiments by disabling components (e.g., MutGen-MF for mutation feedback) to measure their contribution to the improvement of the mutation score [5][8]. In summary, the methodology is a feedback-loop system where: 1. LLMs generate initial test cases [3][2]. 2. Mutants are created for the program under test [3][2]. 3. Surviving mutants are identified [3][1]. 4. Prompts are augmented with this mutant-specific feedback [3][2]. 5. The LLM generates refined tests to address these weaknesses [3][2]. Ablation studies are performed to confirm that each of these stages, particularly the inclusion of mutation feedback, is statistically significant in improving the quality and fault-detection capability of the test suite [5][6][7].
In the context of software testing, particularly within AI-driven code generation and test-hardening workflows, the 50.00% figure refers to a specific drop in fault detection efficacy observed when the mutation-feedback loop is removed [1]. Research on mutation-feedback loops—a process where surviving mutants are explicitly fed back into an automated test generator to guide the creation of new, targeted test cases—indicates that this iterative cycle is critical for maximizing test suite quality [1]. In a study analyzing mutation-feedback test generation (such as the MuTAP research), removing the iterative loop caused the largest measured decline in fault detection rate, specifically identified as 50.00% [1]. This feedback mechanism typically follows a structured process: 1. Running an AI-generated test suite against code mutants [1]. 2. Classifying mutants (killed, survived, etc.) [1]. 3. Identifying surviving mutants as behavioral gaps [1]. 4. Feeding these specific survivors back into the prompt or test generator to produce new tests that address those gaps [1][2]. While mutation scores are sometimes debated regarding their direct correlation to real-world fault detection [3], this specific 50.00% statistic highlights the causal effectiveness of the mutation-feedback loop in improving the ability of automated systems to identify and eliminate gaps in code coverage and assertion logic [1].
Cite empirical claims and correct study attributions.
Cite Yao, Harman, and Jia for ~23%, and state that the estimate came from 18 programs and 4,181 mutants.
Correct the ~50-point claim. The available result is a 50% FDR drop for AdverTest’s w/o Iter ablation on Defects4J, not a MuTAP result isolating mutation feedback.
Scope the ~89.5% MUTGEN result to HumanEval-Java and the 73% ACH result to its reported acceptance study.
Remove or qualify the ~7× token-use claim because no source establishes that multiplier for per-test mutation feedback.
Qualify 15–25% as a context-dependent estimate. The cited study does not establish that rate for survivors in mature triage.
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agents/skills-src/mutation-testing/SKILL.md at line 18, Update
.agents/skills-src/mutation-testing/SKILL.md:18 and :48 to cite Yao, Harman, and
Jia for the ~23% equivalent-mutant estimate, including its basis of 18 programs
and 4,181 mutants, and correct the ~50-point claim to AdverTest’s 50% FDR drop
for the w/o Iter ablation on Defects4J rather than MuTAP. Update
.agents/skills-src/mutation-testing/references/agent-mutation-loop.md:7-9 and
:50 to scope ~89.5% MUTGEN to HumanEval-Java and 73% ACH to its reported
acceptance study, and remove or qualify the unsupported ~7× token-use claim.
Update
.agents/skills-src/mutation-testing/references/surviving-mutant-triage.md:24 to
describe 15–25% as a context-dependent estimate, not an established rate for
survivors in mature triage.
For best results, initiate chat on the files or code changes.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bashset -euo pipefail
git show --stat --oneline 21089e4 29465ca
git diff 21089e4^ 29465ca -- .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py
git diff 21089e4^ 21089e4 -- .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py
ast-grep outline .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py --items all
sed -n '1,240p' .agents/skills-src/mutation-testing/assets/hooks/skill_prehook.py
Length of output: 11000
@Bonobo791, I verified the reported changes in commits 21089e4 and 29465ca.
The hook now logs unreadable SKILL.md failures to stderr and still writes the fallback directive. read_prompt() now accepts bare JSON string literals. The trigger list no longer matches bare mutant(s) or gremlins. The killed mutants? and equivalent mutants? patterns now match singular and plural forms.
The current code preserves the hook contract: it does not interrupt prompt flow and exits with code 0.
I also noted the standalone-installation requirement. Each skill hook must remain self-contained because a skill can be installed with a standalone copy operation.
✏️ Learnings added
Learnt from: Bonobo791
URL: https://github.com/Bonobo791/Moderaty/pull/55
Timestamp: 2026-08-03T15:51:50.248Z
Learning: In this repository, each `.agents/skills-src/<skill>/assets/hooks/skill_prehook.py` hook must remain self-contained. Skills can be installed through standalone copy operations, so shared hook modules are not appropriate unless the installation convention changes.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
Hook failures visible in stdout (Stability) — correct per the repo rule ("fallbacks must log to the server AND show to the user"): stderr-only logging is invisible in stdout-injecting harnesses. Both exception paths now write a WARNING line to stdout as well as stderr; the AGENTS.md skills section documents the two-channel behavior.
are my tests trigger (Functional) — confirmed: fired on "are my tests passing?". Tightened to are my tests (actually)? (good|catching|enough|worth|strong).
mutmut config + CLI outdated (Major ×2) — confirmed against PyPI and the mutmut changelog (3.x renamed paths_to_mutate → source_paths; results/inspection/retest moved to mutmut browse, state in mutants/). Both examples and the workflow rewritten for 3.x, with the 2.x commands noted as pin-and-label.
PIT scmMutationCoverage removed (Major) — confirmed in the pitest changelog ("fully remove deprecated scm maven goal"). Docs now recommend Arcmutate's git integration or a pinned pre-removal pitest-maven.
StrykerJS incremental file (Major) — .stryker-tmp/ is sandbox scratch; the incremental cache is reports/stryker-incremental.json (default incrementalFile). Corrected at both references.
License headers on the 4 markdown files (Major, coding guidelines) — added to all four plus skill_prehook.py. One deviation from your suggestion: SKILL.md's header sits after the YAML front matter, not before it — skill loaders require --- on line 1, so a preceding comment would break parsing. Noted inline.
Mutant-status semantics per tool (Major) — added an explicit note to tools-by-language.md: status buckets and score formulas are tool-specific (StrykerJS NoCoverage vs Survived, Timeout as detected, errors excluded).
Research claims (Maintainability) — dropped the unverifiable precise figures (~50-point ablation, ~89.5% MUTGEN, 73% ACH, ~7× token multiplier); replaced the flat ~23% equivalent-mutant claim with a cited range (Yao, Harman & Jia's 4,181-mutant manual analysis — the 4,181 figure independently confirmed via Kintis et al.); qualified the 15–25% survivor estimate as context-dependent.
MD040 fence tag — text added to the prompt fence in agent-mutation-loop.md.
Declined (1)
MD029 list renumbering in surviving-mutant-triage.md — the 1–9 continuous numbering across the three priority headings is load-bearing: "When to accept a survivor" explicitly references "tier 7–9". Restarting the numbers would silently break that cross-reference.
Your review-body note about requesting-changes permissions is for the repo admin, not actionable by me.
Other bots this round
@Amazon-Q-Developer[bot], @codacy-production[bot], @qodo-code-review[bot] — your round-1/2 findings were addressed in 21089e4 and 29465ca (see the earlier triage comment); no new findings from you this round beyond what was already fixed or declined with reasons there.
Verification: 9-case hook behavior suite re-run (fire/silent/warning channels all correct, including the new stdout WARNING), codacy-analysis clean on every changed file, installed copy at ~/.agents/skills/mutation-testing/ synced.
For best results, initiate chat on the files or code changes.
@Bonobo791, acknowledged.
The reported fixes in PR #58 address the listed findings. The narrower trigger pattern and the two-channel warning output resolve the hook behavior issues.
The documented mutmut, PIT, and StrykerJS updates correct the affected tool guidance. The tool-specific status semantics and qualified research claims improve the accuracy of the reference material.
I also accept the MD029 decision. The continuous tier numbering is required because the document references tiers 7–9.
🐇 ✓
✏️ Learnings added
Learnt from: Bonobo791
URL: https://github.com/Bonobo791/Moderaty/pull/55
Timestamp: 2026-08-03T16:30:46.627Z
Learning: For `.agents/skills-src` skill hooks in this repository, failure fallbacks must write a warning to both stderr and stdout. This supports server logs and stdout-injecting hook harnesses while the hook continues to exit successfully.
Learnt from: Bonobo791
URL: https://github.com/Bonobo791/Moderaty/pull/55
Timestamp: 2026-08-03T16:30:46.627Z
Learning: For `.agents/skills-src/mutation-testing/SKILL.md`, YAML front matter must begin on line 1 for skill-loader compatibility. Put any required license header after the YAML front matter, not before it.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
hook failures visible in stdout (stability) — correct per the repo rule ("fallbacks must log to the server and show to the user"): stderr-only logging is invisible in stdout-injecting harnesses. both exception paths now write a warning line to stdout as well as stderr; the agents.md skills section documents the two-channel behavior.
are my tests trigger (functional) — confirmed: fired on "are my tests passing?". tightened to are my tests (actually)? (good|catching|enough|worth|strong).
mutmut config + cli outdated (major ×2) — confirmed against pypi and the mutmut changelog (3.x renamed paths_to_mutate → source_paths; results/inspection/retest moved to mutmut browse, state in mutants/). both examples and the workflow rewritten for 3.x, with the 2.x commands noted as pin-and-label.
pit scmmutationcoverage removed (major) — confirmed in the pitest changelog ("fully remove deprecated scm maven goal"). docs now recommend arcmutate's git integration or a pinned pre-removal pitest-maven.
strykerjs incremental file (major) — .stryker-tmp/ is sandbox scratch; the incremental cache is reports/stryker-incremental.json (default incrementalfile). corrected at both references.
license headers on the 4 markdown files (major, coding guidelines) — added to all four plus skill_prehook.py. one deviation from your suggestion: skill.md's header sits after the yaml front matter, not before it — skill loaders require --- on line 1, so a preceding comment would break parsing. noted inline.
mutant-status semantics per tool (major) — added an explicit note to tools-by-language.md: status buckets and score formulas are tool-specific (strykerjs nocoverage vs survived, timeout as detected, errors excluded).
research claims (maintainability) — dropped the unverifiable precise figures (~50-point ablation, ~89.5% mutgen, 73% ach, ~7× token multiplier); replaced the flat ~23% equivalent-mutant claim with a cited range (yao, harman & jia's 4,181-mutant manual analysis — the 4,181 figure independently confirmed via kintis et al.); qualified the 15–25% survivor estimate as context-dependent.
md040 fence tag — text added to the prompt fence in agent-mutation-loop.md.
declined (1)
md029 list renumbering in surviving-mutant-triage.md — the 1–9 continuous numbering across the three priority headings is load-bearing: "when to accept a survivor" explicitly references "tier 7–9". restarting the numbers would silently break that cross-reference.
your review-body note about requesting-changes permissions is for the repo admin, not actionable by me.
other bots this round
@Amazon-Q-Developer[bot], @codacy-production[bot], @qodo-code-review[bot] — your round-1/2 findings were addressed in 21089e4 and 29465ca (see the earlier triage comment); no new findings from you this round beyond what was already fixed or declined with reasons there.
verification: 9-case hook behavior suite re-run (fire/silent/warning channels all correct, including the new stdout warning), codacy-analysis clean on every changed file, installed copy at ~/.agents/skills/mutation-testing/ synced.
Answer:
All previously identified issues are addressed in PR #58:
Hook failures are now visible through both stdout and stderr.
The overly broad are my tests trigger was tightened.
Mutmut documentation was updated for 3.x, with legacy 2.x commands clearly labeled.
Deprecated PIT scmmutationcoverage guidance was removed or pinned appropriately.
StrykerJS now references the correct incremental cache file.
License headers were added to all applicable files while preserving SKILL.md front matter requirements.
Tool-specific mutant-status and score semantics are documented.
Unsupported precise research figures were removed or qualified with citations/context.
Markdown fence language was added.
The continuous tier numbering in the triage guide is intentionally retained because later text references tiers 7–9.
The hook behavior suite, Codacy checks, and installed-skill synchronization were also reverified. I have no additional actionable findings.
Hook failures visible in stdout (Stability) — correct per the repo rule ("fallbacks must log to the server AND show to the user"): stderr-only logging is invisible in stdout-injecting harnesses. Both exception paths now write a WARNING line to stdout as well as stderr; the AGENTS.md skills section documents the two-channel behavior.
are my tests trigger (Functional) — confirmed: fired on "are my tests passing?". Tightened to are my tests (actually)? (good|catching|enough|worth|strong).
mutmut config + CLI outdated (Major ×2) — confirmed against PyPI and the mutmut changelog (3.x renamed paths_to_mutate → source_paths; results/inspection/retest moved to mutmut browse, state in mutants/). Both examples and the workflow rewritten for 3.x, with the 2.x commands noted as pin-and-label.
PIT scmMutationCoverage removed (Major) — confirmed in the pitest changelog ("fully remove deprecated scm maven goal"). Docs now recommend Arcmutate's git integration or a pinned pre-removal pitest-maven.
StrykerJS incremental file (Major) — .stryker-tmp/ is sandbox scratch; the incremental cache is reports/stryker-incremental.json (default incrementalFile). Corrected at both references.
License headers on the 4 markdown files (Major, coding guidelines) — added to all four plus skill_prehook.py. One deviation from your suggestion: SKILL.md's header sits after the YAML front matter, not before it — skill loaders require --- on line 1, so a preceding comment would break parsing. Noted inline.
Mutant-status semantics per tool (Major) — added an explicit note to tools-by-language.md: status buckets and score formulas are tool-specific (StrykerJS NoCoverage vs Survived, Timeout as detected, errors excluded).
Research claims (Maintainability) — dropped the unverifiable precise figures (~50-point ablation, ~89.5% MUTGEN, 73% ACH, ~7× token multiplier); replaced the flat ~23% equivalent-mutant claim with a cited range (Yao, Harman & Jia's 4,181-mutant manual analysis — the 4,181 figure independently confirmed via Kintis et al.); qualified the 15–25% survivor estimate as context-dependent.
MD040 fence tag — text added to the prompt fence in agent-mutation-loop.md.
Declined (1)
MD029 list renumbering in surviving-mutant-triage.md — the 1–9 continuous numbering across the three priority headings is load-bearing: "When to accept a survivor" explicitly references "tier 7–9". Restarting the numbers would silently break that cross-reference.
Your review-body note about requesting-changes permissions is for the repo admin, not actionable by me.
Other bots this round
@Amazon-Q-Developer[bot], @codacy-production[bot], @qodo-code-review[bot] — your round-1/2 findings were addressed in 21089e4 and 29465ca (see the earlier triage comment); no new findings from you this round beyond what was already fixed or declined with reasons there.
Verification: 9-case hook behavior suite re-run (fire/silent/warning channels all correct, including the new stdout WARNING), codacy-analysis clean on every changed file, installed copy at ~/.agents/skills/mutation-testing/ synced.
Thanks for the triage update. Based on the current Qodo findings index, findings 1 and 3 are implemented, while finding 2 (missing AGPL headers) and finding 4 (spaces in skill_prehook.py) are still marked active. If the fixes in commits 21089e4 and 29465ca address those items, the review state has not yet been updated in this thread.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
codeant-ai
Bot
added
the
size:XL
![amazon-q-developer[bot]](https://avatars.githubusercontent.com/in/1220912?s=60&v=4){kind=small}
![codacy-production[bot]](https://avatars.githubusercontent.com/in/56611?s=60&v=4){kind=small}
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
User description
What
Adds the mutation-testing agent skill and wires it the same way as
sqlite-engineering/drizzle-engineering(PR #51):.agents/skills-src/mutation-testing/SKILL.md— mutation testing engineering: score interpretation, core workflow, mental mutation testing for small diffs, agent mutation-feedback loop, CI strategy, anti-patterns.references/—tools-by-language.md,surviving-mutant-triage.md,agent-mutation-loop.md.assets/hooks/skill_prehook.py— UserPromptSubmit pre-hook: keyword-matches mutation-testing prompts, injects SKILL.md (or a short directive withMUTATION_TESTING_HOOK_FULL=0), always exits 0, logs failures loudly to stderr. Triggers deliberately exclude generic test terms so ordinary testing prompts stay silent.AGENTS.md— documents the mutation-testing-engineer role (owns test-suite quality per the "every test must fail if the real logic is wrong" rule) and the skills-src → install + pre-hook convention.Machine-local setup (not in this diff, mirrors the other two skills): skill copied to
~/.agents/skills/mutation-testing/and hook registered in~/.kimi-code/config.toml.Verification
check my mutation score/triage the surviving mutants(JSON and raw-text stdin); silent onfix the navbarand genericadd a unit test for the login form; empty stdin exits 0;HOOK_FULL=0emits the short directive.codacy-analysis analyzeon the changed files: 0 issues (Semgrep, Trivy).npm run check: 0 errors, 0 warnings.npm run build: green.npm run test: 45 files / 363 tests pass.Notes
docs-agent-rolesalso inserts an Agent Split section intoAGENTS.md; whichever PR merges second may need a small rebase — sections are adjacent but independent.CodeAnt-AI Description
Add mutation-testing guidance and automatically apply it to relevant agent prompts
What Changed
AGENTS.mdImpact
✅ Mutation-testing guidance available during relevant agent tasks✅ Fewer unnecessary skill injections for ordinary testing requests✅ Prompt processing continues despite hook or skill-file errors💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.