test: add anti-pattern checks for structural vs behavioral tests #507#510
test: add anti-pattern checks for structural vs behavioral tests #507#510Serhan-Asad wants to merge 1 commit intopromptdriven:mainfrom
Conversation
…tdriven#507 Add tests verifying that the Step 7 prompt contains guidance against generating structural tests (inspect.signature checks) instead of behavioral tests that verify actual functionality. Closes promptdriven#507 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive test coverage for verifying that the Step 7 prompt properly guides against generating structural tests (like inspect.signature checks) and instead promotes behavioral testing approaches.
Changes:
- Added new test class
TestStep7PromptStructuralTestAntiPatternwith 6 test methods - Tests verify anti-pattern warnings, behavioral testing guidance, and mock-based verification examples in the Step 7 prompt
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| has_warning = any([ | ||
| "inspect.signature" in content_lower, | ||
| "sig.parameters" in content_lower, | ||
| # The prompt warns against testing signatures via the anti-pattern section | ||
| "anti-pattern" in content_lower and "signature" in content_lower, | ||
| # General guidance against non-behavioral testing | ||
| "testing behavior, not implementation" in content_lower, | ||
| "behavior, not implementation details" in content_lower, | ||
| ]) |
There was a problem hiding this comment.
The variable name has_warning is misleading since the conditions also check for general behavioral testing guidance (lines 227-228) which aren't warnings. Consider renaming to has_structural_test_guidance or similar to better reflect that it checks for either warnings OR guidance.
| has_warning = any([ | ||
| "parameter existence" in content_lower, | ||
| "parameter accepts" in content_lower, | ||
| "function accepts" in content_lower and "parameter" in content_lower, | ||
| "signature check" in content_lower, | ||
| "structural" in content_lower and "test" in content_lower, | ||
| # The prompt has an anti-pattern section warning against testing callee's signature | ||
| "anti-pattern" in content_lower and "callee" in content_lower, | ||
| # Testing the signature vs testing behavior | ||
| "callee's signature" in content_lower or "callee rejects" in content_lower, | ||
| ]) |
There was a problem hiding this comment.
The variable name has_warning is reused with a different meaning than in the previous test method. Consider using a more specific name like has_parameter_check_warning to improve code clarity and avoid confusion.
| has_anti_pattern_example = any([ | ||
| "inspect.signature" in content_lower and ("wrong" in content_lower or "anti" in content_lower or "do not" in content_lower or "don't" in content_lower), | ||
| "sig.parameters" in content_lower and ("wrong" in content_lower or "anti" in content_lower or "do not" in content_lower or "don't" in content_lower), |
There was a problem hiding this comment.
These two lines have duplicated logic with repeated condition checks. Consider extracting the repeated tuple ("wrong" in content_lower or "anti" in content_lower or "do not" in content_lower or "don't" in content_lower) into a variable to reduce duplication and improve readability.
| has_anti_pattern_example = any([ | |
| "inspect.signature" in content_lower and ("wrong" in content_lower or "anti" in content_lower or "do not" in content_lower or "don't" in content_lower), | |
| "sig.parameters" in content_lower and ("wrong" in content_lower or "anti" in content_lower or "do not" in content_lower or "don't" in content_lower), | |
| has_warning_keywords = ( | |
| "wrong" in content_lower | |
| or "anti" in content_lower | |
| or "do not" in content_lower | |
| or "don't" in content_lower | |
| ) | |
| has_anti_pattern_example = any([ | |
| "inspect.signature" in content_lower and has_warning_keywords, | |
| "sig.parameters" in content_lower and has_warning_keywords, |
1 participant
Summary
Adds tests verifying that the Step 7 prompt contains proper guidance against generating structural tests (e.g.,
inspect.signaturechecks) instead of behavioral tests.Test Files
tests/test_agentic_bug_step7_prompt.pyTest Coverage
What These Tests Verify
inspect.signature/sig.parameterspatternsTest Execution
Next Steps
Closes #507
Generated by PDD agentic test workflow