-
Notifications
You must be signed in to change notification settings - Fork 53
Add failing tests for #430: auto-fix fingerprint skip bug #432
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+102
−0
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
b78000f
Add failing tests for issue #430: auto-fix fingerprint skip bug
Serhan-Asad d98edf3
fix: Auto-fix skips fingerprint save causing incomplete metadata (syn…
Serhan-Asad 4f87c03
Fix test fixture to allow individual tests to override mock behavior
Serhan-Asad 6acb8a5
Fix E2E test to check correct Fingerprint fields
Serhan-Asad 5b3958b
Clean up test files: remove emojis, comments, and unrelated files
Serhan-Asad 0d26666
Remove emojis and excessive comments from E2E test
Serhan-Asad 575c6bf
Improve test readability: use named variables instead of magic indices
Serhan-Asad 55fdd0e
Remove misleading E2E test that doesn't exercise actual bug
Serhan-Asad File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4596,3 +4596,102 @@ def capture_subprocess_run(cmd, **kwargs): | |
| f"This verifies the fix: sync_orchestration.py:1266 uses " | ||
| f"pdd_files['code'].resolve().parent instead of hardcoded Path.cwd() / 'src'" | ||
| ) | ||
|
|
||
|
|
||
| # --- Issue #430: Auto-fix Fingerprint Save Bug Tests --- | ||
|
|
||
| def test_auto_fix_success_saves_complete_metadata(orchestration_fixture): | ||
| """ | ||
| Tests that when auto-fix successfully resolves an import error during the crash | ||
| operation, all metadata is properly saved (fingerprint, operations_completed, events). | ||
|
|
||
| This test reproduces issue #430: the auto-fix success path at line 1412 uses | ||
| `continue` which skips the fingerprint save at line 1716, causing incomplete | ||
| metadata tracking. | ||
|
|
||
| Expected behavior (after fix): | ||
| - operations_completed includes 'crash' | ||
| - _save_fingerprint_atomic is called with operation='crash', model='auto-fix', cost=0.0 | ||
| - run_report.json is saved with exit_code=0 | ||
|
|
||
| Actual behavior (before fix): | ||
| - operations_completed is missing 'crash' (BUG) | ||
| - _save_fingerprint_atomic is NOT called for crash operation (BUG) | ||
| - run_report.json is saved correctly (this works) | ||
| """ | ||
| # Create code file for crash operation to detect (fixture chdirs to tmp_path) | ||
| from pathlib import Path | ||
| code_file = Path('src') / 'calculator.py' | ||
| code_file.write_text("# Mock code file\ndef calculator():\n pass\n") | ||
|
|
||
| mock_determine = orchestration_fixture['sync_determine_operation'] | ||
| mock_save_fp = orchestration_fixture['_save_fingerprint_atomic'] | ||
|
|
||
| # Set up workflow: generate → example → crash (with auto-fix success) | ||
| mock_determine.side_effect = [ | ||
| SyncDecision(operation='generate', reason='New unit'), | ||
| SyncDecision(operation='example', reason='Code exists, example missing'), | ||
| SyncDecision(operation='crash', reason='Example crashes'), | ||
| SyncDecision(operation='all_synced', reason='All operations complete'), | ||
| ] | ||
|
|
||
| # Mock the crash operation to simulate auto-fix success path | ||
| # We need to patch the internal functions that detect and fix import errors | ||
| with patch('pdd.sync_orchestration._run_example_with_error_detection') as mock_run_example, \ | ||
| patch('pdd.sync_orchestration._try_auto_fix_import_error') as mock_auto_fix, \ | ||
| patch('pdd.sync_orchestration._save_run_report_atomic') as mock_save_report: | ||
|
|
||
| # First call: example crashes with ModuleNotFoundError (auto-fixable) | ||
| # Second call: retry after auto-fix succeeds (returncode=0) | ||
| mock_run_example.side_effect = [ | ||
| (1, "", "ModuleNotFoundError: No module named 'calculator'"), # Initial crash | ||
| (0, "Example runs successfully", ""), # After auto-fix | ||
| ] | ||
|
|
||
| # Auto-fix successfully fixes the import | ||
| mock_auto_fix.return_value = (True, "Added missing import") | ||
|
|
||
| # Run sync orchestration | ||
| result = sync_orchestration(basename="calculator", language="python") | ||
|
|
||
| # CRITICAL ASSERTIONS: These verify the bug is fixed | ||
|
|
||
| # 1. Verify operations_completed includes 'crash' | ||
| # BUG: Before fix, this assertion FAILS because continue at line 1412 skips | ||
| # the operations_completed.append('crash') at line 1715 | ||
| assert 'crash' in result['operations_completed'], ( | ||
| "Auto-fix success should track 'crash' in operations_completed. " | ||
| "Bug: continue at sync_orchestration.py:1412 skips line 1715" | ||
| ) | ||
|
|
||
| # 2. Verify _save_fingerprint_atomic was called for crash operation | ||
| # BUG: Before fix, this assertion FAILS because continue at line 1412 skips | ||
| # the _save_fingerprint_atomic call at line 1716 | ||
| fingerprint_calls = [ | ||
| call for call in mock_save_fp.call_args_list | ||
| if len(call[0]) >= 3 and call[0][2] == 'crash' # arg[2] is operation | ||
|
||
| ] | ||
| assert len(fingerprint_calls) > 0, ( | ||
| "Auto-fix success should save fingerprint for 'crash' operation. " | ||
| "Bug: continue at sync_orchestration.py:1412 skips _save_fingerprint_atomic at line 1716" | ||
| ) | ||
|
|
||
| # 3. Verify the fingerprint was saved with correct metadata | ||
| crash_fingerprint_call = fingerprint_calls[0] | ||
| call_args = crash_fingerprint_call[0] | ||
| basename, language, operation, pdd_files, cost, model = call_args[:6] | ||
|
|
||
| assert basename == "calculator" | ||
| assert language == "python" | ||
| assert operation == 'crash' | ||
| assert cost == 0.0, "Auto-fix should have cost=0.0" | ||
| assert model == 'auto-fix', "Auto-fix should use model='auto-fix'" | ||
|
|
||
| # 4. Verify run_report was saved (this already works, but verify it) | ||
| assert mock_save_report.called, "run_report should be saved after auto-fix" | ||
|
|
||
| # 5. Verify workflow succeeded and continued normally | ||
| assert result['success'] is True, "Workflow should succeed after auto-fix" | ||
| assert result['operations_completed'] == ['generate', 'example', 'crash'], ( | ||
| "Expected all three operations to complete" | ||
| ) | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The mock for _save_run_report_atomic is created but never verified in the assertions. Consider adding an assertion to verify it was called with the expected arguments to ensure the run_report saving behavior is properly tested.