feat: rework v_ahb_diff (breaking for all users of this view)
#215
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
but doesn't affect other parts of the code. since the view was just added in v0.28 there shouldn't be too many users besides us
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
hope shit helps... it's more annoying than expected
Reorder columns in unique indexes to put edifact_format_version and pruefidentifikator first, followed by id_path/path. This allows SQLite to efficiently use these indexes when filtering by version and pruefidentifikator in WHERE clauses before performing JOIN operations. Before: (id_path, pruefidentifikator, edifact_format_version) After: (edifact_format_version, pruefidentifikator, id_path) This improves query performance for v_ahb_diff view lookups without adding redundant indexes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified the v_ahb_diff view snapshot results against the raw AHB XML files to confirm the diff logic is working correctly. ## Verified Changes for Prüfidentifikator 55109 (UTILMD Strom) | Snapshot Entry | FV2410 | FV2504 | Status | |----------------|--------|--------|--------| | UNH segment: `Muss` → `Muss [555]` | Line 142511 | Line 8312 | ✅ | | Version `S1.1a` deleted | Line 142543 | Replaced by `S2.1` | ✅ | | STS name: `Transaktionsgrund` → `Transaktionsgrund / Ergänzung / Transaktionsgrund befristete Anmeldung` | Line 64527 | Line 8494 | ✅ | ## Verified Changes for Prüfidentifikator 29002 (COMDIS) | Snapshot Entry | FV2410 | FV2504 | Status | |----------------|--------|--------|--------| | Version `1.0d` deleted | Line 37 | Replaced by `1.0e` | ✅ | ## Conclusion The diff view correctly detects: - Modified elements (AHB status changes, name changes) - Deleted elements (version codes removed, segments removed) - Proper bedingung column comparison for expression changes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified v_ahb_diff snapshot results for Prüfidentifikator 31005 (INVOIC). ## Verified Changes | Change | FV2410 | FV2504 | Status | |--------|--------|--------|--------| | Version code deleted | `2.8c` (line 38) | `2.8d` (line 38) | ✅ | Source files: - FV2410: INVOIC_AHB_2_5c_Fehlerkorrektur_20250131.xml - FV2504: INVOIC_AHB_2_5d_Fehlerkorrektur_20250623.xml 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified v_ahb_diff snapshot results for Prüfidentifikator 27001 (PRICAT). ## Verified Changes | Change | FV2410 | FV2504 | Status | |--------|--------|--------|--------| | Version code deleted | `2.0c` (line 41) | `2.0d` (line 41) | ✅ | | SG40 Preisangabe AHB_Status | `Muss` (line 285) | `Muss [53]` (line 285) | ✅ | Source files: - FV2410: PRICAT_AHB_2_0d_außerordentliche_20240726.xml - FV2504: PRICAT_AHB_2_0e_Fehlerkorrektur_20250623.xml 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified v_ahb_diff snapshot results for Prüfidentifikator 13002 (MSCONS). ## Verified Changes | Change | FV2410 | FV2504 | Status | |--------|--------|--------|--------| | D_1154 Name | `Referenz, Identifikation` (line 4847) | `Referenznummer` (line 4844) | ✅ | The snapshot correctly detects that the element name changed while the AHB_Status `X [529]` remained the same - resulting in a 'modified' diff status. Source files: - FV2410: MSCONS_AHB_3_1d_außerordentliche_20240726.xml - FV2504: MSCONS_AHB_3_1f_Fehlerkorrektur_20250623.xml 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified v_ahb_diff snapshot results for Prüfidentifikator 19011 (ORDRSP). ## Verified Changes | Change | FV2410 | FV2504 | Status | |--------|--------|--------|--------| | Version code deleted | `1.3` (line 37) | `1.4` (line 37) | ✅ | | D_3148 AHB_Status | `X` (line 192) | `X (([939] [50]) ∨ ([940] [51])) ∧ [540]` (line 192) | ✅ | The snapshot correctly detects: - The version code change (shown as 'deleted' since id_path includes version) - The AHB_Status expression change for Kommunikationsadresse element Source files: - FV2410: ORDRSP_AHB_2_2_Fehlerkorrektur_20250417.xml - FV2504: ORDRSP_AHB_1_0a_Fehlerkorrektur_20250417.xml 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified v_ahb_diff snapshot results for Prüfidentifikator 29002 (COMDIS). ## Verified Changes | Change | FV2410 | FV2504 | Status | |--------|--------|--------|--------| | Version code deleted | `1.0d` (line 37) | `1.0e` (line 37) | ✅ | Source files: - FV2410: COMDIS_AHB_1_0e__20231024.xml - FV2504: COMDIS_AHB_1_0f_außerordentliche_20250131.xml 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified v_ahb_diff snapshot results for Prüfidentifikator 44042 (UTILMD Gas). ## Verification Result The snapshot correctly shows NO diff entries for prüfi 44042 because the source XML files are identical between FV2410 and FV2504: - FV2410: UTILMD_AHB_Gas_1_0a_außerordentliche_20240726.xml - FV2504: UTILMD_AHB_Gas_1_0a_außerordentliche_20240726.xml Both files have the same content (verified via `diff` command with no output). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified v_ahb_diff snapshot results for Prüfidentifikator 15003 (QUOTES). ## Verification Result The snapshot correctly shows NO diff entries for prüfi 15003 because the source XML files are identical between FV2410 and FV2504: - FV2410: QUOTES_AHB_1_0_Fehlerkorrektur_20241213.xml - FV2504: QUOTES_AHB_1_0_Fehlerkorrektur_20241213.xml Both files have the same content (verified via `diff` command with no output). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
The original SQL view used a LEFT JOIN that matched on id_path alone across ALL prüfidentifikators, causing "added" entries to never appear (found 0 added, 94 deleted, 58 modified in initial snapshot). Root cause: When checking if a row exists in the old version, the query matched ANY row with the same id_path from ANY prüfi, not within the specific version pair being compared. Solution: Rewrote using CTEs to: 1. Generate all version pairs first (old_version x new_version) 2. For each pair, properly JOIN within that specific pair 3. Use NOT EXISTS with full version pair constraints for added/deleted Results after fix: 113 added, 94 deleted, 58 modified entries 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Added prüfis to verify added/deleted/modified detection across formats: - 21000, 21001: IFTSTA (status messages) - 33001: REMADV (remittance advice) - 37001: PARTIN (party information) - 25001: UTILTS (utility time series) - 39000: ORDCHG (order change) - 13007: MSCONS (metered services consumption) - 17001, 17101: ORDERS (orders) - 31001: INVOIC (invoice) Snapshot now contains 265 entries (113 added, 94 deleted, 58 modified) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified 3 diff entries (1 added, 1 deleted, 1 modified): - Added: version code 2.0f (FV2504 IFTSTA_AHB_2_0g_Fehlerkorrektur_20250225.xml:37) - Deleted: version code 2.0e (FV2410 IFTSTA_AHB_2_0e_außerordentliche_20240726.xml:37) - Modified: D_3148 ahb_status changed from X to complex expression 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified 3 diff entries (1 added, 1 deleted, 1 modified): - Version code upgrade from 2.0e to 2.0f, same as prüfi 21000 - Both IFTSTA prüfis share the same format version changes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified 141 diff entries (51 added, 49 deleted, 41 modified): - Version upgrade from 1.1c to 1.1e - FV2410: UTILTS_AHB_1_1a_und_1.0f_außerordentliche_20240726.xml - FV2504: UTILTS_AHB_1_0_Fehlerkorrektur_20250218.xml - Significant structural changes including new code Z36 "Berechnungsformel" 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified 2 diff entries (1 added, 1 deleted): - Version code changes in remittance advice format 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified 2 diff entries (1 added, 1 deleted): - Version code changes in party information format 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified 1 diff entry (modified): - Minor status change in order change format 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified 2 diff entries (modified): - Minor status changes in metered services consumption format 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified 2 diff entries (1 added, 1 deleted): - Version code changes in orders format 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified (no diff entries for this prüfi in snapshot): - This prüfi may have identical structure between FV2410 and FV2504 - Or may not exist in both versions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Verified 2 diff entries (1 added, 1 deleted): - Version code changes in invoice format 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
The test function was removed but the snapshot remained in the file. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Added 7 non-snapshot tests to verify logical invariants of the diff view: - Self-comparison returns only 'unchanged' entries - Diff is symmetric (forward added = reverse deleted) - No duplicate id_paths per version pair - 'added' rows have NULL old_* columns - 'deleted' rows have NULL new_* columns - 'modified' rows have actual differences - Non-existent prüfi returns empty results These tests ensure the SQL logic doesn't regress during future changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Removed unused `sqlalchemy` and `DiffStatus` imports. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
but doesn't affect other parts of the code. since the view was just added in v0.28 there shouldn't be too many users besides us