Skip to content

Conversation

@halconel
Copy link
Contributor

@halconel halconel commented Jan 28, 2026

Summary

Fixes remaining issues with reverse join and changelist processing after PR #368.

Changes

1. Fix ambiguous column errors in JOIN queries (commit 554f7bd)

Problem: JOIN queries generated qualified column names in SELECT (e.g., primary_meta_tbl.id) without aliases, causing "column reference 'id' is ambiguous" errors in WHERE clause.

Solution: Added .label(k) to all qualified columns in SELECT:

# Before: SELECT primary_meta_tbl.id FROM ...
select_cols.append(primary_meta_tbl.c[k])

# After: SELECT primary_meta_tbl.id AS id FROM ...
select_cols.append(primary_meta_tbl.c[k].label(k))

Modified functions:

  • _build_reverse_meta_cte
  • _build_reverse_data_cte
  • _meta_data_sql_helper

2. Use offset=0 for changelist with reverse join support (commits 0e5b668, cb495f2)

Problem: Initial approach (commit 5b1a19a) disabled offset optimization entirely by switching to v1, which broke reverse join functionality.

Solution: Use v2 (with reverse join support) but with offset=0 for all tables:

  • Added changelist_mode flag in get_change_list_process_ids
  • build_changed_idx_sql_v2 checks this flag and uses offset=0 instead of saved offsets
  • This allows reverse join to work while processing all records from filters_idx
  • Fixed ambiguous column error in reverse join by passing ref_meta_tbl to _apply_sql_filters
  • Filter only columns that exist in table in sql_apply_filters_idx_to_subquery

Benefits:

  • ✅ Reverse join with join_keys works in changelist
  • ✅ No offset filtering (processes all records specified in changelist)
  • ✅ Maintains v2 architecture and performance

3. Add production test case (commit 43eaf2c)

Added comprehensive test reproducing the exact production scenario that caused the bug:

  • Primary table: advertising_campaign_to_moderate
  • Reverse join with composite keys: id + ad_campaign_id
  • Reverse join with single key: id only
  • Transform keys spanning multiple tables

Testing

✅ All changelist tests pass
✅ Offset optimization works correctly in run_full
✅ Reverse join with filters_idx works without errors
✅ Production case with composite join_keys passes

…mposite join_keys

Reproduces the exact scenario that caused ambiguous column errors:
- Primary table: advertising_campaign_to_moderate
- Reverse join with composite key: id + ad_campaign_id
- Reverse join with single key: id only
- Transform keys spanning multiple tables
… prevent skipping records in LEFT JOIN scenarios"

This reverts commit 5b1a19a.
@halconel halconel merged commit 40b6cb0 into epoch8:feat/offsets Jan 28, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant