Skip to content

Fix profiler overwrite table semantics - #2581

Open
dgomez04 wants to merge 2 commits into
mainfrom
feat/db-overwrites
Open

Fix profiler overwrite table semantics#2581
dgomez04 wants to merge 2 commits into
mainfrom
feat/db-overwrites

Conversation

@dgomez04

Copy link
Copy Markdown
Contributor

Issue

Profiler overwrite mode replaced rows with TRUNCATE + INSERT while retaining the existing DuckDB table definition. For tables created through inferred schemas, a first run could persist a narrow type such as DECIMAL(10,8). A same-day rerun with larger values then failed during insertion because the stale schema was reused.

Solution

  • Replace inferred-schema tables with DROP + CREATE TABLE AS SELECT on overwrite so both the definition and data reflect the current run.
  • Recreate DDL-owned tables before loading them so explicitly declared types remain authoritative.
  • Align the Python profiler DuckDB helper with the same replace-table semantics.
  • Update the existing overwrite tests to cover schema replacement and narrow-to-wide Decimal values.

Tests: .venv/bin/python -m pytest tests/unit/assessment/test_duckdb_helpers.py -q --tb=short (13 passed)

Follow-up

We need to consolidate profiler writes behind a single API. _save_overwrite, save_to_duckdb, and _save_to_db currently provide overlapping write paths and can drift in behavior. A separate issue will be created to track that consolidation.

@dgomez04
dgomez04 requested a review from a team as a code owner July 21, 2026 16:05
@dgomez04 dgomez04 added the feat/profiler Issues related to profilers label Jul 21, 2026
@dgomez04 dgomez04 self-assigned this Jul 21, 2026
@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 8.69565% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.37%. Comparing base (25225cb) to head (6198245).

Files with missing lines Patch % Lines
...databricks/labs/lakebridge/assessments/pipeline.py 0.00% 21 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2581      +/-   ##
==========================================
- Coverage   70.40%   70.37%   -0.04%     
==========================================
  Files         107      107              
  Lines        9661     9661              
  Branches     1070     1068       -2     
==========================================
- Hits         6802     6799       -3     
- Misses       2653     2657       +4     
+ Partials      206      205       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown

✅ 174/174 passed, 9 flaky, 2 skipped, 1h9m37s total

Flaky tests:

  • 🤪 test_installs_and_runs_local_bladebridge (13.431s)
  • 🤪 test_installs_and_runs_pypi_bladebridge (22.58s)
  • 🤪 test_recon_for_report_type_is_data (37.18s)
  • 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (21.822s)
  • 🤪 test_transpiles_informatica_to_sparksql (24.142s)
  • 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (4.084s)
  • 🤪 test_transpile_teradata_sql_non_interactive[True] (5.756s)
  • 🤪 test_transpile_teradata_sql (27.104s)
  • 🤪 test_transpile_teradata_sql_non_interactive[False] (22.911s)

Running from acceptance #5138

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat/profiler Issues related to profilers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant