feat(ingest/sql): column logic + join extraction

[hsheth2]

Adds a couple new features to our SQL parser

• Best-effort column logic extraction. In case a column goes through multiple layers of transformation, we only take the final one.
  • For each column, we also annotate whether it was a direct copy or if it underwent any transformation - see the is_direct_copy bool.
• Join extraction. For each join clause, we try to figure out (1) the on clause used and (2) the full list of tables/columns involved in the join. The latter is more reliable when there's CTEs.

The column logic extraction is also now integrated into the SqlParsingAggregator, so our main warehouse integrations should benefit from it. The join logic extraction is not yet used anywhere.

All tests are updated, and I also added a couple new ones for tricky edge cases (e.g. self-joins).

[codecov]

Codecov Report

Attention: Patch coverage is 93.75000% with 6 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...gestion/src/datahub/sql_parsing/sqlglot_lineage.py	93.33%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

[sgomezvillamor]

awesome!

beyond self-join, what's the coverage for other join variants such as inner/outer/cross joins? they could be explicitely tested

should user-facing doc be updated with the new features?

[hsheth2]

@sgomezvillamor this should work for all join types, including ones that use subqueries or CTEs

There's no user-facing portion of this yet, so I don't want to add any user-facing docs around this until it shows up in the product somewhere

[treff7es]

Is there a reason we format as a string and not convert it into a JSON object to make it parsable by machine?

[hsheth2]

I believe this is shown in the UI as-is

[treff7es]

Is this correct with the square brackets?

[hsheth2]

yes - mssql uses square brackets for quotes