Skip to content

feat: add data source codec extension seam - #2388

Open
wirybeaver wants to merge 1 commit into
apache:mainfrom
wirybeaver:dfb-open-table
Open

feat: add data source codec extension seam#2388
wirybeaver wants to merge 1 commit into
apache:mainfrom
wirybeaver:dfb-open-table

Conversation

@wirybeaver

Copy link
Copy Markdown

Which issue does this PR close?

Part of #1241.

What changes are included?

Adds a small, format-neutral serialization seam for external data-source integrations:

  • BallistaTableProviderCodec for logical table-provider transport
  • BallistaPhysicalPlanCodec for format-specific execution plans
  • a named envelope so decoding dispatches by stable codec name rather than registration order
  • explicit claim/decline encoding through Option<Vec<u8>>

Ballista's own logical and physical nodes keep their existing path and priority. Core gains no Hudi, Iceberg, Delta, or Lance dependency, and behavior is unchanged when no codec is registered.

A follow-up Hudi integration uses this seam for a distributed copy-on-write read.

Tests

  • cargo test -p ballista-core --lib serde::test
  • cargo clippy -p ballista-core --lib --tests -- -D warnings

The focused tests cover logical provider transport, physical plan transport, stable name dispatch, and the existing Ballista-plan regression.

Allow external data-source integrations to register named logical provider and physical plan codecs while preserving Ballista's built-in serialization.

Refs apache#1241

@milenkovicm milenkovicm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that i understand why is this needed in ballista main ?
ballista codec is fully pluggable at the moment, implementers can plug in its own implementation hence i dont see this needed in main, please correct me if i'm wrong

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants