Skip to content

feat: allow extension DataSources to decode via a per-type registry - #26555

Draft
gabotechs wants to merge 1 commit into
apache:mainfrom
gabotechs:gabotechs/datasource-proto-registry
Draft

gabotechs wants to merge 1 commit into
apache:mainfrom
gabotechs:gabotechs/datasource-proto-registry

Conversation

@gabotechs

@gabotechs gabotechs commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Follow-up to #24631 and #24628. No separate issue.

Rationale for this change

Custom DataSource implementations can encode themselves through try_to_proto, but still need a PhysicalExtensionCodec to decode. Applications combining sources from multiple libraries therefore have to compose their codecs.

Extend the session-scoped decoder registry to sources so each source can own its decoder and register it by name, following the extension execution-plan and expression APIs.

What changes are included in this PR?

  • Add ExtensionDataSourceFromProto, register_data_source, and extension_data_source_node. Constructors receive a typed PhysicalExtensionDataSourceNode and return Arc<dyn DataSource>.
  • Register DataSourceExec as a shared extension-plan decoder. Its payload contains the source name and bytes; the wrapper looks up the source decoder and rebuilds DataSourceExec through the existing plan registry dispatch. No new top-level plan variant is needed. Source names are independent of plan and expression names.
  • Once the wrapper is registered, missing source names, unregistered sources, and decoder errors fail directly. If the wrapper itself is unregistered, existing plan-codec fallback applies to the wrapper payload. Built-in scans and existing codec-encoded sources retain their wire formats.

What is the testing strategy for this PR?

Six registry unit tests and fourteen integration tests cover registration, execution after decoding, reading-session state, malformed wrapper and source payloads, codec dispatch boundaries, legacy codec compatibility, shared names across extension kinds, and binary/JSON round trips.

Validation: 541 affected-crate tests and 14 doctests passed; formatting, workspace-wide Clippy (cargo clippy --all-targets --all-features -- -D warnings), rustdoc with warnings denied, protobuf regeneration, and documentation formatting passed.

Are there any user-facing changes?

New source-registry APIs behind the existing proto feature. Built-in source serialization is unchanged. Readers must register source decoders before writers emit the new wrapper payload. Documented in the 56.0.0 upgrade guide.

AI-assisted implementation with Codex.

@gabotechs gabotechs added the api change Changes the API exposed to users of the crate label Oct 21, 2026
@github-actions github-actions Bot added documentation Improvements or additions to documentation proto Related to proto crate datasource Changes to the datasource crate labels Oct 21, 2026
@gabotechs
gabotechs force-pushed the gabotechs/datasource-proto-registry branch from 86c375a to 8c43f67 Compare October 21, 2026 08:30
@gabotechs

Copy link
Copy Markdown
Contributor Author

cc @adriangb I want to tackle this one myself in order to get some hands-on exp with the approach here. Still iterating on this, so not looking for a review.

@gabotechs
gabotechs force-pushed the gabotechs/datasource-proto-registry branch from 8c43f67 to bceb61d Compare October 21, 2026 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api change Changes the API exposed to users of the crate datasource Changes to the datasource crate documentation Improvements or additions to documentation proto Related to proto crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant