feat: Additional Canonical Extension Types by tobixdev · Pull Request #21291 · apache/datafusion

tobixdev · 2026-04-01T08:41:43Z

Which issue does this PR close?

Related to Implement DFExtensionType for Arrow's Canonical Extension Types #21144 .

Rationale for this change

Implements DFExtensionType for the other canonical extension types (except Parquet types).

What changes are included in this PR?

Implement DFExtensionType for:
- arrow.bool8 (custom formatting with true/false)
- arrow.fixed_shape_tensor (nu custom formatting for now)
- arrow.json (no custom formatting for now)
- arrow.opaque (no custom formatting for now)
- arrow.timestamp_with_offset (custom formatting as timestamp)
- arrow.variable_shape_tensor (no custom formatting)
Introduce a wrapper struct DFUuid for Uuid so that it is consistent with the other extension types. I don't know whether we truly need the wrapper structs for extension types that only have a single valid storage type based on their metadata (e.g., Bool8, Uuid) . Open for any opinions.

@paleolimbot Is this the kind of wrapper structs you imagined?

Should we also add end-to-end tests for the other extension types? Currently, we only have the UUID one from the last PR.

I think for variant supports it's best to wait for general Variant support in DataFusion as this is currently done in https://github.com/datafusion-contrib/datafusion-variant .

Are these changes tested?

Custom formatters are tested. The rest is mainly boiler plate code.

Are there any user-facing changes?

Yes more extension types and renames Uuid to DFUuid. If we plan to keep our own wrapper structs, we should merge that before releasing a version with the old Uuid so we have no breaking changes.

datafusion/common/src/types/canonical_extensions/timestamp_with_offset.rs

paleolimbot

This looks great!

I may be missing something about the arrow-rs ExtensionType, but I'm not sure we need to impl it for these (which can be constrained to datafusion-specific behaviour).

I think datafusion-common is a great place for these now...there might be type-specific crates in the future that include the dependencies required to actually do things (e.g., the datafusion JSON contribution being discussed and perhaps a future variant), and when that's the case the type definition could maybe live there too.

paleolimbot · 2026-04-02T01:55:49Z

datafusion/common/src/types/canonical_extensions/bool8.rs

+impl ExtensionType for DFBool8 {
+    const NAME: &'static str = Bool8::NAME;
+    type Metadata = <Bool8 as ExtensionType>::Metadata;


I'm not sure this is needed (only the impl DFExtensionType bit). Does the fact that these objects implement ExtensionType allow something to be done with them that isn't otherwise possible?

Yep, this was also a thorn in my eye. I think the only thing is the default registration that requires ExtensionType + DFExtensionType.

I've removed this requirement from the default registration. I think it's better now, especially because now there is only a single ExtensionType implementation that can be used for reading/writing metadata. The DFBool8 etc. types can then be ignored by most users, except those that want to customize the registered extension types.

After this change, I think the example is a bit weird as it implements both traits in the same struct.

For example, ExtensionType::supports_data_type now returns true for all supported data types (as it should for the arrow-rs API), but it may seem unexpected as it just stores a single sotrage type. So some users could assume that it should only return true for this specific type.

Do you think it warrants splitting the example into TemperatureExtensionType and DFTemperatureExtensionType too to avoid this?

Or just remove the ExtensionType implementation there? The storage type check can move into new() and you can remove a layer of indirection.

(It is my complaint with the arrow-rs ExtensionType trait that there is pretty much zero incentive to implement it because it doesn't help anybody actually do anything).

Also a good idea! That removed the weirdness I mentioned above.

This also allowed me to change the registration to not depend on ExtensionType. Therefore, the shenanigans around DefaultExtensionTypeRegistration are no longer necessary to support extension types that do not implement the arrow-rs trait.

…n types

tobixdev · 2026-04-13T06:54:16Z

@adriangb Sorry for pinging you! It's just that I think we should merge this (or a similar PR) within this release cycle as it contains breaking changes of DFExtensionType API. After mergeing this one, I (or @paleolimbot if you're still interessted) can polish the draft PR on casting extension types.

adriangb · 2026-04-13T15:22:13Z

@adriangb Sorry for pinging you! It's just that I think we should merge this (or a similar PR) within this release cycle as it contains breaking changes of DFExtensionType API. After mergeing this one, I (or @paleolimbot if you're still interessted) can polish the draft PR on casting extension types.

I'm happy to help but what is the best use of my time here? Do you want me to review the PR (in addition to @paleolimbot to get 2 approvals? I don't see that he's approved it yet)? Or are there high level questions to chime in on?

paleolimbot

(Sorry for being slow to re-review and update the casting PR...my current work project is not in Rust and it's been tricky to flip my brain back and forth to this 😬 )

I think this PR represents minor but useful tweaks to the traits resulting from the tire-kicking of implementing a few more types. The key change is that a DFExtensionType is no longer strictly coupled to the arrow-rs ExtensionType (although can and does effectively reuse the existing implementations that apply to use in DataFusion). The other change is to strictly store the storage type of a DFExtensionType to the instance (it was required to construct the extension type, this just made it explicit). This makes it easier to pass around fewer objects when working with them (i.e., you can just pass around the extension type and not the extension type AND the data type everywhere).

A useful follow-up would be to map the corresponding SQL types (and I will rebase the casting PR on this and get it ready for review this week).

adriangb

Seems reasonable to me! I look forward to seeing how e.g. DFTimestampWithOffsetCol = TimestampUtcCol will work.

datafusion-examples/examples/extension_types/temperature.rs

adriangb · 2026-04-13T15:54:56Z

datafusion/common/src/types/canonical_extensions/bool8.rs

+        metadata: <Bool8 as ExtensionType>::Metadata,
+    ) -> Result<Self> {
+        Ok(Self(<Bool8 as ExtensionType>::try_new(
+            data_type, metadata,


Should the supported data types be checked here as well? Maybe add to the docstring what data_type represents (it seems DFExtensionType::storage_type is hardcoded below so it's not immediately obvious why this is passed into the constructor).

Good point that this is not obvious. The validation happens in <Bool8 as ExtensionType>::try_new.

datafusion/common/src/types/canonical_extensions/bool8.rs

datafusion/common/src/types/canonical_extensions/fixed_shape_tensor.rs

adriangb · 2026-04-13T15:58:52Z

datafusion/common/src/types/canonical_extensions/fixed_shape_tensor.rs

+    ) -> Result<Self> {
+        Ok(Self {
+            inner: <FixedShapeTensor as ExtensionType>::try_new(data_type, metadata)?,
+            storage_type: data_type.clone(),


I will say this is a bit unwieldy. I don't have a better proposal but wanted to note it.

I agree. Users could implement their DFExtensionType without using the arrow-rs trait (like we did in our example) or provide a common function that can be used on both trait implementations. The problem in this case is that <FixedShapeTensor as ExtensionType>::try_new does some processing that I do not want to duplicate.

So I think users can work around that issue.

adriangb · 2026-04-13T15:59:51Z

datafusion/common/src/types/canonical_extensions/fixed_shape_tensor.rs

+    fn storage_type(&self) -> DataType {
+        self.storage_type.clone()
+    }
+
+    fn serialize_metadata(&self) -> Option<String> {
+        self.inner.serialize_metadata()
+    }


Are there cases where the metadata depends on the storage type?

I guess one could think of such a type. E.g., a type that stores the serialized DataType in the metadata and then is only compatible with this exact DataType.

I am not sure how useful that is. Any concerns regarding that?

If it does, self.storage_type() is available here (but I'm not aware of any extension types that duplicate information in the metadata and that would otherwise be inferrable from the storage type)

adriangb · 2026-04-13T16:20:55Z

datafusion/common/src/types/canonical_extensions/opaque.rs

+
+/// Defines the extension type logic for the canonical `arrow.opaque` extension type.
+///
+/// See [`DFExtensionType`] for information on DataFusion's extension type mechanism.


In addition to a link to the canonical extension types (e.g. https://arrow.apache.org/docs/format/CanonicalExtensions.html#opaque) it might be nice to copy a summary of the type into here for discoverability.

Most of the extension types did not have a consice high-level description that I could copy, so I've added a sentence describing the type.

I've also linked to the arrow-rs implementation that contains information regarding the metadata and so forth.

tobixdev · 2026-04-13T16:47:13Z

Thanks! In my head Dewey already re-reviewed the PR and we were waiting for a comitter's blessing. Seems I had remembered that wrongly. Sorry about that!

I'll implement your suggestions tomorrow and I'll look into the SQL type mapping.

paleolimbot · 2026-04-14T14:51:44Z

datafusion/common/src/types/canonical_extensions/fixed_shape_tensor.rs

+    fn storage_type(&self) -> DataType {
+        self.storage_type.clone()
+    }
+
+    fn serialize_metadata(&self) -> Option<String> {
+        self.inner.serialize_metadata()
+    }


If it does, self.storage_type() is available here (but I'm not aware of any extension types that duplicate information in the metadata and that would otherwise be inferrable from the storage type)

adriangb · 2026-04-14T15:22:04Z

@tobixdev good to merge?

tobixdev · 2026-04-14T15:49:47Z

@adriangb Good to merge from my side!

adriangb · 2026-04-14T16:08:44Z

Thanks folks! Looking forward to seeing the cast work! One other thing that comes to mind (more of an arrow-rs request): can we customize how data is encoded to JSON / CSV? E.g. for uuid it would make sense to encode it as a hex string / same as the pretty print, not as the raw bytes in base64 or something.

paleolimbot · 2026-04-14T16:25:10Z

can we customize how data is encoded to JSON / CSV?

For CSV this is doable after the cast work (cast to string!). JSON is probably better special-cased so that the output can be written incrementally if the underlying framework supports this, but you could also get away with a cast to arrow.json (falling back to a cast to string).

tobixdev · 2026-04-14T16:29:14Z

Me too! From what I can gather by quickly looking at the arrow-rs we would need a way of customizing the CSV writer. As it already seems to use ArrayFormatter, maybe something like a HashMap<FieldName, ArrayFormatter> could also address that.

Similar to apache/arrow-rs#8829 for the pretty-printing changes.

You beat me to an answer ;) I'll just leave it here.

tobixdev added 6 commits April 1, 2026 09:55

Add bool8

e3c7f27

Add draft for more extension types

a3604ff

Split tensor extension types into multiple files

754a8d7

Fix license headers

78e9404

Add null support for TimestampWithOffset display

6a42564

Minor changes

d62be49

tobixdev changed the title ~~Additional Canonical Extension Types~~ feat: Additional Canonical Extension Types Apr 1, 2026

github-actions bot added logical-expr Logical plan and expressions common Related to common crate labels Apr 1, 2026

tobixdev commented Apr 1, 2026

View reviewed changes

datafusion/common/src/types/canonical_extensions/timestamp_with_offset.rs Outdated Show resolved Hide resolved

tobixdev added 2 commits April 1, 2026 18:41

Support further offset types

e902af2

Fix issue with use statement

150b685

paleolimbot reviewed Apr 2, 2026

View reviewed changes

tobixdev added 5 commits April 2, 2026 10:23

Remove necessity to implement arrow-rs' ExtensionType for DF extensio…

5cbf369

…n types

Fix temperature example

f7461a6

Fix CI

8e63149

Remove requirement to arrow-rs ExtensionType

28578af

Make ExtensionTypeRegistrations less complicated

7b71349

tobixdev mentioned this pull request Apr 8, 2026

feat: implement DFExtensionType for remaining canonical Arrow extension types #21458

Closed

paleolimbot approved these changes Apr 13, 2026

View reviewed changes

adriangb approved these changes Apr 13, 2026

View reviewed changes

Feedback

fe78a83

paleolimbot approved these changes Apr 14, 2026

View reviewed changes

adriangb added this pull request to the merge queue Apr 14, 2026

tobixdev mentioned this pull request Apr 14, 2026

Support Mapping SQL Types to the Canonical Extension Types #21626

Open

Merged via the queue into apache:main with commit 3763ad4 Apr 14, 2026
35 checks passed

Conversation

tobixdev commented Apr 1, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tobixdev Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tobixdev commented Apr 13, 2026

Uh oh!

adriangb commented Apr 13, 2026

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

adriangb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tobixdev commented Apr 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangb commented Apr 14, 2026

Uh oh!

tobixdev commented Apr 14, 2026

Uh oh!

adriangb commented Apr 14, 2026

Uh oh!

paleolimbot commented Apr 14, 2026

Uh oh!

tobixdev commented Apr 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tobixdev Apr 2, 2026 •

edited

Loading