Skip to content

Use SimpleExtensions for Substrait type variations #11544

Open

Description

@Blizzara

Describe the bug

Substrait has an extension mechanism for defining things not included in the protobuf format. See https://substrait.io/extensions/#simple-extensions for definition and https://substrait.io/tutorial/sql_to_substrait/#plans for an example (the example only has scalar functions but idea is the same for types and type variations).

DF has been using the extensions for functions, and #11510 adds support for extension types. However DF still uses self-defined consts for type variations (see https://github.com/apache/datafusion/blob/12d82c427d6c37f7884a508707ccd3058a446908/datafusion/substrait/src/variation_const.rs), rather than writing the variations as extensions in the plan. That makes it hard for other producers/consumers to work with DF variations, since they'd need to match the constants, rather than matching by name (+uri).

We should move type variations to also produce simple extensions.

To Reproduce

No response

Expected behavior

No hardcoded references; type variations written as SimpleExtensionDeclarations à la https://github.com/substrait-io/substrait/blob/a68c1ac62f92d703da624cb8ac0cef854dd2b35f/extensions/type_variations.yaml

Additional context

DF also doesn't use the extensionUris mechanism correctly, the field isn't filled and all extensionUriReference are just hardcoded into max u32. That's filed separately as #11545

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions