Skip to content

Conversation

@jared-m-combs
Copy link
Contributor

It is sometimes useful to have additional information in the file path that we can access programatically but are not 'normal' partitions. For instance, it can be helpful to encode versioning or hash metadata into the path. However, this can prevent the stale dependencies UTF from correctly matching the file metadata against its dependencies.

To work around this, I've added the notion of a 'special' partition. These partitions are prefixed with an underscore and must be the leaf most nodes in the path. This makes it very easy to omit these form the path in the same way that we already omit file names.

It is sometimes useful to have additional information in the file path
that we can access programatically but are not 'normal' partitions. For
instance, it can be helpful to encode versioning or hash metadata into
the path. However, this can prevent the stale dependencies UTF from
correctly matching the file metadata against its dependencies.

To work around this, I've added the notion of a 'special' partition.
These partitions are prefixed with an underscore and must be the leaf
most nodes in the path. This makes it very easy to omit these form the
path in the same way that we already omit file names.
Copy link
Member

@xudong963 xudong963 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@matthewmturner matthewmturner merged commit 819843c into datafusion-contrib:main Jun 19, 2025
8 checks passed
@github-actions github-actions bot mentioned this pull request Jun 19, 2025
@github-actions github-actions bot mentioned this pull request Oct 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants