Skip to content

fix(spec): accept partition specs produced by spec evolution - #3431

Open
dhruvarya-db wants to merge 1 commit into
apache:mainfrom
dhruvarya-db:fix-bind-evolved-partition-specs
Open

dhruvarya-db wants to merge 1 commit into
apache:mainfrom
dhruvarya-db:fix-bind-evolved-partition-specs

Conversation

@dhruvarya-db

@dhruvarya-db dhruvarya-db commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

  • None filed. This fixes a bug that surfaces when committing partition specs produced by partition spec evolution.

What changes are included in this PR?

Catalogs built on iceberg-rust apply an AddSpec update through TableMetadataBuilder::add_partition_spec, which binds the spec with the same checks used to author a new one. Those checks reject specs that partition spec evolution legitimately produces, and that Java writes and accepts:

  • Several fields on one source column. In Java, updateSpec().addField(hour("ts")) on a table partitioned by day(ts) produces a spec with both ts_day and ts_hour. On a format version 1 table, each removed field stays as a void placeholder, so one column can have several void fields. Binding ran the authoring-time redundancy check, so UnboundPartitionSpec::bind, PartitionSpecBuilder::new_from_unbound and TableMetadataBuilder::add_partition_spec rejected these specs. Loading v1 metadata that has only the legacy partition-spec field did too. Both now skip the check, as Java's PartitionSpec.Builder.add does. PartitionSpecBuilder::add_unbound_field and UnboundPartitionSpecBuilder::add_partition_field still reject redundant fields when authoring a spec.
  • A void field named after its source column. Removing identity(category) from a v1 spec in Java leaves a void field named category. Both the spec builder and TableMetadataBuilder::validate_partition_field_names allowed only identity fields to share their source column's name. Void fields may now do the same, matching Java's alwaysNull(source, name). Java goes further and allows any transform named after its own source column; this PR doesn't.

Metadata with partition-specs already loaded, because that field is deserialized without rebinding. Only committing these specs failed.

#3413 also edits validate_partition_field_names, so whichever of the two lands second needs a small rebase.

Are these changes tested?

Yes, with unit tests:

  • binding a spec with two time granularities and two void fields on one column;
  • rejecting a void field named after a different column;
  • the authoring path still rejecting a second time granularity;
  • TableMetadataBuilder::add_partition_spec accepting the void placeholders of a v1 table;
  • loading legacy v1 partition-spec metadata with two void fields on one column.

Each fails without the fix.

AI Disclosure

This PR was written with Claude Opus 5.5 and reviewed by me.

Partition spec evolution produces specs that the spec builders rejected:

- A source column can have more than one partition field. Adding hour(ts) to a
  spec that has day(ts) is allowed, and format version 1 tables keep each
  removed field as a `void` placeholder. Binding an unbound spec
  (`UnboundPartitionSpec::bind`, `PartitionSpecBuilder::new_from_unbound`, and
  so `TableMetadataBuilder::add_partition_spec`) and loading the legacy v1
  `partition-spec` field ran the authoring-time redundancy check and rejected
  these specs. Both now skip it, as Java does.
  `PartitionSpecBuilder::add_unbound_field` and
  `UnboundPartitionSpecBuilder::add_partition_field` still reject redundant
  fields.
- Removing identity(category) from a format version 1 spec leaves a void field
  named `category`. Both the spec builder and the table metadata builder
  allowed only identity fields to share the name of their source column. Void
  fields may now do the same, matching Java's `alwaysNull`.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants