Rename the partition field and add a field with the same name as the old partition field GOT ERROR

[madeirak]

Apache Iceberg version

1.4.3

Query engine

Spark

Please describe the bug 🐞

CREATE TABLE  db03.test_123 (
  id INT COMMENT '11',
  name STRING COMMENT ''
)USING iceberg
PARTITIONED BY (name1)
TBLPROPERTIES (
  'format-version' = '2'
)

1. Rename the partition field alter table db03.test_123 rename column name to name1;
2. Add a field with the same name as the old partition field alter table db03.test_123 add column name int;

then got error

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[huan233usc]

It seems that the on rename column, if the column is partition column, its name is not updated.
metadata file from my local testing ->
"schemas" : [ {
"type" : "struct",
"schema-id" : 0,
"fields" : [ {
"id" : 1,
"name" : "id",
"required" : false,
"type" : "int",
"doc" : "11"
}, {
"id" : 2,
"name" : "name",
"required" : false,
"type" : "string",
"doc" : ""
} ]
}, {
"type" : "struct",
"schema-id" : 1,
"fields" : [ {
"id" : 1,
"name" : "id",
"required" : false,
"type" : "int",
"doc" : "11"
}, {
"id" : 2,
"name" : "name1",
"required" : false,
"type" : "string",
"doc" : ""
} ]
} ],
"default-spec-id" : 0,
"partition-specs" : [ {
"spec-id" : 0,
"fields" : [ {
"name" : "name",
"transform" : "identity",
"source-id" : 2,
"field-id" : 1000
} ]
} ],

Or should it be updated?

[madeirak]

It seems that the on rename column, if the column is partition column, its name is not updated. metadata file from my local testing -> "schemas" : [ { "type" : "struct", "schema-id" : 0, "fields" : [ { "id" : 1, "name" : "id", "required" : false, "type" : "int", "doc" : "11" }, { "id" : 2, "name" : "name", "required" : false, "type" : "string", "doc" : "" } ] }, { "type" : "struct", "schema-id" : 1, "fields" : [ { "id" : 1, "name" : "id", "required" : false, "type" : "int", "doc" : "11" }, { "id" : 2, "name" : "name1", "required" : false, "type" : "string", "doc" : "" } ] } ], "default-spec-id" : 0, "partition-specs" : [ { "spec-id" : 0, "fields" : [ { "name" : "name", "transform" : "identity", "source-id" : 2, "field-id" : 1000 } ] } ],

Or should it be updated?

I think this is because the directory of the underlying data file contains the partition key name, but I think it should be blocked at the renaming or deletion step of old partition key with data, rather than being discovered later.

[huan233usc]

IIUC data file's name doesn't matter, I propose a fix of generating partition spec on rename identity partition key.