java.lang.IllegalArgumentException: schemas must exist in format v2

[ayush-san]

While updating iceberg flink runtime version from 0.11.0 to 0.12.1, I am getting the following error

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: schemas must exist in format v2
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
	at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: java.lang.IllegalArgumentException: schemas must exist in format v2
	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:413)
	at org.apache.iceberg.TableMetadataParser.fromJson(TableMetadataParser.java:310)
	at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:258)
	at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:252)
	at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:179)
	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:405)
	at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)
	at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:178)
	at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:160)
	at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:200)
	at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:94)
	at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:77)
	at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:93)
	at org.apache.iceberg.catalog.Catalog.tableExists(Catalog.java:270)
	at cdc.FlinkDebezium.getIcebergV2Table(FlinkDebezium.java:171)
	at cdc.FlinkDebezium.run(FlinkDebezium.java:119)
	at cdc.FlinkDebezium.build(FlinkDebezium.java:83)
	at cdc.FlinkDebezium.main(FlinkDebezium.java:194)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)

My iceberg tables were created in v2 format using the following snippet

org.apache.iceberg.Table icebergTable = catalogLoader.loadCatalog().createTable(tableIdentifier,
        schema,
        partitionSpec,
        tableProperties);

// need to upgrade version to 2,otherwise 'java.lang.IllegalArgumentException: Cannot write
// delete files in a v1 table'
TableOperations tableOperations = ((BaseTable) icebergTable).operations();
TableMetadata metadata = tableOperations.current();
tableOperations.commit(metadata, metadata.upgradeToFormatVersion(2));

[ayush-san]

@openinx @stevenzwu Can you please help here

[openinx]

@ayush-san , I think you use the 0.11.x iceberg release to upgrade the iceberg table from v1 to v2. The correct way to upgrade the v1 to v2 is using 0.12.x because 0.12.0 is the correct release that we've voted to stablize the v2 format. You can see the mail list: https://lists.apache.org/thread/ws2gg52d124p7bx9jgrn3kctrtfgtltp

In fact, during the 0.11.x to 0.12.x we've introduced another iceberg spec changes such as https://github.com/apache/iceberg/pull/2096/files. So 0.11.x is not the correct release to upgrade the v1 for v2.

If you want to experience the iceberg table to maintain the flink cdc events, you may want to take a look at this post: https://ververica.github.io/flink-cdc-connectors/master/content/quickstart/build-real-time-data-lake-tutorial.html

[ayush-san]

@openinx My tables were already created in v2 format using 0.11.x iceberg version. Now after the release of 0.12.1, I wanted to move my flow to it.

Can you suggest how can I do that now?

[openinx]

@ayush-san , then I think we will write a simple tool to convert the unstable v2 format from 0.11.x to the stable v2 format like 0.12.1 version. I think the key thing is converting the schema changes which was introduced from #2096.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'