Core: Fix non-setting row-lineage from table properties on initial table creation

[tomtongue]

Overview

Fix the row-lineage table property reflection on enableRowLineage.

Issue

Currently to enable the Row Lineage feature from the Iceberg table properties, it's required to run the following operations:

1. Create an Iceberg table
2. Update table properties

At the first step "Create an Iceberg table", even if you set row-lineage to true in the table properties, the property isn't reflected on the Iceberg table's metadata.json. Therefore, to enable that feature, you need to additionally run table properties update after creating an Iceberg table.

Details

Tested two cases such as Spark and Java API

Spark case

When you create an Iceberg table using Spark like the following query,

spark.sql("""
CREATE TABLE db.rowlin (id int, name string, year int) USING iceberg
TBLPROPERTIES ('format-version'='3', 'row-lineage'='true')
LOCATION 's3://bucket/iceberg-v3/row-lineage'
""")

The relevant metadata.json is stored in the specified bucket and path as below:

aws s3 ls s3://bucket/iceberg-v3/row-lineage/ --recursive
2025-02-18 16:56:28       1194 iceberg-v3/row-lineage/metadata/00000-1eb8c96e-f503-4ff9-b4e0-53cb3ede0116.metadata.json

At this point, the metadata content (partial) is below. The content doesn't have row-lineage even if the parameter is in the properties part.

{
  "format-version" : 3,
  "table-uuid" : "eaf5dec9-7866-49a5-81c6-11af8f344e1f",
  "location" : "s3://bucket/iceberg-v3/row-lineage",
  "last-sequence-number" : 0,
  "last-updated-ms" : 1739865386995,
  "last-column-id" : 3,
  "current-schema-id" : 0,
  "schemas" : [ {
    "type" : "struct",
    "schema-id" : 0,
    "fields" : [ { ... } ]
  } ],
  "default-spec-id" : 0,
  "partition-specs" : [ {
    "spec-id" : 0,
    "fields" : [ ]
  } ],
  "last-partition-id" : 999,
  "default-sort-order-id" : 0,
  "sort-orders" : [ {
    "order-id" : 0,
    "fields" : [ ]
  } ],
  "properties" : {
    "owner" : "hadoop",
    "write.update.mode" : "merge-on-read",
    "write.parquet.compression-codec" : "zstd",
    "row-lineage" : "true"
  },
  "current-snapshot-id" : null,
...
}

And then, update the table property by the same table property like ALTER TABLE db.rowlin SET TBLPROPERTIES('row-lineage'= 'true').

After the query is complete, the content of the new metadata.json is below. row-lineage and next-row-id is added.

{
  "format-version" : 3,
  "table-uuid" : "eaf5dec9-7866-49a5-81c6-11af8f344e1f",
  "location" : "s3://bucket/iceberg-v3/row-lineage",
  "last-sequence-number" : 0,
  "last-updated-ms" : 1739865514775,
  "last-column-id" : 3,
  "current-schema-id" : 0,
  "schemas" : [ {
    "type" : "struct",
    "schema-id" : 0,
    "fields" : [ { ... } ]
  } ],
  "default-spec-id" : 0,
  "partition-specs" : [ {
    "spec-id" : 0,
    "fields" : [ ]
  } ],
  "last-partition-id" : 999,
  "default-sort-order-id" : 0,
  "sort-orders" : [ {
    "order-id" : 0,
    "fields" : [ ]
  } ],
  "properties" : {
    "owner" : "hadoop",
    "write.update.mode" : "merge-on-read",
    "write.parquet.compression-codec" : "zstd",
    "row-lineage" : "true"
  },
  "current-snapshot-id" : null,
  "row-lineage" : true,  // <= ADDED
  "next-row-id" : 0, // <= ADDED
  "refs" : { },
  "snapshots" : [ ],
  "statistics" : [ ],
  "partition-statistics" : [ ],
  "snapshot-log" : [ ],
  "metadata-log" : [ {
    "timestamp-ms" : 1739865386995,
    "metadata-file" : "s3://bucket/iceberg-v3/row-lineage/metadata/00000-1eb8c96e-f503-4ff9-b4e0-53cb3ede0116.metadata.json"
  } ]
}

Here's the diff between two metadata files:

$ diff 00000-1eb8c96e-f503-4ff9-b4e0-53cb3ede0116.metadata.json 00001-ebf641c8-9603-45d5-92c6-dafac315375e.metadata.json
6c6
<   "last-updated-ms" : 1739865386995,
---
>   "last-updated-ms" : 1739865514775,
46a47,48
>   "row-lineage" : true,
>   "next-row-id" : 0,
52c54,57
<   "metadata-log" : [ ]
---
>   "metadata-log" : [ {
>     "timestamp-ms" : 1739865386995,
>     "metadata-file" : "s3://gsweep/iceberg-v3/row-lineage-mor13/metadata/00000-1eb8c96e-f503-4ff9-b4e0-53cb3ede0116.metadata.json"
>   } ]

Java API case

Example script:

public class Main {
    private static final RESTCatalog catalog = new RESTCatalog();
    private static final Schema schema = new Schema(
            Types.NestedField.required(1, "id", Types.IntegerType.get()),
            Types.NestedField.required(2, "name", Types.StringType.get()),
            Types.NestedField.required(3, "year", Types.IntegerType.get()),
            Types.NestedField.required(4, "category", Types.StringType.get())
    );

    public static void main(String[] args) throws IOException {
        Map<String, String> properties = new HashMap<>();
        properties.put(CatalogProperties.CATALOG_IMPL, "org.apache.iceberg.rest.RESTCatalog");
        properties.put(CatalogProperties.URI, "http://localhost:8181");
        properties.put(CatalogProperties.FILE_IO_IMPL, S3FileIO.class.getName());

        catalog.setConf(new Configuration());
        catalog.initialize("rest", properties);

        // Params
        String db = "db";
        Namespace ns = Namespace.of(db);
        String tableName = "rowlin_java";
        String location = String.format("s3://bucket/iceberg-java/%s/%s", db, tableName);
        Map<String, String> tblprops = new HashMap<>();
        tblprops.put("format-version", "3");
        tblprops.put(TableProperties.ROW_LINEAGE, "true");
        TableIdentifier tbl = TableIdentifier.of(ns, tableName);

        // Create a table
        createTableIfNotExists(db, tableName, schema, location, tblprops);

        // Load the table
        Table table = catalog.loadTable(tbl);

        // Update table property
        table.updateProperties().set(TableProperties.ROW_LINEAGE, "true").commit();
    }

    private static void createTableIfNotExists(String db, String table, Schema schema, String location, Map<String, String> properties) {
        TableIdentifier tbl = TableIdentifier.of(db, table);
        Namespace ns = Namespace.of(db);
        if (!catalog.listTables(ns).contains(tbl)) {
            catalog.createTable(tbl, schema, null, location, properties);
        }
    }
}

After running the script, two versions of metadata.json files are created in the specified s3 bucket:

$ aws s3 ls s3://bucket/iceberg-java/db/rowlin_java --recursive
2025-02-18 19:54:11       1224 iceberg-java/db/rowlin_java/metadata/00000-6e7ccb2c-4eba-45e5-8cda-500e487729a3.metadata.json
2025-02-18 19:54:15       1441 iceberg-java/db/rowlin_java/metadata/00001-74955cfb-05d3-4b95-9f74-651c16389b3e.metadata.json

Each content of the metadata file is below:

// 00000-6e7ccb2c-4eba-45e5-8cda-500e487729a3.metadata.json
$ aws s3 cp s3://bucket/iceberg-java/db/rowlin_java/metadata/00000-6e7ccb2c-4eba-45e5-8cda-500e487729a3.metadata.json - | grep row -n3 -> there's no row-lineage

1-{
2-  "format-version" : 3,
3-  "table-uuid" : "27529028-4f30-4100-a3e7-c23984ab2ccb",
4:  "location" : "s3://bucket/iceberg-java/db/rowlin_java",
5-  "last-sequence-number" : 0,
6-  "last-updated-ms" : 1739876049987,
7-  "last-column-id" : 4,
--
44-  } ],
45-  "properties" : {
46-    "write.parquet.compression-codec" : "zstd",
47:    "row-lineage" : "true"
48-  },
49-  "current-snapshot-id" : null,
50-  "refs" : { },

// 00001-74955cfb-05d3-4b95-9f74-651c16389b3e.metadata.json
$ aws s3 cp s3://bucket/iceberg-java/db/rowlin_java/metadata/00001-74955cfb-05d3-4b95-9f74-651c16389b3e.metadata.json - | grep row -n3 -> row-lineage is added.

1-{
2-  "format-version" : 3,
3-  "table-uuid" : "27529028-4f30-4100-a3e7-c23984ab2ccb",
4:  "location" : "s3://bucket/iceberg-java/db/rowlin_java",
5-  "last-sequence-number" : 0,
6-  "last-updated-ms" : 1739876054823,
7-  "last-column-id" : 4,
--
44-  } ],
45-  "properties" : {
46-    "write.parquet.compression-codec" : "zstd",
47:    "row-lineage" : "true"
48-  },
49-  "current-snapshot-id" : null,
50:  "row-lineage" : true, // <= ADDED
51:  "next-row-id" : 0, // <= ADDED
52-  "refs" : { },
53-  "snapshots" : [ ],
54-  "statistics" : [ ],
--
56-  "snapshot-log" : [ ],
57-  "metadata-log" : [ {
58-    "timestamp-ms" : 1739876049987,
59:    "metadata-file" : "s3://gsweep/iceberg-java/db/rowlin_java/metadata/00000-6e7ccb2c-4eba-45e5-8cda-500e487729a3.metadata.json"
60-

[tomtongue]

@RussellSpitzer I believe you're working on the row-lineage feature. When you have a chance, could you check this issue and review the change?

[RussellSpitzer]

This was intentional. We are avoiding writing the property so that Table Metadata Versions that do not include row-lineage at all will not populate the fields.

[RussellSpitzer]

Ah wait I understand this more in your code than in this description. You are just saying that you can't enable row-lineage during a Create statement. That sounds like it's fine to fix.

[RussellSpitzer]

We don't want to set RowLineage to false if it is unset. So we have 2 options here

1. Set it to false but only if the table is V3
2. Set it to true only if it is true and leave it absent otherwise.

[tomtongue]

Sure, thanks for checking. Let me fix this.

[RussellSpitzer]

You actually don't have to do anything! I forgot I already did this in "setRowLineage". It won't set the field unless it's getting set to true.

[tomtongue]

@RussellSpitzer If you don't set row-lineage in the table properties, it's unset, I mean it's kept empty because setRowLineage can handle null argument such as:

iceberg/core/src/main/java/org/apache/iceberg/TableMetadata.java

Lines 1525 to 1528 in bcbbd03

	private Builder setRowLineage(Boolean newRowLineage) {
	if (newRowLineage == null) {
	return this;
	}

[tomtongue]

Oh yes, it is. Thanks for letting me know. So I believe this part should be fine.

[tomtongue]

Ah wait I understand this more in your code than in this description. You are just saying that you can't enable row-lineage during a Create statement. That sounds like it's fine to fix.

Thanks so much for the quick review! Yes, currently, it's not possible to enable row-lineage in CreateTable statement, and it's always required to set row-lineage: true again even if row-lineage is set true in CreateTable.

Additionally, after creating an Iceberg table without setting row-lineage: true in the table properties, even if you set the row lineage parameter into the table properties, the row lineage feature is not enabled. So, for now, it's always needed to update table properties with ALTER TABLE SET TBLPROPERTIES, updateProperties etc.

This fix enables the row lineage feature once row-lineage: true is set into the Iceberg table properties.

[RussellSpitzer]

Thanks @tomtongue !

[tomtongue]

Thanks so much for the quick review! @RussellSpitzer