[Feature Request] Identity Column

[felipepessoto]

Feature request

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Overview

Identity Column (writer version 6) as defined by https://github.com/delta-io/delta/blob/master/PROTOCOL.md#identity-columns.

Design doc: https://docs.google.com/document/d/1G8Vj6wOxswMx1JklllLoSn-obEpJ-iE_Lhpbd-RfIr4/edit?usp=sharing

PR:

• [Spark] Identity Columns APIs in DeltaColumnBuilder #2857
• [Spark] Add IdentityColumnsTableFeature #2859
• [Spark] Add GenerateIdentityValues UDF #2915
• [Spark] Add IdentityColumn.scala #2916
• [Spark] ALTER TABLE ALTER COLUMN SYNC IDENTITY SQL support #3005
• [Spark] Identity Columns Value Generation (without MERGE support) #3023
• [Spark] Python DeltaTableBuilder API for Identity Columns #3404
• [Spark] Block unsupported operations on identity columns #3457
• [Spark] Support clone and restore for Identity Columns #3459
• [SPARK] Relax metadata conflict for identity column #3525
• [SPARK][TEST-ONLY] Add more tests for Identity Column #3526
• [SPARK] Allow non-deterministic expressions in actions of merge #3558
• [SPARK] Refactor IdentityColumnTestUtils #3568
• [SPARK] Add MERGE support for tables with Identity Columns #3566
• [SPARK][TEST-ONLY] Use unique table name for identity column tests #3594
• [SPARK] Move identity column feature out of dev mode #3598

Motivation

This is probably the biggest missing part in Open Source Spark Delta.

Further details

Willingness to contribute

@c27kwan volunteered to work on this feature and posted a design doc here.

[felipepessoto]

@dennyglee, @allisonport-db, do you have any update about this? This feature probably is the most important missing feature in OSS Delta.

Thanks.

[felipepessoto]

@tdas any chance this can be prioritized for next release?

Thanks.

[keen85]

duplicate of #1072?

[felipepessoto]

I think so. But I would update #1072 to be broader. The way the request is made it seems the Identity feature is already done, and it is only the DeltaTableBuilder API that is missing.

[c27kwan]

I'm interested on working on this!

[c27kwan]

I can't modify the main comment because i'm not a maintainer. Here's the design doc : https://docs.google.com/document/d/1G8Vj6wOxswMx1JklllLoSn-obEpJ-iE_Lhpbd-RfIr4/edit?usp=sharing

[felipepessoto]

@c27kwan that is great.

Have you discussed with any of the maintainers about your intention to contribute? I’m asking because this is a big feature and I just want to make sure they aren’t internally working on it and they are open to accept your implementation.

Thanks.

[vkorukanti]

Hi @felipepessoto, we don't have anyone else working on this feature. Had an offline chat with @c27kwan before assigning the issue to @c27kwan. Feel free to look at the design doc and post any questions you have.