[Spark] Identity Columns APIs in DeltaColumnBuilder

[c27kwan]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

This PR is part of #1959

• We introduce generateAlwaysAsIdentity and generatedByDefaultAsIdentityAPIs into DeltaColumnBuilder so that users can create Delta table with Identity column.
• We guard the creation of identity column tables with a feature flag until development is complete.

How was this patch tested?

New tests.

Does this PR introduce any user-facing changes?

Yes, we introduce generateAlwaysAsIdentity and generatedByDefaultAsIdentity interfaces to DeltaColumnBuilder for creating identity columns.
Interfaces

def generatedAlwaysAsIdentity(): DeltaColumnBuilder
def generatedAlwaysAsIdentity(start: Long, step: Long): DeltaColumnBuilder
def generatedByDefaultAsIdentity(): DeltaColumnBuilder
def generatedByDefaultAsIdentity(start: Long, step: Long): DeltaColumnBuilder

When the start and the step parameters are not specified, they default to 1L. generatedByDefaultAsIdentity allows users to insert values into the column while a column specified withgeneratedAlwaysAsIdentity can only ever have system generated values.

Example Usage

// Creates a Delta identity column.
io.delta.tables.DeltaTable.columnBuilder(spark, "id")
      .dataType(LongType)
      .generatedAlwaysAsIdentity()
// Which is equivalent to the call
io.delta.tables.DeltaTable.columnBuilder(spark, "id")
      .dataType(LongType)
      .generatedAlwaysAsIdentity(start = 1L, step = 1L)

[c27kwan]

I split the table feature part into a separate PR, which will need to be merged first: #2859

[larsk-db]

LGTM

[felipepessoto]

Wow, glad to see this PR so quickly. Are you planning to release it in 3.2.0?
Thanks.

[c27kwan]

It's a stretch goal at the moment, since the release for Delta 3.2 is quite soon. I should know by end of week whether I can realistically make 3.2. I plan to leave this PR up for at least one week anyways. :)

It's safe to start merging the code even if we can only make 4.0 since the DeltaSQLConf guards creating identity columns.

[c27kwan]

For context, this PR is on the critical path for identity columns implementation in Delta 3.2. We need an interface to specify identity columns. The SQL syntax can only be introduced in the next Spark release (Spark 4), which means we can only have it for Delta 4.0 earliest.

[felipepessoto]

Got it. Just for my knowledge, the required changes in Spark 4 are only to support SQL syntax, meaning Scala could be supported?

[c27kwan]

That's part of what I'm assessing this week in my prototype. To put the UDF for generating identity column values in only the Delta repo requires adding workarounds so that it works on Spark 3.5. Once I have enough confidence about the idea, I'll post a PR.

Everything else doesn't seem to require Spark changes.

[c27kwan]

I'm not going to make Delta 3.2. I've discussed with @tdas offline and we're aiming for the next Delta version, Delta 3.3. The API doc comment in this PR has been adjusted accordingly.

[ala]

It would be nice to tell the user what is the supported type and the unsupported type :)

[c27kwan]

I checked and it seems the other existing error message don't provide the list of supported types. It would be more consistent to not report the supported types.

[ala]

That's OK.

[ala]

LGTM.
Thank you!