Add support for default column values in engine

[ebyhr]

Description

This PR adds support for the DEFAULT clause with literal values when creating a new table or adding a column.
The SHOW CREATE TABLE statement now includes the specified default values.
These defaults are respected in INSERT and MERGE operations.
Non-literal expressions such as <datetime value function>, USER, and similar are not supported in this PR.

<default clause> ::=
  DEFAULT <default option> 

<default option> ::=
    <literal> 
  | <datetime value function> 
  | USER
  | CURRENT_USER
  | CURRENT_ROLE
  | SESSION_USER
  | SYSTEM_USER
  | CURRENT_CATALOG
  | CURRENT_SCHEMA
  | CURRENT_PATH
  | <implicitly typed value specification> 

Release notes

## General, Memory
* Add support for default column values. ({issue}`25679`)

[kasiafi]

If the connector supportsMissingColumnsOnInsert, then it is supposed to apply the default value on its own, right?
Do we have tests to cover both cases (connector applies the default value versus the engine applies it here)?

[ebyhr]

Right. We don't have such a test. I'm wondering if we can write a helpful test covering both cases in the engine.

[ebyhr]

Addressed comments.

[kasiafi]

Do we plan to support other variants of the default value in the future? For values like CURRENT_USER or <datetime value function>, we cannot store them statically. I think it's fine to special-case for constants for now, but if we want to support the other options, we will need another api to store expressions.

[ebyhr]

I'm a little wondering whether fallback to existing implementation is really good idea. The behavior between default-null and missing-columns might be different in remote connectors. For instance, ClickHouse has default values in each type (e.g. 1 in integer). DEFAULT null might be used to enforce null in such connectors.

I changed NullableValue to ConnectorExpression so we can support non-constants in the future.

[kasiafi]

I added a comment about type coercion.

@martint could you please check if it makes sense to you? Other than that, I think it's ready for you to review.

[ebyhr]

Rebased on master (no change).

[martint]

This leads me to think that the SPI should take and return a SQL expression (in text form, like we do with row filters and column masks) to represent the default values. Let me ponder this.

[ebyhr]

How about translating to IR and building literals with Type.getObjectValue? It will simplify this method:

io.trino.sql.ir.Constant constant = (io.trino.sql.ir.Constant) ConnectorExpressionTranslator.translate(session, expression, plannerContext, Map.of());
Object value = constant.value();
if (value == null) {
    return new NullLiteral();
}
...

case TimestampWithTimeZoneType _ -> new GenericLiteral("TIMESTAMP", type.getObjectValue(null, constant.getValueAsBlock(), 0).toString());

[ebyhr]

Pushed another commit to show the above idea.

[martint]

Part of the issue with this approach is that it requires explicit hardcoded handling of all the types. It doesn't work with extended types (e.g., Geometry / Geography, and others)

[martint]

After thinking about this more, I've come to the conclusion that the default values should be described as SQL expressions, similar to what we do for column masks, row filters and views. Some reasons for this:

• ConnectorExpression is an IR concept, and its type system won't necessarily match the SQL type system over time (especially as we keep evolving and unifying the IR). This will make it harder to describe and recover the SQL-level syntax for operationrs such as DESCRIBE.
• Some of the constructs such as CURRENT_USER, CURRENT_TIMESTAMP, etc. have no correspondence in the IR -- they are desugared during initial planning. It would make it harder to represent them without introducing artificial functions that the connectors need to understand.

[ebyhr]

Faced some problems during string migration:

• CREAT TABLE and ADD COLUMN: How can we convert evaluated objects to string when passing to SPI? One approach seems casting it as varchar/char type and using the string value. I can't find a better way.
• SHOW CREATE TABLE: We need to convert string to Expression for ColumnDefinition in ShowQueriesRewrite class. Formatting the expression without hard-coded type handling looks still difficult. For instance, I suppose we want to only show true or false without BOOLEAN '...' on boolean type. Converting to string in AstBuilder resolves this, but it looks incorrect approach to me.

Pushed new commits with 2 TODO comments in ColumnDefaultOptions and AstBuilder to showcase the above problems.

[martint]

How can we convert evaluated objects to string when passing to SPI

There shouldn't be a need to convert evaluated objects. The expressions should be analyzed during analysis/planning of CREATE TABLE and ADD COLUMN, but then the original string should be passed to the connector for storage.

SHOW CREATE TABLE: We need to convert string to Expression for ColumnDefinition

It should do the same thing SHOW CREATE VIEW does -- parse the expression to create the corresponding AST node.

[ebyhr]

Thanks for your explanation. Addressed comments.

[ebyhr]

Rebased on master to resolve conflicts.

[chenjian2664]

A question not relates to this : I see that the value is assigned to ir.Expression here. Could you clarify the boundary between when we should use ir.Expression vs. Expression? I’m trying to understand the intended separation or convention

[chenjian2664]

And does it now actually already to be the Constant ?

[chenjian2664]

when the !element.isNullable(), what if added defaultValue always is NULL?