Tracking issue for new expression framework

[andylokandy]

This is a tracking issue for the design and implementation of the new scalar expression framework.

Summary

The new expression framework contains these improvements:

1. Compile-time (for SQL, aka, when planning) type checking, which could capture all type errors in the beginning, and provide nice-looking error reports.
2. Untyped expression evaluation, which means function evaluation will no longer worry about data typing.
3. Auto vectorization, generics, auto downcasting, and more ergonomic improvements for writing SQL function.
4. Distributed evaluation in mind on the first day.

Reference

RFC: Formal Type System
Demo: Typed Type Exercise in Rust

Migration plan

The new expression framework will replace the legacy common-datavalues, common-datablocks and refactor all functions in common-function. So it's a tough task, and we are going to break it into small steps:

1. Add a new common-expression crate with expression definition (AST), type checking, and evaluation runtime.
2. Run benchmark on the new framework and make necessary improvements.
3. Migrate all functions to the new framework.

Tasks

features

• feat(expr): add new crate common-expression #6576
• feat(expr): Implement domain calculation #6649
• feat(expr): implement adaptive constant folding #7054
• feat(expr): support conversion between arrow #6674
• feat(expr): support user-defined CAST and TRY_CAST #6663
• feat(expr): implement error report #6661
• feat(expr): allow function to return runtime error #6662
• feat(expr): implement pretty print for Chunk #6597
• feat(expr): add serializable expression #6712
• feat(chunk): make chunk support scalar values #6918
• feat(expr): support 3 args function #7075
• Filter, take, shuffle on Column and Chunk #7020
• Support UInt32, UInt64, Int32, Int64, Float32, Float64 in new expression framework #6636
• Support Timestamp, Date, Interval in new expression framework #6635
• Support JSON in new expression framework #6634
• feat(function-v2): support constructing array and CAST(... AS VARIANT) #7781
• [ ] Conversion between Boolean and other types no plan to support
• [ ] Conversion between String and other types no plan to support
• Literal deserialization

refactor

• fix(expr): fix domain calculation for nullable type #6677
• refactor(expr): make auto-nullable and auto-vectorization independent #6787
• chore(expr): auto generate domain for test cases #6756
• refactor(expr): move some methods from ArgType to ValueType #6856
• refactor(expr): add NullableColumn and NullableColumnBuilder #6867
• refactor(expr): use Scalar to store constant in Expr #6923
• Enrich doc comment

migration

We can generate new test files via using env REGENERATE_GOLDENFILES=1 cargo test and git diff to show differs

• Migrate Boolean functions to new expression framework #6763
• Migrate String functions to new expression framework #6766
• Migrate math functions to new expression framework  #7255
• Migrate control-flow functions to new expression framework #6833
• Migrate comparison functions to new expression framework #7636
• Supportrand()
• Migrate arithmetics functions to new expression framework #7091
• Migrate aggregation functions to function-v2 #7740
• Migrate date & timestamp functions to new expression framework  #8048

[Xuanwo]

Make Databend Type System Great Again!