Fast fail when creating table of iceberg with multi key fields

[zuston]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.8.0 (latest release)

Please describe the bug 🐞

When creating an Iceberg table with multiple primary keys, insert or upsert operations fail with the error message:
“Key fields must have exactly one field for Iceberg format, but got …”.

This error is triggered by the validation logic in IcebergKeyEncoder. However, this constraint is only detected at runtime during insert/upsert. From a usability and correctness perspective, the same validation should be performed earlier

Solution

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[zuston]

cc @wuchong @luoyuxia Could you help check this? If this is OK, I'm happy to fix this

[luoyuxia]

@zuston Thanks. I think we can do the validation during create iceberg table.

[zuston]

@zuston Thanks. I think we can do the validation during create iceberg table.

Ok. let me take this.

[wuchong]

@luoyuxia why iceberg can't have multiple keys? Do you have plan/ideas to support this?

[luoyuxia]

@luoyuxia why iceberg can't have multiple keys? Do you have plan/ideas to support this?

It's in the future plan of the FIP
https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg

"Support multiple bucket keys. There is already a PR apache/iceberg#8259 in Apache Iceberg to support multiple bucket keys, but it has been closed since there has been no activity for a while. We can push the iceberg community to merge this PR, and then we can support multiple bucket keys. That's the best solution for supporting multiple bucket keys"

Since iceberg have already have the spec for multiple bucket transform, it maye not be much hard to push forward the implementation. cc @MehulBatra

[wuchong]

@luoyuxia thank you for the explanation. I think it's still very tricky for iceberg users to add such a bucket transformation. A better solution should be supporting deletion vector, and then we don't need the bucket alignment any more.

[luoyuxia]

@luoyuxia thank you for the explanation. I think it's still very tricky for iceberg users to add such a bucket transformation. A better solution should be supporting deletion vector, and then we don't need the bucket alignment any more.

Thank you for your suggestion. Yes, support deletion vector will make ervery thing simple and natural. I'll incease the priority for supporting deletion vector.