[FLINK-37923][sql] Introduce VARIANT type and PARSE_JSON to Flink SQL

[Sxnan]

What is the purpose of the change

This pull request introduces the Variant data structure to represent semi-structured data, a new SQL type variant, and a builtin method to parse json string to the variant type.

Brief change log

• Patch calcite to support variant type
• Introduce Variant and BinaryVariant
• Introduce variant type and PARSE_JSON to Flink SQL

Verifying this change

This change added tests and can be verified.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): yes
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? JavaDocs

[flinkbot]

CI report:

• 9513d8e Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[twalthr]

Awesome work @Sxnan! I left a couple of comments that should improve consistency when integrating this type into the existing type system.

[Sxnan]

@twalthr Thanks for the detailed review! I updated the PR accordingly, and all the comments should be addressed. Please take another look.

[twalthr]

Thanks @Sxnan. I added it for my list for tomorrow. I'm sure it can still make it before the feature freeze.

[Sxnan]

Hi @twalthr, could you take another look at your earliest convenience, in case we need to make some final adjustment to the PR before the feature freeze tomorrow.

[twalthr]

Thank you @Sxnan. I think the PR should definitely make it to 2.1 release. I added my last set of comments.

[twalthr]

I could imagine that users will request it to more easily pass it to constructors of ProcessFunction or having it in member variables of ProcessFunction as well. This is also the reason why Row is Serializable. But this can be a followup. Given all the utilities we have, ensuring serializability should not be too difficult to implement with a custom readObject/writeObject.

[Sxnan]

Hi @twalthr, thanks for the review! I updated the PR accordingly in the last two fixup commits. Please take another look.

[twalthr]

Thanks for this great feature @Sxnan!

[Sxnan]

Fixup commits are squashed. Will merge after the test passes.

[gustavodemorais]

Very nice feature, @Sxnan 🙂 I somehow didn't spot the docs. Are they covered in another PR?

[Sxnan]

@gustavodemorais Yes, it will be covered later in another PR.