Support selective nullable columns

[gortiz]

Pinot support for null support is configured at table level by setting tableIndexConfg.nullHandlingEnabled to true in TableConfig.

By setting this to true, all columns in the table are treated as nullable.
Given that nullable columns negatively impact in the heap and/or other optimizations, ideally, we should let users to individually decide the nullability of each column.

In order to do that, we have to:

• Change NullValueIndexType.createDeserializer to actually read indexes like other indexes does.
• Change places where tableIndexConfg.nullHandlingEnabled is read (directly or indirectly) in order to use getIndex(StandardIndexes.nullValueVector()).isEnabled(). This includes
  • SegmentColumnarIndexCreator
  • BaseDefaultColumnHandler.createDefaultValueColumnV1Indices

[Jackie-Jiang]

From the config perspective, instead of treating null value vector as an index similar to inverted index, I'd suggest putting it as part of the field spec (nullable). See #10428 for prior discussions

[gortiz]

I like the idea of defining it at field spec level. The configuration shouldn't be a problem. IndexType.getConfig receives the schema as an argument, so it should be easy to read the field spec there. I guess I'm biased given that I don't have enough knowledge in that area, but I think the challenge will be in actually modifying the engine behavior in order to understand that the nullability granularity is at column level instead of table level.

[Jackie-Jiang]

When null handling is enabled, the engine should pull the null value vector from the column, and if the vector doesn't exist or is empty, we handle it as non-null. Basically using the actual index as source of choose when processing the queries

[gortiz]

See #10653 and #10381

[yashmayya]

Closed by #11960.