spec: Variant lower/upper bounds

[aihuaxu]

This is to update the bound spec for Variant type columns. We are collecting all the shredded bounds for a column in a file into a Variant object. E.g.

For a Variant schema like,

{
  "event_type": "login",
  "user.name": "Alex", 
  "tags": ["action", "drama"]
}

when the fields are shredded, the bound object looks like:

{
  "$['event_type']": "login",
  "$['user.name']": "Alex",
  "$['tags'][0]":"action"
}

and then it's serialized into binary. In order to support dot in the key name, we will use JSON normalized path as the subcolumn path.

[aihuaxu]

cc @rdblue, @danielcweeks, @RussellSpitzer

[rdblue]

I think that $.event.["user.name"] should not have a . after event in the format. I would also say that if we are asking for JSON path notation, then we should use the canonical "Normalized Path" form (Section 2.7), which is to always use ['name'] syntax (with single quotes, not double). I think that's probably a good idea.

[aihuaxu]

Make sense. That would reduce the variation. I also update to add the array representation to use always index 0 like $['tags'][0]. Let me know your thoughts.

[XBaith]

For variant type, the lower_bounds and upper_bounds store the minimum and maximum values for all shredded subcolumns within a file.

I'm not quite sure if we need to add the collection behavior of lower_bound and upper_bound under different conditions as mentioned in the proposal. What do you think?

[aihuaxu]

@XBaith What kind of collection behavior you are referring to? Can you clarify?

[XBaith]

Lower and upper bound statistics for subcolumns are collected for each data file based on the following conditions:
Uniform Value Types:
If all subcolumn values match the shredded type, lower/upper bounds are collected.
Example: For event.location.longitude, if all values are of the double type, the lower/upper bounds are written to the manifest file.
Mixed Value Types:
If the subcolumn contains multiple types (e.g., double and string), lower/upper bound statistics are not collected.
Example: For event.location.longitude, if the values include both double and string, lower/upper bounds are excluded.
Some subcolumn values are nulls or missing:
If some subcolumn values are null or missing in a file, but the available values match the shredded type, lower/upper bounds are still collected.
If all the subcolumn values are nulls, then lower/upper bounds are not collected. null_value_counts stat can be collected in later implementation to be used with value_counts to know they are all nulls.

[rdblue]

There shouldn't be a need to include array paths. Bounds for array data are not tracked.

[rdblue]

Variant is not something that we can reference here because that is a class in the implementation. I think what you want to say is that the serialized value is a variant metadata (v1) concatenated with a variant object. The object's fields are field paths in the normalized JSON path format and the values are the upper or lower bound corresponding to the shredded type in the data file.

[rdblue]

This should not state "all shredded subcolumns". First, "all" is not a requirement and is actually misleading because the fields are only those for which we can guarantee the lower and upper bounds are accurate. I would also not use the somewhat confusing term "subcolumn". In the variant spec we refer to fields. You may also want to call out a special case where the root is not an object. In that case the lower or upper bound is tracked by the field name "$" indicating the root.