HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries

[ayushtkn]

What changes were proposed in this pull request?

For invalid entries get_variant() doesn't return null

Why are the changes needed?

To correct the behaviour of to_variant()

Does this PR introduce any user-facing change?

Yes, get_variant() throws instead of returning null

How was this patch tested?

UT

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[deniskuzZ]

wouldn't it better to do this in itest?

[deniskuzZ]

not sure about this, similar change in get_json_object implemented by https://issues.apache.org/jira/browse/HIVE-29277 returns NULL
cc @wecharyu

variant_get(parse_json('{"a": 1}'), '$[0]')

✔ Expected result

NULL (not an error) — because the path '$[0]' does not exist in the JSON structure.

This is consistent with typical JSON-path behavior:
• navigating an invalid path returns NULL

there is a difference between the path that is valid, but matches nothing and invalid syntax

[ayushtkn]

in our case: try_variant_get should return null, in case someone wants null he should use try_variant_get

    name = "try_variant_get",
    value = "_FUNC_(variant, path[, type]) - Extracts a sub-variant from variant according to path, and casts it to type. Returns null on error.",
    extended = """

[deniskuzZ]

in our case: try_variant_get should return null, in case someone wants null he should use try_variant_get

    name = "try_variant_get",
    value = "_FUNC_(variant, path[, type]) - Extracts a sub-variant from variant according to path, and casts it to type. Returns null on error.",
    extended = """

I imagined it the following way:

✅ variant_get

Throws an error if the extraction fails

Errors happen when:

• JSONPath is invalid (syntax error)

  variant_get(col, '$.a[') -- broken bracket
  variant_get(col, '$..a') -- unsupported recursive descent
  variant_get(col, '$[abc]') -- invalid array index
  variant_get(col, '$.a..b') -- illegal step

• The value cannot be converted to the requested type

  variant_get(parse_json('{"city": "Budapest"}'), '$.city', 'int') -- can't cast String to int

• The variant structure is malformed

  variant_get(parse_json('{"city": "Buda'), '$.city') -- malformed json

If the path is valid but does not match anything, Iceberg returns NULL (same as normal JSONPath behavior)

✔️ Safe for normal use
✖️ Not safe if the data may be malformed

✅ try_variant_get

Never throws an error

If anything goes wrong, it returns NULL

It swallows:

• Invalid JSONPath syntax

• Type conversion failures

• Unexpected data types

• Corrupted or partial JSON/VARIANT values

✔️ Safe for messy/unknown data
✔️ Useful for schema evolution
✔️ Good in SELECT queries where one bad row would otherwise fail the job

[ayushtkn]

I am confused. Isn't this what we fix here: variant_get was returning null instead of an error

[deniskuzZ]

I am confused. Isn't this what we fix here: variant_get was returning null instead of an error

variant_get(parse_json('{"a": 1}'), '$[0]') should return NULL, not error. I've added examples, please take a look

[ayushtkn]

ohhk, you mean to say the current behaviour is correct, we need not to changed anything?

[deniskuzZ]

ohhk, you mean to say the current behaviour is correct, we need not to changed anything?

it's probably correct for the empty path case, but not for the rest of scenarios (JSONPath is invalid, unsupported type cast, etc) where we return NULL.
Could we check the same in Spark or ask Spark folks to run these scenarios for us?