Support serializing & deserializing UDTs when pushing down scripts

[yuancu]

Description

Before this PR, pushed-down scripts does not work when there are arguments with user-defined types (UDTs). For example, the following simple script does not work if pushdown is enabled: source=bank | eval t = unix_timestamp(birthdate) | stats count() by t

Error response

{
  "error": {
    "reason": "Error occurred in OpenSearch engine: all shards failed",
    "details": "Shard[0]: org.opensearch.sql.exception.ExpressionEvaluationException: invalid to get doubleValue from value of type STRING\n\nFor more details, please send request for Json format to see the raw response from OpenSearch engine.",
    "type": "SearchPhaseExecutionException"
  },
  "status": 500
}

The problem arises from the serialization process. The serialization of UDTs is processed by Calcite's RelJson serializer. When serializing a UDT, it only keeps its SqlTypeName for future restoration. However, as many of UDTs are mapped to SqlTypeName.VARCHAR, all UDTs are restored as a VARCHAR type instead of the original UDT.

This PR fixes the issue by implementing custom logic when serializing & de-serializing UDTs.

Related Issues

Resolves #4063 , resolves #4322, and resolves #4340

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[yuancu]

Another issue arises after supporting serializing & pushing down UDTs: testQ12 and testQ21 in CalcitePPLTpchIT fail with reason: NoSuchMethodException[org.apache.calcite.runtime.SqlFunctions.lt(java.lang.Object,java.lang.Object)]

Before this PR, the UDT are pushed down as strings, so it went for SqlFunctions.lt(java.lang.String,java.lang.String)` when resolving date comparison. Now, the they have became UDTs, it can not resolve UDT date comparison with proper implementation.

For example, Q12 query contains timestamp comparison:

source = orders
| join ON o_orderkey = l_orderkey lineitem
| where l_commitdate < l_receiptdate
    and l_shipdate < l_commitdate
    and l_shipmode in ('MAIL', 'SHIP')
    and l_receiptdate >= date('1994-01-01')
    and l_receiptdate < date_add(date('1994-01-01'), interval 1 year)
| stats sum(case(o_orderpriority = '1-URGENT' or o_orderpriority = '2-HIGH', 1 else 0)) as high_line_count,
        sum(case(o_orderpriority != '1-URGENT' and o_orderpriority != '2-HIGH', 1 else 0)) as low_line_count
        by l_shipmode
| fields l_shipmode, high_line_count, low_line_count
| sort l_shipmode

Before this PR, the comparison script is like below:

rowTypet: {
  "fields": [
    {
      "type": "VARCHAR",
      "nullable": true,
      "precision": -1,
      "name": "l_receiptdate"
    },
    {
      "type": "VARCHAR",
      "nullable": true,
      "precision": -1,
      "name": "l_shipmode"
    },
    {
      "type": "VARCHAR",
      "nullable": true,
      "precision": -1,
      "name": "l_commitdate"
    },
...
  ],
  "nullable": false

  "op": {
    "name": "<",
    "kind": "LESS_THAN",
    "syntax": "BINARY"
  },
...
}

With this PR, the type becomes UDT:

rowTypet:{
  "fields": [
    {
      "udt": "EXPR_TIMESTAMP",
      "type": "VARCHAR",
      "nullable": true,
      "precision": -1,
      "name": "l_receiptdate"
    },
    {
      "type": "VARCHAR",
      "nullable": true,
      "precision": -1,
      "name": "l_shipmode"
    },
    {
      "udt": "EXPR_TIMESTAMP",
      "type": "VARCHAR",
      "nullable": true,
      "precision": -1,
      "name": "l_commitdate"
    },
    ...
    }
  ],
  "nullable": false
},
  "op": {
    "name": "<",
    "kind": "LESS_THAN",
    "syntax": "BINARY"
  },
...
  ]
}

The problem arises from where l_commitdate < l_receiptdate and l_shipdate < l_commitdate. It has problem resolving comparing UDT. In calcite without push-down enabled, timestamp UDT comparison is resolved to string comparison since the underlying type of EXPR_TIMESTAMP is a RexLiteral storing a string constant (see RexToLibTranslator.java#L1250). I don't know why it does not work when UDT is pushed down.

Working on fixing it.

[songkant-aws]

Is the pushed-down generated code the same as that before pushdown? I notice we probably use the Calcite default BinaryImplementor to generate the comparison code. The primitive it compares depends on the reflected type. That's a possible root cause because we're not sure if the reflected UDT is the same as before.

[yuancu]

The data type differs, the rest, including the comparison code, remains the same.

For example, for this query: source = lineitem | where l_commitdate < l_receiptdate

Plan without this PR

{
  "calcite": {
    "logical": """LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])
  LogicalProject(l_receiptdate=[$0], l_returnflag=[$1], l_tax=[$2], l_shipmode=[$3], l_suppkey=[$4], l_shipdate=[$5], l_commitdate=[$6], l_partkey=[$7], l_orderkey=[$8], l_quantity=[$9], l_comment=[$10], l_linestatus=[$11], l_extendedprice=[$12], l_linenumber=[$13], l_discount=[$14], l_shipinstruct=[$15])
    LogicalFilter(condition=[<($6, $0)])
      CalciteLogicalIndexScan(table=[[OpenSearch, lineitem]])
""",
    "physical": """CalciteEnumerableIndexScan(table=[[OpenSearch, lineitem]], PushDownContext=[[PROJECT->[l_receiptdate, l_returnflag, l_tax, l_shipmode, l_suppkey, l_shipdate, l_commitdate, l_partkey, l_orderkey, l_quantity, l_comment, l_linestatus, l_extendedprice, l_linenumber, l_discount, l_shipinstruct], SCRIPT-><($6, $0), LIMIT->10000], OpenSearchRequestBuilder(sourceBuilder={"from":0,"size":10000,"timeout":"1m","query":{"script":{"script":{"source":"{\"langType\":\"calcite\",\"script\":\"rO0ABXNyABFqYXZhLnV0aWwuQ29sbFNlcleOq7Y6G6gRAwABSQADdGFneHAAAAADdwQAAAAGdAAHcm93VHlwZXQGdnsKICAiZmllbGRzIjogWwogICAgewogICAgICAidHlwZSI6ICJWQVJDSEFSIiwKICAgICAgIm51bGxhYmxlIjogdHJ1ZSwKICAgICAgInByZWNpc2lvbiI6IC0xLAogICAgICAibmFtZSI6ICJsX3JlY2VpcHRkYXRlIgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiVkFSQ0hBUiIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJwcmVjaXNpb24iOiAtMSwKICAgICAgIm5hbWUiOiAibF9yZXR1cm5mbGFnIgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiRE9VQkxFIiwKICAgICAgIm51bGxhYmxlIjogdHJ1ZSwKICAgICAgIm5hbWUiOiAibF90YXgiCiAgICB9LAogICAgewogICAgICAidHlwZSI6ICJWQVJDSEFSIiwKICAgICAgIm51bGxhYmxlIjogdHJ1ZSwKICAgICAgInByZWNpc2lvbiI6IC0xLAogICAgICAibmFtZSI6ICJsX3NoaXBtb2RlIgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiQklHSU5UIiwKICAgICAgIm51bGxhYmxlIjogdHJ1ZSwKICAgICAgIm5hbWUiOiAibF9zdXBwa2V5IgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiVkFSQ0hBUiIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJwcmVjaXNpb24iOiAtMSwKICAgICAgIm5hbWUiOiAibF9zaGlwZGF0ZSIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogIlZBUkNIQVIiLAogICAgICAibnVsbGFibGUiOiB0cnVlLAogICAgICAicHJlY2lzaW9uIjogLTEsCiAgICAgICJuYW1lIjogImxfY29tbWl0ZGF0ZSIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogIkJJR0lOVCIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJuYW1lIjogImxfcGFydGtleSIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogIkJJR0lOVCIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJuYW1lIjogImxfb3JkZXJrZXkiCiAgICB9LAogICAgewogICAgICAidHlwZSI6ICJET1VCTEUiLAogICAgICAibnVsbGFibGUiOiB0cnVlLAogICAgICAibmFtZSI6ICJsX3F1YW50aXR5IgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiVkFSQ0hBUiIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJwcmVjaXNpb24iOiAtMSwKICAgICAgIm5hbWUiOiAibF9jb21tZW50IgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiVkFSQ0hBUiIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJwcmVjaXNpb24iOiAtMSwKICAgICAgIm5hbWUiOiAibF9saW5lc3RhdHVzIgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiRE9VQkxFIiwKICAgICAgIm51bGxhYmxlIjogdHJ1ZSwKICAgICAgIm5hbWUiOiAibF9leHRlbmRlZHByaWNlIgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiSU5URUdFUiIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJuYW1lIjogImxfbGluZW51bWJlciIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogIkRPVUJMRSIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJuYW1lIjogImxfZGlzY291bnQiCiAgICB9LAogICAgewogICAgICAidHlwZSI6ICJWQVJDSEFSIiwKICAgICAgIm51bGxhYmxlIjogdHJ1ZSwKICAgICAgInByZWNpc2lvbiI6IC0xLAogICAgICAibmFtZSI6ICJsX3NoaXBpbnN0cnVjdCIKICAgIH0KICBdLAogICJudWxsYWJsZSI6IGZhbHNlCn10AARleHBydADKewogICJvcCI6IHsKICAgICJuYW1lIjogIjwiLAogICAgImtpbmQiOiAiTEVTU19USEFOIiwKICAgICJzeW50YXgiOiAiQklOQVJZIgogIH0sCiAgIm9wZXJhbmRzIjogWwogICAgewogICAgICAiaW5wdXQiOiA2LAogICAgICAibmFtZSI6ICIkNiIKICAgIH0sCiAgICB7CiAgICAgICJpbnB1dCI6IDAsCiAgICAgICJuYW1lIjogIiQwIgogICAgfQogIF0KfXQACmZpZWxkVHlwZXNzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAYdwgAAAAgAAAAEHQADWxfcmVjZWlwdGRhdGVzcgA6b3JnLm9wZW5zZWFyY2guc3FsLm9wZW5zZWFyY2guZGF0YS50eXBlLk9wZW5TZWFyY2hEYXRlVHlwZZ4tUq4QfcqvAgABTAAHZm9ybWF0c3QAEExqYXZhL3V0aWwvTGlzdDt4cgA6b3JnLm9wZW5zZWFyY2guc3FsLm9wZW5zZWFyY2guZGF0YS50eXBlLk9wZW5TZWFyY2hEYXRhVHlwZcJjvMoC+gU1AgADTAAMZXhwckNvcmVUeXBldAArTG9yZy9vcGVuc2VhcmNoL3NxbC9kYXRhL3R5cGUvRXhwckNvcmVUeXBlO0wAC21hcHBpbmdUeXBldABITG9yZy9vcGVuc2VhcmNoL3NxbC9vcGVuc2VhcmNoL2RhdGEvdHlwZS9PcGVuU2VhcmNoRGF0YVR5cGUkTWFwcGluZ1R5cGU7TAAKcHJvcGVydGllc3QAD0xqYXZhL3V0aWwvTWFwO3hwfnIAKW9yZy5vcGVuc2VhcmNoLnNxbC5kYXRhLnR5cGUuRXhwckNvcmVUeXBlAAAAAAAAAAASAAB4cgAOamF2YS5sYW5nLkVudW0AAAAAAAAAABIAAHhwdAAJVElNRVNUQU1QfnIARm9yZy5vcGVuc2VhcmNoLnNxbC5vcGVuc2VhcmNoLmRhdGEudHlwZS5PcGVuU2VhcmNoRGF0YVR5cGUkTWFwcGluZ1R5cGUAAAAAAAAAABIAAHhxAH4AEnQABERhdGVzcgA8c2hhZGVkLmNvbS5nb29nbGUuY29tbW9uLmNvbGxlY3QuSW1tdXRhYmxlTWFwJFNlcmlhbGl6ZWRGb3JtAAAAAAAAAAACAAJMAARrZXlzdAASTGphdmEvbGFuZy9PYmplY3Q7TAAGdmFsdWVzcQB+ABl4cHVyABNbTGphdmEubGFuZy5PYmplY3Q7kM5YnxBzKWwCAAB4cAAAAAB1cQB+ABsAAAAAc3EAfgAAAAAAAXcEAAAAAHh0AAxsX3JldHVybmZsYWd+cQB+ABF0AAZTVFJJTkd0AAVsX3RheH5xAH4AEXQABkRPVUJMRXQACmxfc2hpcG1vZGVxAH4AIHQACWxfc3VwcGtleX5xAH4AEXQABExPTkd0AApsX3NoaXBkYXRlc3EAfgAKcQB+ABNxAH4AFnEAfgAacQB+AB50AAxsX2NvbW1pdGRhdGVzcQB+AApxAH4AE3EAfgAWcQB+ABpxAH4AHnQACWxfcGFydGtleXEAfgAndAAKbF9vcmRlcmtleXEAfgAndAAKbF9xdWFudGl0eXEAfgAjdAAJbF9jb21tZW50c3IAOm9yZy5vcGVuc2VhcmNoLnNxbC5vcGVuc2VhcmNoLmRhdGEudHlwZS5PcGVuU2VhcmNoVGV4dFR5cGWtg6OTBOMxRAIAAUwABmZpZWxkc3EAfgAPeHEAfgAMfnEAfgARdAAHVU5LTk9XTn5xAH4AFXQABFRleHRxAH4AGnNxAH4AAAAAAAN3BAAAAAB4dAAMbF9saW5lc3RhdHVzcQB+ACB0AA9sX2V4dGVuZGVkcHJpY2VxAH4AI3QADGxfbGluZW51bWJlcn5xAH4AEXQAB0lOVEVHRVJ0AApsX2Rpc2NvdW50cQB+ACN0AA5sX3NoaXBpbnN0cnVjdHEAfgAgeHg=\"}","lang":"opensearch_compounded_script","params":{"utcTimestamp":1758005112179638000}},"boost":1.0}},"_source":{"includes":["l_receiptdate","l_returnflag","l_tax","l_shipmode","l_suppkey","l_shipdate","l_commitdate","l_partkey","l_orderkey","l_quantity","l_comment","l_linestatus","l_extendedprice","l_linenumber","l_discount","l_shipinstruct"],"excludes":[]},"sort":[{"_doc":{"order":"asc"}}]}, requestedTotalSize=10000, pageSize=null, startFrom=0)])
"""
  }
}

Plan with this PR

{
  "calcite": {
    "logical": """LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])
  LogicalProject(l_receiptdate=[$0], l_returnflag=[$1], l_tax=[$2], l_shipmode=[$3], l_suppkey=[$4], l_shipdate=[$5], l_commitdate=[$6], l_partkey=[$7], l_orderkey=[$8], l_quantity=[$9], l_comment=[$10], l_linestatus=[$11], l_extendedprice=[$12], l_linenumber=[$13], l_discount=[$14], l_shipinstruct=[$15])
    LogicalFilter(condition=[<($6, $0)])
      CalciteLogicalIndexScan(table=[[OpenSearch, lineitem]])
""",
    "physical": """CalciteEnumerableIndexScan(table=[[OpenSearch, lineitem]], PushDownContext=[[PROJECT->[l_receiptdate, l_returnflag, l_tax, l_shipmode, l_suppkey, l_shipdate, l_commitdate, l_partkey, l_orderkey, l_quantity, l_comment, l_linestatus, l_extendedprice, l_linenumber, l_discount, l_shipinstruct], SCRIPT-><($6, $0), LIMIT->10000], OpenSearchRequestBuilder(sourceBuilder={"from":0,"size":10000,"timeout":"1m","query":{"script":{"script":{"source":"{\"langType\":\"calcite\",\"script\":\"rO0ABXNyABFqYXZhLnV0aWwuQ29sbFNlcleOq7Y6G6gRAwABSQADdGFneHAAAAADdwQAAAAGdAAHcm93VHlwZXQG03sKICAiZmllbGRzIjogWwogICAgewogICAgICAidWR0IjogIkVYUFJfVElNRVNUQU1QIiwKICAgICAgInR5cGUiOiAiVkFSQ0hBUiIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJwcmVjaXNpb24iOiAtMSwKICAgICAgIm5hbWUiOiAibF9yZWNlaXB0ZGF0ZSIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogIlZBUkNIQVIiLAogICAgICAibnVsbGFibGUiOiB0cnVlLAogICAgICAicHJlY2lzaW9uIjogLTEsCiAgICAgICJuYW1lIjogImxfcmV0dXJuZmxhZyIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogIkRPVUJMRSIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJuYW1lIjogImxfdGF4IgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiVkFSQ0hBUiIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJwcmVjaXNpb24iOiAtMSwKICAgICAgIm5hbWUiOiAibF9zaGlwbW9kZSIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogIkJJR0lOVCIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJuYW1lIjogImxfc3VwcGtleSIKICAgIH0sCiAgICB7CiAgICAgICJ1ZHQiOiAiRVhQUl9USU1FU1RBTVAiLAogICAgICAidHlwZSI6ICJWQVJDSEFSIiwKICAgICAgIm51bGxhYmxlIjogdHJ1ZSwKICAgICAgInByZWNpc2lvbiI6IC0xLAogICAgICAibmFtZSI6ICJsX3NoaXBkYXRlIgogICAgfSwKICAgIHsKICAgICAgInVkdCI6ICJFWFBSX1RJTUVTVEFNUCIsCiAgICAgICJ0eXBlIjogIlZBUkNIQVIiLAogICAgICAibnVsbGFibGUiOiB0cnVlLAogICAgICAicHJlY2lzaW9uIjogLTEsCiAgICAgICJuYW1lIjogImxfY29tbWl0ZGF0ZSIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogIkJJR0lOVCIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJuYW1lIjogImxfcGFydGtleSIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogIkJJR0lOVCIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJuYW1lIjogImxfb3JkZXJrZXkiCiAgICB9LAogICAgewogICAgICAidHlwZSI6ICJET1VCTEUiLAogICAgICAibnVsbGFibGUiOiB0cnVlLAogICAgICAibmFtZSI6ICJsX3F1YW50aXR5IgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiVkFSQ0hBUiIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJwcmVjaXNpb24iOiAtMSwKICAgICAgIm5hbWUiOiAibF9jb21tZW50IgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiVkFSQ0hBUiIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJwcmVjaXNpb24iOiAtMSwKICAgICAgIm5hbWUiOiAibF9saW5lc3RhdHVzIgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiRE9VQkxFIiwKICAgICAgIm51bGxhYmxlIjogdHJ1ZSwKICAgICAgIm5hbWUiOiAibF9leHRlbmRlZHByaWNlIgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiSU5URUdFUiIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJuYW1lIjogImxfbGluZW51bWJlciIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogIkRPVUJMRSIsCiAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICJuYW1lIjogImxfZGlzY291bnQiCiAgICB9LAogICAgewogICAgICAidHlwZSI6ICJWQVJDSEFSIiwKICAgICAgIm51bGxhYmxlIjogdHJ1ZSwKICAgICAgInByZWNpc2lvbiI6IC0xLAogICAgICAibmFtZSI6ICJsX3NoaXBpbnN0cnVjdCIKICAgIH0KICBdLAogICJudWxsYWJsZSI6IGZhbHNlCn10AARleHBydADKewogICJvcCI6IHsKICAgICJuYW1lIjogIjwiLAogICAgImtpbmQiOiAiTEVTU19USEFOIiwKICAgICJzeW50YXgiOiAiQklOQVJZIgogIH0sCiAgIm9wZXJhbmRzIjogWwogICAgewogICAgICAiaW5wdXQiOiA2LAogICAgICAibmFtZSI6ICIkNiIKICAgIH0sCiAgICB7CiAgICAgICJpbnB1dCI6IDAsCiAgICAgICJuYW1lIjogIiQwIgogICAgfQogIF0KfXQACmZpZWxkVHlwZXNzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAYdwgAAAAgAAAAEHQADWxfcmVjZWlwdGRhdGVzcgA6b3JnLm9wZW5zZWFyY2guc3FsLm9wZW5zZWFyY2guZGF0YS50eXBlLk9wZW5TZWFyY2hEYXRlVHlwZZ4tUq4QfcqvAgABTAAHZm9ybWF0c3QAEExqYXZhL3V0aWwvTGlzdDt4cgA6b3JnLm9wZW5zZWFyY2guc3FsLm9wZW5zZWFyY2guZGF0YS50eXBlLk9wZW5TZWFyY2hEYXRhVHlwZcJjvMoC+gU1AgADTAAMZXhwckNvcmVUeXBldAArTG9yZy9vcGVuc2VhcmNoL3NxbC9kYXRhL3R5cGUvRXhwckNvcmVUeXBlO0wAC21hcHBpbmdUeXBldABITG9yZy9vcGVuc2VhcmNoL3NxbC9vcGVuc2VhcmNoL2RhdGEvdHlwZS9PcGVuU2VhcmNoRGF0YVR5cGUkTWFwcGluZ1R5cGU7TAAKcHJvcGVydGllc3QAD0xqYXZhL3V0aWwvTWFwO3hwfnIAKW9yZy5vcGVuc2VhcmNoLnNxbC5kYXRhLnR5cGUuRXhwckNvcmVUeXBlAAAAAAAAAAASAAB4cgAOamF2YS5sYW5nLkVudW0AAAAAAAAAABIAAHhwdAAJVElNRVNUQU1QfnIARm9yZy5vcGVuc2VhcmNoLnNxbC5vcGVuc2VhcmNoLmRhdGEudHlwZS5PcGVuU2VhcmNoRGF0YVR5cGUkTWFwcGluZ1R5cGUAAAAAAAAAABIAAHhxAH4AEnQABERhdGVzcgA8c2hhZGVkLmNvbS5nb29nbGUuY29tbW9uLmNvbGxlY3QuSW1tdXRhYmxlTWFwJFNlcmlhbGl6ZWRGb3JtAAAAAAAAAAACAAJMAARrZXlzdAASTGphdmEvbGFuZy9PYmplY3Q7TAAGdmFsdWVzcQB+ABl4cHVyABNbTGphdmEubGFuZy5PYmplY3Q7kM5YnxBzKWwCAAB4cAAAAAB1cQB+ABsAAAAAc3EAfgAAAAAAAXcEAAAAAHh0AAxsX3JldHVybmZsYWd+cQB+ABF0AAZTVFJJTkd0AAVsX3RheH5xAH4AEXQABkRPVUJMRXQACmxfc2hpcG1vZGVxAH4AIHQACWxfc3VwcGtleX5xAH4AEXQABExPTkd0AApsX3NoaXBkYXRlc3EAfgAKcQB+ABNxAH4AFnEAfgAacQB+AB50AAxsX2NvbW1pdGRhdGVzcQB+AApxAH4AE3EAfgAWcQB+ABpxAH4AHnQACWxfcGFydGtleXEAfgAndAAKbF9vcmRlcmtleXEAfgAndAAKbF9xdWFudGl0eXEAfgAjdAAJbF9jb21tZW50c3IAOm9yZy5vcGVuc2VhcmNoLnNxbC5vcGVuc2VhcmNoLmRhdGEudHlwZS5PcGVuU2VhcmNoVGV4dFR5cGWtg6OTBOMxRAIAAUwABmZpZWxkc3EAfgAPeHEAfgAMfnEAfgARdAAHVU5LTk9XTn5xAH4AFXQABFRleHRxAH4AGnNxAH4AAAAAAAN3BAAAAAB4dAAMbF9saW5lc3RhdHVzcQB+ACB0AA9sX2V4dGVuZGVkcHJpY2VxAH4AI3QADGxfbGluZW51bWJlcn5xAH4AEXQAB0lOVEVHRVJ0AApsX2Rpc2NvdW50cQB+ACN0AA5sX3NoaXBpbnN0cnVjdHEAfgAgeHg=\"}","lang":"opensearch_compounded_script","params":{"utcTimestamp":1758005410734926000}},"boost":1.0}},"_source":{"includes":["l_receiptdate","l_returnflag","l_tax","l_shipmode","l_suppkey","l_shipdate","l_commitdate","l_partkey","l_orderkey","l_quantity","l_comment","l_linestatus","l_extendedprice","l_linenumber","l_discount","l_shipinstruct"],"excludes":[]},"sort":[{"_doc":{"order":"asc"}}]}, requestedTotalSize=10000, pageSize=null, startFrom=0)])
"""
  }
}

The above comment shows part of the decoded script, their main difference lies in operand types.

I guess part of the reason that the UDT comparison is not supported is that we are not using PPLFuncImpTable when resolving functions in the script?

# RelJsonSerializer.java#L54
private static final SqlOperatorTable pplSqlOperatorTable =
    SqlOperatorTables.chain(
        PPLBuiltinOperators.instance(),
        SqlStdOperatorTable.instance(),
        // Add a list of necessary SqlLibrary if needed
        SqlLibraryOperatorTableFactory.INSTANCE.getOperatorTable(
            SqlLibrary.MYSQL, SqlLibrary.BIG_QUERY, SqlLibrary.SPARK, SqlLibrary.POSTGRESQL));

Btw, we did not implement any special comparison operator for date time UDTs like that for IP. But it has been working well somehow with BinaryImplementor:

// RexImpTable.java#L3181
final Type type0 = argValueList.get(0).getType();
final Type type1 = argValueList.get(1).getType();
final SqlBinaryOperator op = (SqlBinaryOperator) call.getOperator();
final RelDataType relDataType0 = call.getOperands().get(0).getType();

type0 and type1 are resolved to VARCHAR for date time UDTs here.

[songkant-aws]

I pulled the code and checked the example ppl. I think I figured out the root cause. The obvious error happens when Calcite code generation cannot find the less than function signature by provided Linq4j Expressions.
See this error:

RuntimeException[while resolving method 'lt[class java.lang.Object, class java.lang.Object]' in class class org.apache.calcite.runtime.SqlFunctions]

Calcite lt(.., ...) method doesn't have such signature while resolving our UDT. See Calcite code: https://github.com/apache/calcite/blob/main/core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java#L2183-L2248

When we hardcode the UDT type as STRING, it apparently generates such code for runtime evaluation. We can see that it's STRING for sure.

public Object[] apply(Object root0) {
  final String input_value = ((org.apache.calcite.DataContext) root0).get("l_commitdate") == null ? null : ((org.apache.calcite.DataContext) root0).get("l_commitdate").toString();
  final String input_value0 = ((org.apache.calcite.DataContext) root0).get("l_receiptdate") == null ? null : ((org.apache.calcite.DataContext) root0).get("l_receiptdate").toString();
  return new Object[] {
      input_value == null || input_value0 == null ? null : Boolean.valueOf(org.apache.calcite.runtime.SqlFunctions.lt(input_value, input_value0))};
}

Then what caused this wrong method resolution if we set the field as ExprSqlType.Timestamp UDT? From the previous normal generated code, we can see it assigns the input_value and input_value0 to the type String. In our case, it has to be assigned to Object. It means we resolved the wrong Java class for UDT. This reminds me that during compilation of script code, we might used incorrect type factory for input because we just simply copied Calcite code.

This line causes the issue: https://github.com/opensearch-project/sql/blob/main/opensearch/src/main/java/org/opensearch/sql/opensearch/storage/script/CalciteScriptEngine.java#L131-L132. It uses default JavaTypeFactory instead of our own OpenSearchTypeFactory that tunes UDT logic. Our OpenSearchTypeFactory overrides the getJavaClass method to resolve UDT's Java class: https://github.com/opensearch-project/sql/blob/main/core/src/main/java/org/opensearch/sql/calcite/utils/OpenSearchTypeFactory.java#L323-L329

So I think a simple fix is to remove the problematic line and pass OpenSearchTypeFactory.TYPE_FACTORY to our
ScriptInputGetter. And we need to make sure no impact of other cases.

[yuancu]

Thank you for help debugging!

[songkant-aws]

[Minor and Non-blocking] It's better to add some cases of serializing and deserializing nested structure like List[String, UDT_TIMESTAMP, UDT_IP] or Map{(key1, Integer), (key2, UDT_TIMESTAMP) }

[yuancu]

Thanks for the suggestion! Tests added

[LantaoJin]

can you add the IT:

source=%s | eval t = date_add(birthdate, interval 1 day) | stats count() by span(t, 1d)

[yuancu]

Added

[LantaoJin]

ditto

[yuancu]

Added