bug: Binary op between map and array failed

[comphead]

Describe the bug

Caused by: org.apache.comet.CometNativeException: Cannot evaluate binary expression because of type mismatch: left List(Field { name: "item", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }), right List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) .
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues

Subtask of #1043

Steps to reproduce

  test("native reader - nested type comparison") {
    withSQLConf(
      CometConf.COMET_EXEC_ENABLED.key -> "true",
      SQLConf.USE_V1_SOURCE_LIST.key -> "parquet",
      CometConf.COMET_ENABLED.key -> "true",
      CometConf.COMET_EXPLAIN_FALLBACK_ENABLED.key -> "true",
      CometConf.COMET_NATIVE_SCAN_IMPL.key -> "native_iceberg_compat") {
      testSingleLineQuery(
        """
          |select map('a', 1) m
          |""".stripMargin,
        "select map_keys(m) != array('a') from tbl")
    }
  }

Expected behavior

No response

Additional context

No response

[comphead]

The issue is in nullability flag, the binary comparison in DF is done by

        if left_data_type.is_nested() {
            if !left_data_type.equals_datatype(&right_data_type) {
                return internal_err!("Cannot evaluate binary expression because of type mismatch: left {}, right {} ", left_data_type, right_data_type);
            }
            return apply_cmp_for_nested(self.op, &lhs, &rhs);
        }

And equals_datatype checks the nulls flag

[comphead]

map_keys in comet returns nullability flag not the same as Spark

[comphead]

The issue is in the discrepancy how nullable fields calculated, The Comet always hardcode nullability flag as null for the ScalarFunction like map_keys(nullable == true), but DF when creates literal calculates the nullability based on value (nullability == false)

[comphead]

Will be included in DF 50 and depends on #2286

[comphead]

Closed via #2286