Upgrade DataFusion 48.0.0

[xudong963]

Related to apache/datafusion#15771

[xudong963]

DF48 made the config true by default, so it'll break the test_rewrite test:

TestCase {
    name: "range filter + equality predicate",
    base:
        "SELECT column1, column2 FROM t1 WHERE column1 = column3 AND column1 >= '2022'",
    query:
    // Since column1 = column3 in the original view,
    // we are allowed to substitute column1 for column3 and vice versa.
        "SELECT column2, column3 FROM t1 WHERE column1 = column3 AND column3 >= '2023'",
},

->

+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type    | plan                                                                                                                                                  |
+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| logical_plan | Projection: t1.column1, t1.column2                                                                                                                    |
|              |   Filter: t1.column1 = CAST(t1.column3 AS Utf8View) AND t1.column1 >= Utf8View("2022")                                                                |
|              |     TableScan: t1 projection=[column1, column2, column3], partial_filters=[t1.column1 = CAST(t1.column3 AS Utf8View), t1.column1 >= Utf8View("2022")] |
+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+

Currently, the insert_binary_expr method in Predicate can't infer eq class from t1.column1 = CAST(t1.column3 AS Utf8View).

t1 is from here:

CREATE EXTERNAL TABLE t1 (
    column1 VARCHAR, 
    column2 BIGINT, 
    column3 CHAR
)

So I'm wondering if CHAR also should be mapped to utf8view.

[xudong963]

Opened an issue in upstream: apache/datafusion#16277

[suremarc]

That's unfortunate, it may be possible to generalize the equivalence class logic to handle certain types of casts (particularly invertible ones). It would probably be somewhat complex to implement though.

[xudong963]

Yes, we can add this to our backlist.

[xudong963]

DF48 will generate the following plan:

+--------------+---------------------------------------------------------------------------------------------------------------------------------+
| plan_type    | plan                                                                                                                            |
+--------------+---------------------------------------------------------------------------------------------------------------------------------+
| logical_plan | Projection: t1.column1, t1.column2                                                                                              |
|              |   Filter: t1.column3 = t1.column1 AND t1.column1 >= Utf8("2022")                                                                |
|              |     TableScan: t1 projection=[column1, column2, column3], partial_filters=[t1.column3 = t1.column1, t1.column1 >= Utf8("2022")] |
+--------------+---------------------------------------------------------------------------------------------------------------------------------+

That is, t1.column3 = t1.column1 will be processed twice, and the second time they have the same index -> 0. If we don't add the fast return, it'll make column2's index 0, which will break the test (I guess we didn't consider the i==j case before)

[suremarc]

LGTM, thanks for your hard work as always

[suremarc]

That's unfortunate, it may be possible to generalize the equivalence class logic to handle certain types of casts (particularly invertible ones). It would probably be somewhat complex to implement though.