Input field name $f3 does not match with the projection expression ... #16063
Description
Describe the bug
Physical planning errors out with the message: Error: Internal("Input field name $f3 does not match with the projection expression Utf8(\"people\")") (line where error arises). This happens though the substrait consumer path, it generates a logical plan out of a substrait plan sucessfully but it fails after, while doing the physical plan. I've observed this occurs mostly while doing UNIONs, some substrait plans come with the alias $fN for some columns. For example, for the substrait reproducer I shared below we get the following Logical Plan:
Projection: Utf8("people") AS product_category, Utf8("people")__temp__0 AS product_type, product_key
Union
Projection: Utf8("people"), Utf8("people") AS Utf8("people")__temp__0, sales.product_key
Left Join: sales.product_key = food.@food_id
TableScan: sales
TableScan: food
Union
Projection: people.$f3, people.$f5, people.product_key0
Left Join: people.product_key0 = food.@food_id
TableScan: people
TableScan: food
TableScan: more_products
It fails while building the uppermost ProjectionExecUtf8("people") AS product_category, Utf8("people")__temp__0 AS product_type, product_key, debugging, I noticed that even though the LogicalPlan Union node has as a schema "Utf8(\"people\")" the UnionExec physcial node ends up having "$f3"
To Reproduce
The following substrait plan:
{
"extensionUris": [{
"extensionUriAnchor": 1,
"uri": "/functions_comparison.yaml"
}],
"extensions": [{
"extensionFunction": {
"extensionUriReference": 1,
"functionAnchor": 0,
"name": "equal:any_any"
}
}],
"relations": [{
"root": {
"input": {
"set": {
"common": {
"direct": {
}
},
"inputs": [{
"project": {
"common": {
"emit": {
"outputMapping": [2, 3, 4]
}
},
"input": {
"join": {
"common": {
"direct": {
}
},
"left": {
"read": {
"common": {
"direct": {
}
},
"baseSchema": {
"names": ["product_key"],
"struct": {
"types": [{
"string": {
"nullability": "NULLABILITY_NULLABLE"
}
}],
"nullability": "NULLABILITY_REQUIRED"
}
},
"namedTable": {
"names": [
"sales"
]
}
}
},
"right": {
"read": {
"common": {
"direct": {
}
},
"baseSchema": {
"names": ["@food_id"],
"struct": {
"types": [{
"string": {
"nullability": "NULLABILITY_NULLABLE"
}
}],
"nullability": "NULLABILITY_REQUIRED"
}
},
"namedTable": {
"names": [
"food"
]
}
}
},
"expression": {
"scalarFunction": {
"functionReference": 0,
"outputType": {
"bool": {
"nullability": "NULLABILITY_NULLABLE"
}
},
"arguments": [{
"value": {
"selection": {
"directReference": {
"structField": {
"field": 0
}
},
"rootReference": {
}
}
}
}, {
"value": {
"selection": {
"directReference": {
"structField": {
"field": 1
}
},
"rootReference": {
}
}
}
}]
}
},
"type": "JOIN_TYPE_LEFT"
}
},
"expressions": [{
"literal": {
"string": "people"
}
}, {
"literal": {
"string": "people"
}
}, {
"selection": {
"directReference": {
"structField": {
"field": 0
}
},
"rootReference": {
}
}
}]
}
}, {
"set": {
"common": {
"direct": {
}
},
"inputs": [{
"project": {
"common": {
"emit": {
"outputMapping": [4, 5, 6]
}
},
"input": {
"join": {
"common": {
"direct": {
}
},
"left": {
"read": {
"common": {
"direct": {
}
},
"baseSchema": {
"names": ["$f3", "$f5", "product_key0"],
"struct": {
"types": [{
"string": {
"nullability": "NULLABILITY_REQUIRED"
}
}, {
"string": {
"nullability": "NULLABILITY_REQUIRED"
}
}, {
"string": {
"nullability": "NULLABILITY_NULLABLE"
}
}],
"nullability": "NULLABILITY_REQUIRED"
}
},
"namedTable": {
"names": [
"people"
]
}
}
},
"right": {
"read": {
"common": {
"direct": {
}
},
"baseSchema": {
"names": ["@food_id"],
"struct": {
"types": [{
"string": {
"nullability": "NULLABILITY_NULLABLE"
}
}],
"nullability": "NULLABILITY_REQUIRED"
}
},
"namedTable": {
"names": [
"food"
]
}
}
},
"expression": {
"scalarFunction": {
"functionReference": 0,
"outputType": {
"bool": {
"nullability": "NULLABILITY_NULLABLE"
}
},
"arguments": [{
"value": {
"selection": {
"directReference": {
"structField": {
"field": 2
}
},
"rootReference": {
}
}
}
}, {
"value": {
"selection": {
"directReference": {
"structField": {
"field": 3
}
},
"rootReference": {
}
}
}
}]
}
},
"type": "JOIN_TYPE_LEFT"
}
},
"expressions": [{
"selection": {
"directReference": {
"structField": {
"field": 0
}
},
"rootReference": {
}
}
}, {
"selection": {
"directReference": {
"structField": {
"field": 1
}
},
"rootReference": {
}
}
}, {
"selection": {
"directReference": {
"structField": {
"field": 2
}
},
"rootReference": {
}
}
}]
}
}, {
"read": {
"common": {
"direct": {
}
},
"baseSchema": {
"names": ["$f1000", "$f2000", "more_products_key0000"],
"struct": {
"types": [{
"string": {
"nullability": "NULLABILITY_REQUIRED"
}
}, {
"string": {
"nullability": "NULLABILITY_REQUIRED"
}
}, {
"string": {
"nullability": "NULLABILITY_NULLABLE"
}
}],
"nullability": "NULLABILITY_REQUIRED"
}
},
"namedTable": {
"names": [
"more_products"
]
}
}
}],
"op": "SET_OP_UNION_ALL"
}
}],
"op": "SET_OP_UNION_ALL"
}
},
"names": ["product_category", "product_type", "product_key"]
}
}]
}
Expected behavior
Build physical plan sucessfully
Additional context
Using version 47.0