-
Notifications
You must be signed in to change notification settings - Fork 1.5k
[multistage] bridge v2 query engine for leaf stage v1 multi-value column #11117
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
b4b1440
1a07cee
cb7ae8b
fe22439
b7f6c82
4b72515
d3ed2c7
82d2a56
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -90,6 +90,11 @@ public enum TransformFunctionType { | |
| // date type conversion functions | ||
| CAST("cast"), | ||
|
|
||
| // object type | ||
| ARRAY_TO_MV("arrayToMV", | ||
| ReturnTypes.cascade(opBinding -> positionalComponentReturnType(opBinding, 0), SqlTypeTransforms.FORCE_NULLABLE), | ||
|
||
| OperandTypes.family(SqlTypeFamily.ARRAY), "array_to_mv"), | ||
|
|
||
| // string functions | ||
| JSONEXTRACTSCALAR("jsonExtractScalar", | ||
| ReturnTypes.cascade(opBinding -> positionalReturnTypeInferenceFromStringLiteral(opBinding, 2, | ||
|
|
@@ -280,6 +285,13 @@ private static RelDataType positionalReturnTypeInferenceFromStringLiteral(SqlOpe | |
| return opBinding.getTypeFactory().createSqlType(defaultSqlType); | ||
| } | ||
|
|
||
| private static RelDataType positionalComponentReturnType(SqlOperatorBinding opBinding, int pos) { | ||
| if (opBinding.getOperandCount() > pos) { | ||
| return opBinding.getOperandType(pos).getComponentType(); | ||
| } | ||
| throw new IllegalArgumentException("Invalid number of arguments for function " + opBinding.getOperator().getName()); | ||
| } | ||
|
|
||
| private static RelDataType dateTimeConverterReturnTypeInference(SqlOperatorBinding opBinding) { | ||
| int outputFormatPos = 2; | ||
| if (opBinding.getOperandCount() > outputFormatPos | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -31,6 +31,7 @@ | |
| import org.apache.pinot.spi.config.table.TableConfig; | ||
| import org.apache.pinot.spi.data.Schema; | ||
| import org.apache.pinot.util.TestUtils; | ||
| import org.testng.Assert; | ||
| import org.testng.annotations.AfterClass; | ||
| import org.testng.annotations.BeforeClass; | ||
| import org.testng.annotations.Test; | ||
|
|
@@ -95,17 +96,17 @@ protected boolean useMultiStageQueryEngine() { | |
|
|
||
| @Test | ||
| @Override | ||
| public void testHardcodedQueriesMultiStage() | ||
| public void testHardcodedQueries() | ||
| throws Exception { | ||
| super.testHardcodedQueriesMultiStage(); | ||
| super.testHardcodedQueries(); | ||
| } | ||
|
|
||
| @Test | ||
| @Override | ||
| public void testGeneratedQueries() | ||
| throws Exception { | ||
| // test multistage engine, currently we don't support MV columns. | ||
| super.testGeneratedQueries(false, true); | ||
| super.testGeneratedQueries(true, true); | ||
| } | ||
|
|
||
| @Test | ||
|
|
@@ -485,6 +486,25 @@ public void testLiteralOnlyFunc() | |
| assertEquals(results.get(10).asText(), "hello!"); | ||
| } | ||
|
|
||
| @Test | ||
| public void testMultiValueColumnGroupBy() | ||
| throws Exception { | ||
| String pinotQuery = "SELECT count(*), arrayToMV(RandomAirports) FROM mytable " | ||
xiangfu0 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| + "GROUP BY arrayToMV(RandomAirports)"; | ||
| JsonNode jsonNode = postQuery(pinotQuery); | ||
| Assert.assertEquals(jsonNode.get("resultTable").get("rows").size(), 154); | ||
| } | ||
|
|
||
| @Test | ||
| public void testMultiValueColumnGroupByOrderBy() | ||
| throws Exception { | ||
| String pinotQuery = "SELECT count(*), arrayToMV(RandomAirports) FROM mytable " | ||
|
||
| + "GROUP BY arrayToMV(RandomAirports) " | ||
| + "ORDER BY arrayToMV(RandomAirports) DESC"; | ||
| JsonNode jsonNode = postQuery(pinotQuery); | ||
| Assert.assertEquals(jsonNode.get("resultTable").get("rows").size(), 154); | ||
| } | ||
|
|
||
| @AfterClass | ||
| public void tearDown() | ||
| throws Exception { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO this is confusing name. the data type is already MV running an ARRAY_TO_MV is a bit weird.
should we named it
USE_AS_MV? and we can say that MV columns are by defaultUSE_AS_ARRAYThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
USE_AS_MVmight be confusing, since we repurposedMVasARRAYin v2,ARRAY_TO_MVmight be more explicit.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah we need to figure out a proper way b/c out put of a select is an array, but the table config / schema will still call this as MV column as convension. both have some confusion, but as long as the document is proper we should be good.