Skip to content

Commit

Permalink
[Feat](Nereids) add join hints (#27819)
Browse files Browse the repository at this point in the history
* [Feat](Nereids) Add leading and ordered hint (#22057)

Add leading hint and ordered hint. Usage:
select /*+ ordered / * from a join b on xxx; which will limit join order to original order
select /+ leading ({b a}) */ from a join b on xxx; which will change join order to b join a.

* [Fix](Nereids) fix test leading change disable join reorder parameter (#23657)

Problem:
when running pipeline, we get randomly failed of test_leading
Reason:
physical distribute was generated and choosed to be the best plan because we can not get any statistic information of empty table. So we would get some unexpect result because we can not expect the order in memo
Solved:
Add statistic of columns used in test_leading, try repeatly in pipeline

* [Fix](Nereids) fix test leading suite and add tpch shape checking base on leading (#25842)

- fix test leading suite caused by sessionvariable setting error
- add tpch shape checking base on leading

* [Feat](Nereids) join hint support stage one (#27378)

support view as a independent unit of leading hint
add random test check of leading hint query
add more test with data of leading hint query
add random test check of distribute hint

* [Fix](Nereids) fix leading rebase problems
  • Loading branch information
LiBinfeng-01 authored Dec 29, 2023
1 parent 4b25611 commit 8eefd58
Show file tree
Hide file tree
Showing 76 changed files with 11,006 additions and 168 deletions.
622 changes: 622 additions & 0 deletions docs/en/docs/query-acceleration/hint/joinHint.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ grammar:

```sql
SELECT
[hint_statement, ...]
[ALL | DISTINCT | DISTINCTROW | ALL EXCEPT ( col_name1 [, col_name2, col_name3, ...] )]
select_expr [, select_expr ...]
[FROM table_references
Expand Down Expand Up @@ -86,6 +87,8 @@ SELECT
11. SELECT supports explicit partition selection using PARTITION containing a list of partitions or subpartitions (or both) following the name of the table in `table_reference`

12. `[TABLET tids] TABLESAMPLE n [ROWS | PERCENT] [REPEATABLE seek]`: Limit the number of rows read from the table in the FROM clause, select a number of Tablets pseudo-randomly from the table according to the specified number of rows or percentages, and specify the number of seeds in REPEATABLE to return the selected samples again. In addition, you can also manually specify the TableID, Note that this can only be used for OLAP tables.

13. `hint_statement`: hint in front of the selectlist indicates that hints can be used to influence the behavior of the optimizer in order to obtain the desired execution plan. Details refer to [joinHint using document] (https://doris.apache.org/en/docs/query-acceleration/hint/joinHint.md)

**Syntax constraints:**

Expand Down
607 changes: 607 additions & 0 deletions docs/zh-CN/docs/query-acceleration/hint/joinHint.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ SELECT

```sql
SELECT
[hint_statement, ...]
[ALL | DISTINCT | DISTINCTROW | ALL EXCEPT ( col_name1 [, col_name2, col_name3, ...] )]
select_expr [, select_expr ...]
[FROM table_references
Expand Down Expand Up @@ -87,6 +88,8 @@ SELECT

12. `[TABLET tids] TABLESAMPLE n [ROWS | PERCENT] [REPEATABLE seek]`: 在FROM子句中限制表的读取行数,根据指定的行数或百分比从表中伪随机的选择数个Tablet,REPEATABLE指定种子数可使选择的样本再次返回,此外也可手动指定TableID,注意这只能用于OLAP表。

13. `hint_statement`: 在selectlist前面使用hint表示可以通过hint去影响优化器的行为以期得到想要的执行计划,详情可参考[joinHint 使用文档](https://doris.apache.org/zh-CN/docs/query-acceleration/hint/joinHint.md)

**语法约束:**

1. SELECT也可用于检索计算的行而不引用任何表。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -312,8 +312,10 @@ LABEL: 'LABEL';
LARGEINT: 'LARGEINT';
LAST: 'LAST';
LATERAL: 'LATERAL';
LAZY: 'LAZY';
LDAP: 'LDAP';
LDAP_ADMIN_PASSWORD: 'LDAP_ADMIN_PASSWORD';
LEADING: 'LEADING';
LEFT: 'LEFT';
LESS: 'LESS';
LEVEL: 'LEVEL';
Expand Down Expand Up @@ -374,6 +376,7 @@ OPEN: 'OPEN';
OPTIMIZED: 'OPTIMIZED';
OR: 'OR';
ORDER: 'ORDER';
ORDERED: 'ORDERED';
OUTER: 'OUTER';
OUTFILE: 'OUTFILE';
OVER: 'OVER';
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ statement
identifierOrText
: errorCapturingIdentifier
| STRING_LITERAL
| LEADING_STRING
;

userIdentify
Expand Down Expand Up @@ -191,7 +192,7 @@ havingClause
selectHint: HINT_START hintStatements+=hintStatement (COMMA? hintStatements+=hintStatement)* HINT_END;

hintStatement
: hintName=identifier LEFT_PAREN parameters+=hintAssignment (COMMA parameters+=hintAssignment)* RIGHT_PAREN
: hintName=identifier (LEFT_PAREN parameters+=hintAssignment (COMMA? parameters+=hintAssignment)* RIGHT_PAREN)?
;

hintAssignment
Expand Down Expand Up @@ -683,6 +684,7 @@ nonReserved
| LAST
| LDAP
| LDAP_ADMIN_PASSWORD
| LEADING
| LEFT_BRACE
| LESS
| LEVEL
Expand Down Expand Up @@ -718,6 +720,7 @@ nonReserved
| ONLY
| OPEN
| OPTIMIZED
| ORDERED
| PARAMETER
| PARSED
| PASSWORD
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import org.apache.doris.nereids.analyzer.UnboundOneRowRelation;
import org.apache.doris.nereids.analyzer.UnboundRelation;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.hint.Hint;
import org.apache.doris.nereids.jobs.Job;
import org.apache.doris.nereids.jobs.JobContext;
import org.apache.doris.nereids.jobs.executor.Analyzer;
Expand Down Expand Up @@ -112,6 +113,10 @@ public class CascadesContext implements ScheduleContext {
private final Optional<CTEId> currentTree;
private final Optional<CascadesContext> parent;

private boolean isLeadingJoin = false;

private final Map<String, Hint> hintMap = Maps.newLinkedHashMap();

/**
* Constructor of OptimizerContext.
*
Expand Down Expand Up @@ -608,4 +613,16 @@ public void updateConsumerStats(CTEId cteId, Statistics statistics) {
p.value().setStatistics(updatedConsumerStats);
}
}

public boolean isLeadingJoin() {
return isLeadingJoin;
}

public void setLeadingJoin(boolean leadingJoin) {
isLeadingJoin = leadingJoin;
}

public Map<String, Hint> getHintMap() {
return hintMap;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
import org.apache.doris.nereids.glue.LogicalPlanAdapter;
import org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator;
import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
import org.apache.doris.nereids.hint.Hint;
import org.apache.doris.nereids.jobs.executor.Optimizer;
import org.apache.doris.nereids.jobs.executor.Rewriter;
import org.apache.doris.nereids.memo.Group;
Expand Down Expand Up @@ -405,36 +406,77 @@ private ScheduledExecutorService runTimeoutExecutor() {
return executor;
}

/**
* getting hints explain string, which specified by enumerate and show in lists
* @param hints hint map recorded in statement context
* @return explain string shows using of hint
*/
public String getHintExplainString(List<Hint> hints) {
String used = "";
String unUsed = "";
String syntaxError = "";
for (Hint hint : hints) {
switch (hint.getStatus()) {
case UNUSED:
unUsed = unUsed + " " + hint.getExplainString();
break;
case SYNTAX_ERROR:
syntaxError = syntaxError + " " + hint.getExplainString()
+ " Msg:" + hint.getErrorMessage();
break;
case SUCCESS:
used = used + " " + hint.getExplainString();
break;
default:
break;
}
}
return "\nHint log:" + "\nUsed:" + used + "\nUnUsed:" + unUsed + "\nSyntaxError:" + syntaxError;
}

@Override
public String getExplainString(ExplainOptions explainOptions) {
ExplainLevel explainLevel = getExplainLevel(explainOptions);
String plan = "";
switch (explainLevel) {
case PARSED_PLAN:
return parsedPlan.treeString();
plan = parsedPlan.treeString();
break;
case ANALYZED_PLAN:
return analyzedPlan.treeString();
plan = analyzedPlan.treeString();
break;
case REWRITTEN_PLAN:
return rewrittenPlan.treeString();
plan = rewrittenPlan.treeString();
break;
case OPTIMIZED_PLAN:
return "cost = " + cost + "\n" + optimizedPlan.treeString();
plan = "cost = " + cost + "\n" + optimizedPlan.treeString();
break;
case SHAPE_PLAN:
return optimizedPlan.shape("");
plan = optimizedPlan.shape("");
break;
case MEMO_PLAN:
return cascadesContext.getMemo().toString()
plan = cascadesContext.getMemo().toString()
+ "\n\n========== OPTIMIZED PLAN ==========\n"
+ optimizedPlan.treeString();
break;
case ALL_PLAN:
return "========== PARSED PLAN ==========\n"
plan = "========== PARSED PLAN ==========\n"
+ parsedPlan.treeString() + "\n\n"
+ "========== ANALYZED PLAN ==========\n"
+ analyzedPlan.treeString() + "\n\n"
+ "========== REWRITTEN PLAN ==========\n"
+ rewrittenPlan.treeString() + "\n\n"
+ "========== OPTIMIZED PLAN ==========\n"
+ optimizedPlan.treeString();
break;
default:
return super.getExplainString(explainOptions);
plan = super.getExplainString(explainOptions);
}
if (statementContext != null && !statementContext.getHints().isEmpty()) {
String hint = getHintExplainString(statementContext.getHints());
return plan + hint;
}
return plan;
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import org.apache.doris.analysis.StatementBase;
import org.apache.doris.common.IdGenerator;
import org.apache.doris.common.Pair;
import org.apache.doris.nereids.hint.Hint;
import org.apache.doris.nereids.memo.Group;
import org.apache.doris.nereids.rules.analysis.ColumnAliasGenerator;
import org.apache.doris.nereids.trees.expressions.CTEId;
Expand All @@ -41,6 +42,7 @@
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
Expand Down Expand Up @@ -86,6 +88,9 @@ public class StatementContext {
private final Map<CTEId, LogicalPlan> rewrittenCteConsumer = new HashMap<>();

private final Set<String> viewDdlSqlSet = Sets.newHashSet();
private final Map<String, Hint> hintMap = Maps.newLinkedHashMap();

private final List<Hint> hints = new ArrayList<>();

public StatementContext() {
this.connectContext = ConnectContext.get();
Expand Down Expand Up @@ -231,4 +236,12 @@ public void addViewDdlSql(String ddlSql) {
public List<String> getViewDdlSqls() {
return ImmutableList.copyOf(viewDdlSqlSet);
}

public void addHint(Hint hint) {
this.hints.add(hint);
}

public List<Hint> getHints() {
return ImmutableList.copyOf(hints);
}
}
85 changes: 85 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/hint/Hint.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

package org.apache.doris.nereids.hint;

import java.util.Objects;

/**
* select hint.
* e.g. set_var(query_timeout='1800', exec_mem_limit='2147483648')
*/
public class Hint {
// e.g. set_var
private String hintName;

private HintStatus status;

private String errorMessage = "";

/**
* hint status which need to show in explain when it is not used or have syntax error
*/
public enum HintStatus {
UNUSED,
SYNTAX_ERROR,
SUCCESS
}

public Hint(String hintName) {
this.hintName = Objects.requireNonNull(hintName, "hintName can not be null");
this.status = HintStatus.UNUSED;
}

public void setHintName(String hintName) {
this.hintName = hintName;
}

public void setStatus(HintStatus status) {
this.status = status;
}

public HintStatus getStatus() {
return status;
}

public boolean isSuccess() {
return getStatus() == HintStatus.SUCCESS;
}

public boolean isSyntaxError() {
return getStatus() == HintStatus.SYNTAX_ERROR;
}

public String getHintName() {
return hintName;
}

public String getErrorMessage() {
return errorMessage;
}

public void setErrorMessage(String errorMessage) {
this.errorMessage = errorMessage;
}

public String getExplainString() {
StringBuilder out = new StringBuilder();
out.append("\nHint:\n");
return out.toString();
}
}
Loading

0 comments on commit 8eefd58

Please sign in to comment.