planner: fix correlated aggregates which should be evaluated in outer query (#21431)

[ti-srebot]

cherry-pick #21431 to release-4.0

---

What problem does this PR solve?

Issue Number: close #18350, close #17748

Problem Summary:

• Some correlated aggregate (belonging to outer query) should be evaluated in outer query instead of in subquery.

# These aggregates should be evaluated in outer query
select (select count(a)) from t;
select (select count(n.a) from t limit 1) from t n;
select (select 1 from t where count(n.a) > 1 limit 1) from t n;
select (select 1 from t order by count(n.a) limit 1) from t n;
select (select 1 from t having count(n.a) > 1 limit 1) from t n;
select (select cnt from (select count(a) as cnt) n) from t;

# These aggregates should be evaluated in sub-query
select (select count(a + n.a) from t) from t n;
select (select count(a) from t) from t n;

• Cannot handle nested aggregates.

select (select sum(count(a))) from t; # report "invalid group function use"

What is changed and how it works?

What's Changed:

• Add a procedure resolveCorrelatedAggregates before buildAggregation to collect correlated aggregates from sub-queries.
• Add cache for ResultSetNode to avoid rebuilding plans.
• Add correlatedAggMap for sub-queries to build correct plans.

How it Works:

1. For example, we have

create table t (a int, b int);

2. Start from query:

select (select count(n.a) from t limit 1) from t n;

3. Inside b.resolveCorrelatedAggregates, count(n.a) will be recognized as correlated aggregate since column n.a is from outer schema (from table n instead of t). So count(n.a) is collected and returned to outer buildSelect, now the query is like:

select (select count(n.a) from t limit 1), count(n.a) as `sel_subq_agg_1` from t n;

4. Inside b.buildAggregation, a correlated column for sel_subq_agg_1 will be created and added to b.correlatedAggMap. Then b.buildProjection will build plan for sub-query (select count(n.a) from t) limit 1.

5. The sub-query will recognize count(n.a) as correlated aggregate and skip it the process of building aggregate plan. Then inside b.buildProjection of the sub-query, count(n.a) will be rewritten as a correlated column to sel_subq_agg_1, now the query is like:

select (select CorrelatedColumn{`sel_subq_agg_1`} from t limit 1), count(n.a) as `sel_subq_agg_1` from t n;

6. Finally we get the correct query plan.

Related changes

• Need to cherry-pick to the release branch 4.0

Check List

Tests

• Unit test
• Integration test

Side effects

• Performance regression
  • Consumes more CPU
  • Consumes more MEM

Release note

• Fix correlated aggregates which should be evaluated in outer query instead of in subqueries.
• Support nested aggregate in subqueries (correlated aggregates inside normal aggregate).

[ti-srebot]

/run-all-tests

[ti-srebot]

@dyzsr you're already a collaborator in bot's repo.

[dyzsr]

/run-all-tests tidb-test=pr/1147

[dyzsr]

/run-all-tests tidb-test=pr/1147

[dyzsr]

/run-integration-copr-test tidb-test=pr/1147

[qw4990]

/run-all-tests

[qw4990]

/run-all-tests

[dyzsr]

/run-common-test tidb-test=pr/1147
/run-integration-common-test tidb-test=pr/1147

[dyzsr]

/run-integration-common-test tidb-test=pr/1147

[qw4990]

Rest LGTM

[dyzsr]

/run-all-tests

[qw4990]

/run-all-tests

[qw4990]

/run-mybatis-test

[qw4990]

LGTM