TiDB crash for the error of cannot find mpp task

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

SELECT * from table_10_utf8mb4collate_utf8mb4_bin_undef as t2 where t2. `col_char(20)_key_signed` = ALL ( SELECT `col_int_key_unsigned` from table_1_latin1_undef as t1 where not t1. `pk` <> t2. `col_int_key_signed`);
/*
+-------------------------------------------+---------+-------------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                                        | estRows | task              | access object | operator info                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+-------------------------------------------+---------+-------------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection_11                             | 10.00   | root              |               | aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.pk, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_int_key_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_int_key_unsigned, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_int_undef_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_int_undef_unsigned, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_datetime_key_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_datetime_undef_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_date_key_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_date_undef_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_timestamp_key_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_timestamp_undef_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_float_key_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_float_key_unsigned, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_float_undef_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_float_undef_unsigned, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_double_key_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_double_key_unsigned, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_double_undef_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_double_undef_unsigned, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_char(20)_key_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_char(20)_undef_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_varchar(20)_key_signed, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_varchar(20)_undef_signed |
| └─Apply_13                                | 10.00   | root              |               | CARTESIAN inner join, other cond:or(and(le(Column#48, 1), and(eq(cast(aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_char(20)_key_signed, double BINARY), cast(Column#47, double UNSIGNED BINARY)), if(ne(Column#49, 0), NULL, 1))), or(eq(Column#50, 0), if(isnull(aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_char(20)_key_signed), NULL, 0)))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
|   ├─TableReader_15(Build)                 | 10.00   | root              |               | data:TableFullScan_14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|   │ └─TableFullScan_14                    | 10.00   | cop[tikv]         | table:t2      | keep order:false, stats:pseudo                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|   └─TableReader_31(Probe)                 | 1.00    | root              |               | data:ExchangeSender_30                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
|     └─ExchangeSender_30                   | 1.00    | batchCop[tiflash] |               | ExchangeType: PassThrough                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|       └─Projection_26                     | 1.00    | batchCop[tiflash] |               | Column#47, Column#48, Column#49, Column#50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|         └─HashAgg_27                      | 1.00    | batchCop[tiflash] |               | funcs:firstrow(Column#53)->Column#47, funcs:count(distinct aggregation2.table_1_latin1_undef.col_int_key_unsigned)->Column#48, funcs:sum(Column#54)->Column#49, funcs:sum(Column#55)->Column#50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|           └─ExchangeReceiver_29           | 1.00    | batchCop[tiflash] |               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|             └─ExchangeSender_28           | 1.00    | batchCop[tiflash] |               | ExchangeType: PassThrough                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|               └─HashAgg_19                | 1.00    | batchCop[tiflash] |               | group by:Column#58, funcs:firstrow(Column#56)->Column#53, funcs:sum(Column#57)->Column#54, funcs:count(1)->Column#55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
|                 └─Projection_32           | 1.00    | batchCop[tiflash] |               | aggregation2.table_1_latin1_undef.col_int_key_unsigned, cast(isnull(aggregation2.table_1_latin1_undef.col_int_key_unsigned), decimal(20,0) BINARY)->Column#57, aggregation2.table_1_latin1_undef.col_int_key_unsigned                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|                   └─TableRangeScan_25     | 1.00    | batchCop[tiflash] | table:t1      | range: decided by [eq(aggregation2.table_1_latin1_undef.pk, aggregation2.table_10_utf8mb4collate_utf8mb4_bin_undef.col_int_key_signed)], keep order:false, stats:pseudo                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+-------------------------------------------+---------+-------------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
*/

2. What did you expect to see? (Required)

successfully

3. What did you see instead (Required)

ERROR 1105 (HY000): cannot find mpp task
tidb crash

[2022/01/13 10:30:21.103 +08:00] [INFO] [fragment.go:75] ["Mpp will generate tasks"] [plan=Recv()->HashAgg->Projection->Send()]
[2022/01/13 10:30:21.103 +08:00] [INFO] [batch_coprocessor.go:319] ["detecting available mpp stores"]
[2022/01/13 10:30:21.107 +08:00] [INFO] [conn.go:1115] ["command dispatched failed"] [conn=9395] [connInfo="id:9395, addr:10.233.101.128:12324 status:10, collation:utf8mb4_0900_ai_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="SELECT * from table_10_utf8mb4collate_utf8mb4_bin_undef as t2 where t2. `col_char(20)_key_signed` = ALL ( SELECT `col_int_key_unsigned` from table_1_latin1_undef as t1 where not t1. `pk` <> t2. `col_int_key_signed`)"] [txn_mode=PESSIMISTIC] [err="cannot find mpp task\ngithub.com/pingcap/tidb/planner/core.(*mppTaskGenerator).generateMPPTasksForFragment\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/fragment.go:264\ngithub.com/pingcap/tidb/planner/core.(*mppTaskGenerator).generateMPPTasksForExchangeSender\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/fragment.go:230\ngithub.com/pingcap/tidb/planner/core.(*mppTaskGenerator).generateMPPTasksForFragment\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/fragment.go:243\ngithub.com/pingcap/tidb/planner/core.(*mppTaskGenerator).generateMPPTasksForExchangeSender\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/fragment.go:230\ngithub.com/pingcap/tidb/planner/core.(*mppTaskGenerator).generateMPPTasks\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/fragment.go:80\ngithub.com/pingcap/tidb/planner/core.GenerateRootMPPTasks\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/fragment.go:71\ngithub.com/pingcap/tidb/executor.(*MPPGather).Open\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/mpp_gather.go:106\ngithub.com/pingcap/tidb/executor.(*NestedLoopApplyExec).fetchAllInners\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:965\ngithub.com/pingcap/tidb/executor.(*NestedLoopApplyExec).Next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:1032\ngithub.com/pingcap/tidb/executor.Next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:286\ngithub.com/pingcap/tidb/executor.(*ProjectionExec).unParallelExecute\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/projection.go:194\ngithub.com/pingcap/tidb/executor.(*ProjectionExec).Next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/projection.go:180\ngithub.com/pingcap/tidb/executor.Next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:286\ngithub.com/pingcap/tidb/executor.(*recordSet).Next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:149\ngithub.com/pingcap/tidb/server.(*tidbResultSet).Next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/driver_tidb.go:306\ngithub.com/pingcap/tidb/server.(*clientConn).writeChunks\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2165\ngithub.com/pingcap/tidb/server.(*clientConn).writeResultset\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2116\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1994\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1841\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1336\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1091\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]

4. What is your TiDB version? (Required)

Release Version: v5.4.0
Edition: Community
Git Commit Hash: 65e167a
Git Branch: heads/refs/tags/v5.4.0
UTC Build Time: 2022-01-12 10:36:52
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

[LittleFall]

minimal reproduce:

create table t1(a int(11) primary key CLUSTERED);
alter table t1 set tiflash replica 1;
create table t2(a int(11), b char(20)) ;
alter table t2 set tiflash replica 1;
insert into t2 value (null, null);
explain SELECT * from t2 where t2.a = ALL ( SELECT b from t1 where t1.a = t2.a);

reproduce in 5.3.0 & 5.4.0

reproduce in 5.2.0, but tidb won't crashed.

mysql>  SELECT * from t2 where t2.a = ALL ( SELECT b from t1 where t1.a = t2.a);
ERROR 1105 (HY000): cannot find mpp task

in 5.1-, planner will only read t1 from tikv, and when I enforce it read from tiflash, it will report:

explain SELECT * from t2 where t2.a = ALL ( SELECT /*+ read_from_storage(tiflash[t1]) */ b from t1 where t1.a = t2.a);
ERROR 1815 (HY000): Internal : Can't find a proper physical plan for this query

[LittleFall]

Since very far versions, this statement fails to generate mpp tasks. But the reason for the crash is that when the task balancer is introduced, it is assumed that there is at least one task, causing the array to go out of bounds.

balancer code: https://github.com/pingcap/tidb/pull/26130/files#diff-dec6808c7659579faf2c9c6f9d8b7a7e0d7095b61b9a07cfd2cbc9546a3c7c86R161

[LittleFall]

another original reproduce:

set @@tidb_enforce_mpp=on;
sELECT /*+ hash_agg() */ count( distinct t2. `pk` ) from table_10_utf8mb4collate_utf8mb4_bin_undef as t2 where t2. `col_varchar(20)_key_signed` > ( SELECT `col_date_key_signed` from table_1000_binary_undef as t1 where not t1. `pk` <> t2. `col_int_undef_signed` order by pk asc limit 1 );

[github-actions]

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.