[Bug]: stability-test: tpcc test communication link failure.

[Ariznawlll]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

3.0-dev

Commit ID

725b723

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job：https://github.com/matrixorigin/mo-nightly-regression/actions/workflows/stability-test-on-distributed.yaml

log：https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22UAt%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-stb-725b723-202604081652%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221775722733000%22,%22to%22:%221775729066000%22%7D%7D%7D&schemaVersion=1&orgId=1

2026-04-09 18:04:26 FATAL jTPCCTerminal:296 - [UNEXPECTED][TT_NEW_ORDER][CONNECTION] The connection[32579665] has not been valid, caused by: Communications link failure

2026-04-09 18:04:26 ERROR jTPCCConnection:327 - The connection[32579665] has not been valid.

2026-04-09 17:54:54 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][CONNECTION] The connection[32700669] has not been valid, caused by: Communications link failure

2026-04-09 17:54:54 ERROR jTPCCConnection:327 - The connection[32700669] has not been valid.

2026-04-09 17:28:49 FATAL jTPCCTerminal:296 - [UNEXPECTED][TT_NEW_ORDER][CONNECTION] The connection[32638269] has not been valid, caused by: Communications link failure

2026-04-09 17:28:49 ERROR jTPCCConnection:327 - The connection[32638269] has not been valid.

2026-04-09 17:25:36 FATAL jTPCCTerminal:214 - [UNEXPECTED][TT_NEW_ORDER][CONNECTION] The connection[32266475] has not been valid, caused by: Communications link failure

2026-04-09 17:25:36 ERROR jTPCCConnection:327 - The connection[32266475] has not been valid.

2026-04-09 17:22:28 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][CONNECTION] The connection[31343353] has not been valid, caused by: Communications link failure

2026-04-09 17:22:28 ERROR jTPCCConnection:327 - The connection[31343353] has not been valid.

2026-04-09 17:06:02 FATAL jTPCCTerminal:214 - [UNEXPECTED][TT_NEW_ORDER][CONNECTION] The connection[32117507] has not been valid, caused by: Communications link failure

2026-04-09 17:06:02 ERROR jTPCCConnection:327 - The connection[32117507] has not been valid.

2026-04-09 16:58:13 FATAL jTPCCTerminal:296 - [UNEXPECTED][TT_NEW_ORDER][CONNECTION] The connection[32063290] has not been valid, caused by: Communications link failure

2026-04-09 16:58:13 ERROR jTPCCConnection:327 - The connection[32063290] has not been valid.

2026-04-09 16:55:17 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][CONNECTION] The connection[31964909] has not been valid, caused by: Communications link failure

2026-04-09 16:55:17 ERROR jTPCCConnection:327 - The connection[31964909] has not been valid.

2026-04-09 16:50:15 FATAL jTPCCTerminal:296 - [UNEXPECTED][TT_NEW_ORDER][CONNECTION] The connection[31119739] has not been valid, caused by: Communications link failure

2026-04-09 16:50:15 ERROR jTPCCConnection:327 - The connection[31119739] has not been valid.

2026-04-09 16:18:53 FATAL jTPCCTerminal:214 - [UNEXPECTED][TT_NEW_ORDER][CONNECTION] The connection[31184221] has not been valid, caused by: Communications link failure

2026-04-09 16:18:53 ERROR jTPCCConnection:327 - The connection[31184221] has not been valid.

Expected Behavior

No response

Steps to Reproduce

trigger stability test

Additional information

No response

[Ariznawlll]

commit：90ae8dc0c07d99307db274ea6926455e5dfc3d5b
tpcc测试还是有些连不上的问题：

job：https://github.com/matrixorigin/mo-nightly-regression/actions/runs/24646841494

2026-04-22 05:46:48 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, none

2026-04-22 05:46:48 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, none

2026-04-22 05:46:48 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, none

2026-04-22 05:46:46 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, none

2026-04-22 05:46:46 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, none

2026-04-22 02:36:51 FATAL jTPCCTerminal:214 - [UNEXPECTED][TT_PAYMENT][EXECUTION] ErrorCode : 20701, ErrorMessage : deadlock detected

2026-04-22 02:36:50 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, none

2026-04-22 02:36:48 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, none

2026-04-22 02:36:47 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, none

2026-04-22 02:36:47 FATAL jTPCCTerminal:214 - [UNEXPECTED][TT_PAYMENT][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, none

2026-04-22 00:47:01 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, writeto tcp4 10.10.34.54:33784->10.10.187.121:6003: read tcp4 10.10.34.54:33784->10.10.187.121:6003: i/o timeout

2026-04-22 00:47:01 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, writeto tcp4 10.10.34.54:33784->10.10.187.121:6003: read tcp4 10.10.34.54:33784->10.10.187.121:6003: i/o timeout

2026-04-22 00:47:01 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, writeto tcp4 10.10.34.54:33784->10.10.187.121:6003: read tcp4 10.10.34.54:33784->10.10.187.121:6003: i/o timeout

2026-04-22 00:47:01 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, writeto tcp4 10.10.34.54:33784->10.10.187.121:6003: read tcp4 10.10.34.54:33784->10.10.187.121:6003: i/o timeout

2026-04-22 00:47:01 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, writeto tcp4 10.10.34.54:33784->10.10.187.121:6003: read tcp4 10.10.34.54:33784->10.10.187.121:6003: i/o timeout

2026-04-21 23:59:03 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, writeto tcp4 10.10.34.54:39574->10.10.187.121:6003: read tcp4 10.10.34.54:39574->10.10.187.121:6003: i/o timeout

2026-04-21 23:59:03 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, writeto tcp4 10.10.34.54:39574->10.10.187.121:6003: read tcp4 10.10.34.54:39574->10.10.187.121:6003: i/o timeout

2026-04-21 23:59:03 FATAL jTPCCTerminal:327 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20505, ErrorMessage : can not connect to remote backend, writeto tcp4 10.10.34.54:39574->10.10.187.121:6003: read tcp4 10.10.34.54:39574->10.10.187.121:6003: i/o timeout

2026-04-21 23:32:45 FATAL jTPCCTerminal:214 - [UNEXPECTED][TT_NEW_ORDER][CONNECTION] The connection[24014676] has not been valid, caused by: Communications link failure

2026-04-21 23:32:45 ERROR jTPCCConnection:327 - The connection[24014676] has not been valid.

log：https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22Y46%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bhost%3D%5C%2210-222-1-128%5C%22,filename%3D~%5C%22%2Fdata1%2Fstability-test%2F24646841494_%5Ba-f0-9%5D%2B%2Ftpcc-longrunning-test%2Fmo-tpcc%2Ftpcc.log%5C%22%7D%20%7C~%20%5C%22%28%3Fi%29%28Communications%20link%20failure%7Cfailed%7Cerror%7Cpanic%7CFATAL%7CLost%20connection%29%5C%22%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%221776655487000%22,%22to%22:%221776825419801%22%7D%7D%7D&schemaVersion=1&orgId=1

tpch报错：

2026-04-21 17:01:38 The query q20  has failed to  be executed,and cost: 48974

ERROR 20500 (HY000) at line 1: rpc timeout

tpch工具日志：https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22Y46%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bhost%3D%5C%2210-222-1-128%5C%22,filename%3D~%5C%22%2Fdata1%2Fstability-test%2F24646841494_%5Ba-f0-9%5D%2B%2Ftpch-longrunning-test%2Fmo-tpch%2Ftpch.log%5C%22%7D%20%7C~%20%5C%22%28%3Fi%29%28failed%7Cerror%7Crpc%20timeout%7Cwait%20notify%20message%20timeout%7Cpanic%7CFATAL%7CLost%20connection%29%5C%22%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%221776655487000%22,%22to%22:%221776826020328%22%7D%7D%7D&schemaVersion=1&orgId=1

集群日志：https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%224Vs%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-stb-90ae8dc-202604201123%5C%22%7D%20%7C%3D%20%60panic%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221776614400000%22,%22to%22:%221776697259000%22%7D%7D%7D&schemaVersion=1&orgId=1

[XuPeng-SH]

不好解决，长期调研