Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

data inconsistency after redo apply #8591

Closed
fubinzh opened this issue Mar 20, 2023 · 3 comments · Fixed by #8620
Closed

data inconsistency after redo apply #8591

fubinzh opened this issue Mar 20, 2023 · 3 comments · Fixed by #8620
Assignees
@CharlesCheung96
Labels
affects-6.5 affects-7.0 area/ticdc Issues or PRs related to TiCDC. found/automation Bugs found by automation cases severity/critical type/bug The issue is confirmed as a bug.

Comments

@fubinzh
Copy link

fubinzh commented Mar 20, 2023

What did you do?

  1. Run case redo_apply_basic_real_time_sync

What did you expect to see?

  1. data should be consistent after redo apply

What did you see instead?

  1. data inconsistency is seen
2023-03-19T00:31:48.790Z	INFO	host/ticdc.go:283	Execute command on ticdc	{"command": "/cdc  cli  changefeed  pause \"--server=127.0.0.1:8301\" \"--changefeed-id=redo-apply-basic-real-time-sync\"", "timeout": "10s", "ticdc peer": 0}
2023-03-19T00:31:48.790Z	INFO	k8s/client.go:132	it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
[2023/03/19 00:31:49.690 +00:00] [INFO] [cdc.go:81] ["run changeFeed task finished"] [result=null] []
[2023/03/19 00:31:49.690 +00:00] [INFO] [step.go:30] ["changeFeedTask: flag=&{}"]
2023-03-19T00:31:49.690Z	INFO	host/ticdc.go:299	Execute command on ticdc	{"command": "/cdc version", "ticdc peer": 0}
2023-03-19T00:31:49.691Z	INFO	k8s/client.go:132	it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
[2023/03/19 00:31:50.159 +00:00] [INFO] [ticdc.go:199] ["get TiCDC version"] [cdcVersion=v6.7.0-alpha]
2023-03-19T00:31:50.159Z	INFO	host/ticdc.go:283	Execute command on ticdc	{"command": "/cdc  cli  changefeed  list \"--server=127.0.0.1:8301\"", "timeout": "10s", "ticdc peer": 0}
2023-03-19T00:31:50.159Z	INFO	k8s/client.go:132	it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
2023-03-19T00:31:50.629Z	INFO	k8s/client.go:132	it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
[2023/03/19 00:31:50.629 +00:00] [INFO] [cdc.go:81] ["run changeFeed task finished"] [result="[{\"id\":\"redo-apply-basic-real-time-sync\",\"info\":null,\"status\":null,\"count\":0,\"task-status\":null,\"summary\":{\"state\":\"stopped\",\"tso\":440188383660867591,\"checkpoint\":\"2023-03-19 00:23:43.511\",\"error\":null}}]"] []
[2023/03/19 00:31:50.629 +00:00] [INFO] [step.go:30] ["redo: flag=&{Storage:s3://tmp/test-infra-redolog/redo-apply-basic-real-time-sync-bd90c86c-2038-4064-b811-cbbb3dcfda80?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http://172.16.6.5:9000&force-path-style=true TmpDir:redo-apply-basic-real-time-sync}"]
2023-03-19T00:31:50.629Z	INFO	host/ticdc.go:283	Execute command on ticdc	{"command": "/cdc  redo  apply \"--sink-uri=mysql://root:@downstream.cdc-testbed-tps-1658689-1-113:3306\" \"--storage=s3://tmp/test-infra-redolog/redo-apply-basic-real-time-sync-bd90c86c-2038-4064-b811-cbbb3dcfda80?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http://172.16.6.5:9000&force-path-style=true\" \"--tmp-dir=/tmp/nfs/redo-apply-basic-real-time-sync\"", "timeout": "2h0m0s", "ticdc peer": 0}
[2023/03/19 01:29:11.286 +00:00] [INFO] [redo.go:29] ["run redo cmd finished"] [result="[2023/03/19 00:31:50.949 +00:00] [INFO] [helper.go:54] [\"init log\"] [file=] [level=info]\n[2023/03/19 00:31:50.952 +00:00] [INFO] [s3.go:392] [\"succeed to get bucket region from s3\"] [\"bucket region\"=us-east-1]\n[2023/03/19 00:31:51.070 +00:00] [INFO] [mem_quota.go:82] [\"New memory quota\"] [namespace=default] [changefeed=redo-applier] [total=1073741824]\n[2023/03/19 00:31:51.070 +00:00] [INFO] [redo.go:160] [\"apply redo log starts\"] [checkpointTs=440188383660867591] [resolvedTs=440188384840515588]\n[2023/03/19 00:31:51.071 +00:00] [INFO] [sink.go:313] [\"succeed to parse parameter from sink uri\"] [protocol=] [txnAtomicity=]\n[2023/03/19 00:31:51.073 +00:00] [INFO] [s3.go:392] [\"succeed to get bucket region from s3\"] [\"bucket region\"=us-east-1]\n[2023/03/19 00:31:51.074 +00:00] [INFO] [s3.go:392] [\"succeed to get bucket region from s3\"] [\"bucket region\"=us-east-1]\n[2023/03/19 00:31:51.088 +00:00] [INFO] [db_helper.go:173] [\"sink uri is configured\"] [dsn=\"root:******@tcp(downstream.cdc-testbed-tps-1658689-1-113:3306)/?interpolateParams=true&multiStatements=true&charset=utf8mb4&foreign_key_checks=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%2CNO_AUTO_VALUE_ON_ZERO%22&time_zone=%22UTC%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m\"]\n[2023/03/19 00:31:51.093 +00:00] [ERROR] [db_helper.go:318] [\"check tidb version error\"] [error=\"Error 1046 (3D000): No database selected\"]\n[2023/03/19 00:31:51.093 +00:00] [INFO] [mysql.go:180] [\"MySQL backends is created\"] [changefeed=.] [workerCount=16] [forceReplicate=false] [enableOldValue=true]\n[2023/03/19 00:31:51.096 +00:00] [INFO] [sink.go:313] [\"succeed to parse parameter from sink uri\"] [protocol=] [txnAtomicity=]\n[2023/03/19 00:31:51.096 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=0]\n[2023/03/19 00:31:51.096 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=1]\n[2023/03/19 00:31:51.096 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=2]\n[2023/03/19 00:31:51.096 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=3]\n[2023/03/19 00:31:51.096 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=4]\n[2023/03/19 00:31:51.096 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=5]\n[2023/03/19 00:31:51.097 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=6]\n[2023/03/19 00:31:51.097 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=7]\n[2023/03/19 00:31:51.097 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=8]\n[2023/03/19 00:31:51.097 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=9]\n[2023/03/19 00:31:51.097 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=10]\n[2023/03/19 00:31:51.097 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=11]\n[2023/03/19 00:31:51.097 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=12]\n[2023/03/19 00:31:51.097 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=13]\n[2023/03/19 00:31:51.097 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=14]\n[2023/03/19 00:31:51.097 +00:00] [INFO] [worker.go:136] [\"Transaction dmlSink worker starts\"] [changefeedID=.] [workerID=15]\n[2023/03/19 00:31:51.112 +00:00] [INFO] [db_helper.go:173] [\"sink uri is configured\"] [dsn=\"root:******@tcp(downstream.cdc-testbed-tps-1658689-1-113:3306)/?interpolateParams=true&multiStatements=true&charset=utf8mb4&foreign_key_checks=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%2CNO_AUTO_VALUE_ON_ZERO%22&time_zone=%22UTC%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m\"]\n[2023/03/19 00:31:51.115 +00:00] [INFO] [mysql_ddl_sink.go:96] [\"MySQL DDL sink is created\"] [namespace=] [changefeed=]\n[2023/03/19 00:31:51.806 +00:00] [INFO] [file.go:119] [\"succeed to download and sort redo logs\"] [type=ddl] [duration=736.02094ms]\n[2023/03/19 00:42:35.786 +00:00] [INFO] [file.go:119] [\"succeed to download and sort redo logs\"] [type=row] [duration=10m44.714461084s]\n[2023/03/19 01:29:10.198 +00:00] [INFO] [table_sink_impl.go:145] [\"Stopping table sink\"] [namespace=default] [changefeed=redo-applier] [span={table_id:101,start_key:7480000000000000ff655f720000000000fa,end_key:7480000000000000ff655f730000000000fa}] [checkpointTs=440188384840515588]\n[2023/03/19 01:29:10.298 +00:00] [INFO] [table_sink_impl.go:153] [\"Table sink stopped\"] [namespace=default] [changefeed=redo-applier] [span={table_id:101,start_key:7480000000000000ff655f720000000000fa,end_key:7480000000000000ff655f730000000000fa}] [checkpointTs=440188384840515588] [duration=100.639081ms]\n[2023/03/19 01:29:10.298 +00:00] [INFO] [table_sink_impl.go:145] [\"Stopping table sink\"] [namespace=default] [changefeed=redo-applier] [span={table_id:96,start_key:7480000000000000ff605f720000000000fa,end_key:7480000000000000ff605f730000000000fa}] [checkpointTs=440188384840515588]\n[2023/03/19 01:29:10.398 +00:00] [INFO] [table_sink_impl.go:153] [\"Table sink stopped\"] [namespace=default] [changefeed=redo-applier] [span={table_id:96,start_key:7480000000000000ff605f720000000000fa,end_key:7480000000000000ff605f730000000000fa}] [checkpointTs=440188384840515588] [duration=100.050555ms]\n[2023/03/19 01:29:10.399 +00:00] [INFO] [table_sink_impl.go:145] [\"Stopping table sink\"] [namespace=default] [changefeed=redo-applier] [span={table_id:98,start_key:7480000000000000ff625f720000000000fa,end_key:7480000000000000ff625f730000000000fa}] [checkpointTs=440188384840515588]\n[2023/03/19 01:29:10.499 +00:00] [INFO] [table_sink_impl.go:153] [\"Table sink stopped\"] [namespace=default] [changefeed=redo-applier] [span={table_id:98,start_key:7480000000000000ff625f720000000000fa,end_key:7480000000000000ff625f730000000000fa}] [checkpointTs=440188384840515588] [duration=100.082879ms]\n[2023/03/19 01:29:10.499 +00:00] [INFO] [table_sink_impl.go:145] [\"Stopping table sink\"] [namespace=default] [changefeed=redo-applier] [span={table_id:93,start_key:7480000000000000ff5d5f720000000000fa,end_key:7480000000000000ff5d5f730000000000fa}] [checkpointTs=440188384840515588]\n[2023/03/19 01:29:10.599 +00:00] [INFO] [table_sink_impl.go:153] [\"Table sink stopped\"] [namespace=default] [changefeed=redo-applier] [span={table_id:93,start_key:7480000000000000ff5d5f720000000000fa,end_key:7480000000000000ff5d5f730000000000fa}] [checkpointTs=440188384840515588] [duration=100.150565ms]\n[2023/03/19 01:29:10.599 +00:00] [INFO] [table_sink_impl.go:145] [\"Stopping table sink\"] [namespace=default] [changefeed=redo-applier] [span={table_id:94,start_key:7480000000000000ff5e5f720000000000fa,end_key:7480000000000000ff5e5f730000000000fa}] [checkpointTs=440188384840515588]\n[2023/03/19 01:29:10.699 +00:00] [INFO] [table_sink_impl.go:153] [\"Table sink stopped\"] [namespace=default] [changefeed=redo-applier] [span={table_id:94,start_key:7480000000000000ff5e5f720000000000fa,end_key:7480000000000000ff5e5f730000000000fa}] [checkpointTs=440188384840515588] [duration=100.245001ms]\n[2023/03/19 01:29:10.699 +00:00] [INFO] [table_sink_impl.go:145] [\"Stopping table sink\"] [namespace=default] [changefeed=redo-applier] [span={table_id:99,start_key:7480000000000000ff635f720000000000fa,end_key:7480000000000000ff635f730000000000fa}] [checkpointTs=440188384840515588]\n[2023/03/19 01:29:10.800 +00:00] [INFO] [table_sink_impl.go:153] [\"Table sink stopped\"] [namespace=default] [changefeed=redo-applier] [span={table_id:99,start_key:7480000000000000ff635f720000000000fa,end_key:7480000000000000ff635f730000000000fa}] [checkpointTs=440188384840515588] [duration=100.506163ms]\n[2023/03/19 01:29:10.800 +00:00] [INFO] [table_sink_impl.go:145] [\"Stopping table sink\"] [namespace=default] [changefeed=redo-applier] [span={table_id:100,start_key:7480000000000000ff645f720000000000fa,end_key:7480000000000000ff645f730000000000fa}] [checkpointTs=440188384840515588]\n[2023/03/19 01:29:10.900 +00:00] [INFO] [table_sink_impl.go:153] [\"Table sink stopped\"] [namespace=default] [changefeed=redo-applier] [span={table_id:100,start_key:7480000000000000ff645f720000000000fa,end_key:7480000000000000ff645f730000000000fa}] [checkpointTs=440188384840515588] [duration=100.350887ms]\n[2023/03/19 01:29:10.900 +00:00] [INFO] [table_sink_impl.go:145] [\"Stopping table sink\"] [namespace=default] [changefeed=redo-applier] [span={table_id:97,start_key:7480000000000000ff615f720000000000fa,end_key:7480000000000000ff615f730000000000fa}] [checkpointTs=440188384840515588]\n[2023/03/19 01:29:11.001 +00:00] [INFO] [table_sink_impl.go:153] [\"Table sink stopped\"] [namespace=default] [changefeed=redo-applier] [span={table_id:97,start_key:7480000000000000ff615f720000000000fa,end_key:7480000000000000ff615f730000000000fa}] [checkpointTs=440188384840515588] [duration=100.375646ms]\n[2023/03/19 01:29:11.001 +00:00] [INFO] [table_sink_impl.go:145] [\"Stopping table sink\"] [namespace=default] [changefeed=redo-applier] [span={table_id:95,start_key:7480000000000000ff5f5f720000000000fa,end_key:7480000000000000ff5f5f730000000000fa}] [checkpointTs=440188384840515588]\n[2023/03/19 01:29:11.101 +00:00] [INFO] [table_sink_impl.go:153] [\"Table sink stopped\"] [namespace=default] [changefeed=redo-applier] [span={table_id:95,start_key:7480000000000000ff5f5f720000000000fa,end_key:7480000000000000ff5f5f730000000000fa}] [checkpointTs=440188384840515588] [duration=100.392659ms]\n[2023/03/19 01:29:11.101 +00:00] [INFO] [table_sink_impl.go:145] [\"Stopping table sink\"] [namespace=default] [changefeed=redo-applier] [span={table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}] [checkpointTs=440188384840515588]\n[2023/03/19 01:29:11.201 +00:00] [INFO] [table_sink_impl.go:153] [\"Table sink stopped\"] [namespace=default] [changefeed=redo-applier] [span={table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}] [checkpointTs=440188384840515588] [duration=100.344366ms]\n[2023/03/19 01:29:11.201 +00:00] [INFO] [redo.go:216] [\"apply redo log finishes\"] [appliedLogCount=403774] [appliedDDLCount=0] [currentCheckpoint=440188384840515588]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=0]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=0]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=0] [elapsed=725.173µs]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=1]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=1]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=1] [elapsed=13.416µs]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=2]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=2]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=2] [elapsed=23.534µs]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=3]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=3]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=3] [elapsed=12.545µs]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=4]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=4]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=4] [elapsed=20.901µs]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=5]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=5]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=5] [elapsed=20.385µs]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=6]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=6]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=6] [elapsed=11.666µs]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=7]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=7]\n[2023/03/19 01:29:11.202 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=7] [elapsed=17.2µs]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=8]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=8]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=8] [elapsed=23.515µs]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=9]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=9]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=9] [elapsed=27.11µs]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=10]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=10]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=10] [elapsed=20.437µs]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=11]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=11]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=11] [elapsed=13.829µs]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=12]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=12]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=12] [elapsed=13.614µs]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=13]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=13]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=13] [elapsed=22.32µs]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=14]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=14]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=14] [elapsed=15.428µs]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:97] [\"Closing txn worker\"] [changefeed=.] [worker=15]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:157] [\"Transaction dmlSink worker exits as closed\"] [changefeedID=.] [workerID=15]\n[2023/03/19 01:29:11.203 +00:00] [INFO] [worker.go:104] [\"Closed txn worker\"] [changefeed=.] [worker=15] [elapsed=18.004µs]\nApply redo log successfully"] []
[2023/03/19 01:29:11.287 +00:00] [INFO] [step.go:30] ["redo: flag=&{Storage:s3://tmp/test-infra-redolog/redo-apply-basic-real-time-sync-bd90c86c-2038-4064-b811-cbbb3dcfda80?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http://172.16.6.5:9000&force-path-style=true TmpDir:redo-apply-basic-real-time-sync}"]
2023-03-19T01:29:11.287Z	INFO	host/ticdc.go:283	Execute command on ticdc	{"command": "/cdc  redo  meta \"--storage=s3://tmp/test-infra-redolog/redo-apply-basic-real-time-sync-bd90c86c-2038-4064-b811-cbbb3dcfda80?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http://172.16.6.5:9000&force-path-style=true\" \"--tmp-dir=/tmp/nfs/meta/redo-apply-basic-real-time-sync\"", "timeout": "2h0m0s", "ticdc peer": 0}
2023-03-19T01:29:11.287Z	INFO	k8s/client.go:132	it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
[2023/03/19 01:29:11.631 +00:00] [INFO] [redo.go:29] ["run redo cmd finished"] [result="{\"checkPointTs\":440188383660867591,\"resolvedTs\":440188384840515588}"] []
[2023/03/19 01:29:11.631 +00:00] [INFO] [step.go:30] ["checkConsistency: dbName=workload"]
W0319 01:29:11.631670      24 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0319 01:29:11.673482      24 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2023-03-19T01:29:11.697Z	INFO	consistency/crc32.go:205	start to get source checksum	{"databases": ["workload"]}
2023-03-19T01:29:11.697Z	INFO	consistency/crc32.go:217	start to get sink checksum	{"databases": ["workload"]}
2023-03-19T01:29:11.698Z	INFO	consistency/crc32.go:25	start to get tables from databases	{"databases": ["workload"]}
2023-03-19T01:29:11.698Z	INFO	consistency/crc32.go:25	start to get tables from databases	{"databases": ["workload"]}
2023-03-19T01:29:11.701Z	INFO	consistency/crc32.go:40	set tidb_snapshot successfully	{"snapshotTS": "440188384840515588"}
2023-03-19T01:29:11.702Z	INFO	consistency/crc32.go:62	get all tables	{"tables": ["workload.sbtest1", "workload.sbtest10", "workload.sbtest2", "workload.sbtest3", "workload.sbtest4", "workload.sbtest5", "workload.sbtest6", "workload.sbtest7", "workload.sbtest8", "workload.sbtest9"]}
2023-03-19T01:29:11.704Z	INFO	consistency/crc32.go:129	set tidb_snapshot successfully	{"snapshotTS": "440188384840515588"}
2023-03-19T01:29:11.704Z	INFO	consistency/crc32.go:138	set tidb_mem_quota_query successfully	{"quota": "4294967296"}
2023-03-19T01:29:11.705Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest1"}
2023-03-19T01:29:11.705Z	INFO	consistency/crc32.go:62	get all tables	{"tables": ["workload.sbtest1", "workload.sbtest10", "workload.sbtest2", "workload.sbtest3", "workload.sbtest4", "workload.sbtest5", "workload.sbtest6", "workload.sbtest7", "workload.sbtest8", "workload.sbtest9"]}
2023-03-19T01:29:11.710Z	INFO	consistency/crc32.go:129	set tidb_snapshot successfully	{"snapshotTS": "440188384840515588"}
2023-03-19T01:29:11.710Z	INFO	consistency/crc32.go:129	set tidb_snapshot successfully	{"snapshotTS": "440188384840515588"}
2023-03-19T01:29:11.710Z	INFO	consistency/crc32.go:138	set tidb_mem_quota_query successfully	{"quota": "4294967296"}
2023-03-19T01:29:11.710Z	INFO	consistency/crc32.go:129	set tidb_snapshot successfully	{"snapshotTS": "440188384840515588"}
2023-03-19T01:29:11.710Z	INFO	consistency/crc32.go:138	set tidb_mem_quota_query successfully	{"quota": "4294967296"}
2023-03-19T01:29:11.710Z	INFO	consistency/crc32.go:129	set tidb_snapshot successfully	{"snapshotTS": "440188384840515588"}
2023-03-19T01:29:11.710Z	INFO	consistency/crc32.go:129	set tidb_snapshot successfully	{"snapshotTS": "440188384840515588"}
2023-03-19T01:29:11.710Z	INFO	consistency/crc32.go:129	set tidb_snapshot successfully	{"snapshotTS": "440188384840515588"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:129	set tidb_snapshot successfully	{"snapshotTS": "440188384840515588"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:129	set tidb_snapshot successfully	{"snapshotTS": "440188384840515588"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest4"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:138	set tidb_mem_quota_query successfully	{"quota": "4294967296"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:129	set tidb_snapshot successfully	{"snapshotTS": "440188384840515588"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:138	set tidb_mem_quota_query successfully	{"quota": "4294967296"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:138	set tidb_mem_quota_query successfully	{"quota": "4294967296"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:138	set tidb_mem_quota_query successfully	{"quota": "4294967296"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest3"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest5"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:138	set tidb_mem_quota_query successfully	{"quota": "4294967296"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest7"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest2"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:138	set tidb_mem_quota_query successfully	{"quota": "4294967296"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:138	set tidb_mem_quota_query successfully	{"quota": "4294967296"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest8"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest9"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest2"}
2023-03-19T01:29:11.711Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest10"}
2023-03-19T01:29:11.712Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest9"}
2023-03-19T01:29:11.712Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest6"}
2023-03-19T01:29:11.712Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest5"}
2023-03-19T01:29:11.712Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest10"}
2023-03-19T01:29:11.713Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest7"}
2023-03-19T01:29:11.713Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest3"}
2023-03-19T01:29:11.714Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest6"}
2023-03-19T01:29:11.724Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest4"}
2023-03-19T01:29:11.725Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest1"}
2023-03-19T01:29:11.735Z	INFO	consistency/crc32.go:169	do checksum start	{"table": "workload.sbtest8"}
2023-03-19T01:29:18.612Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest1", "checkSum": 4241478648}
2023-03-19T01:29:19.146Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest9", "checkSum": 1930460330}
2023-03-19T01:29:19.740Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest1", "checkSum": 4241478648}
2023-03-19T01:29:19.908Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest2", "checkSum": 954960948}
2023-03-19T01:29:20.528Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest2", "checkSum": 954960948}
2023-03-19T01:29:20.612Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest6", "checkSum": 4022943676}
2023-03-19T01:29:21.045Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest4", "checkSum": 2446382771}
2023-03-19T01:29:21.111Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest5", "checkSum": 320043325}
2023-03-19T01:29:21.426Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest8", "checkSum": 321527155}
2023-03-19T01:29:21.597Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest5", "checkSum": 320043325}
2023-03-19T01:29:21.722Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest7", "checkSum": 1042511973}
2023-03-19T01:29:21.776Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest3", "checkSum": 2390360086}
2023-03-19T01:29:21.798Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest7", "checkSum": 1042511973}
2023-03-19T01:29:22.050Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest9", "checkSum": 1930460330}
2023-03-19T01:29:22.170Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest6", "checkSum": 1133977101}
2023-03-19T01:29:22.173Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest10", "checkSum": 1192527216}
2023-03-19T01:29:22.275Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest4", "checkSum": 2446382771}
2023-03-19T01:29:22.469Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest3", "checkSum": 2390360086}
2023-03-19T01:29:22.701Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest10", "checkSum": 1192527216}
2023-03-19T01:29:22.863Z	INFO	consistency/crc32.go:185	do checkSum success	{"table": "workload.sbtest8", "checkSum": 321527155}
2023-03-19T01:29:22.863Z	INFO	consistency/crc32.go:238	get source checksum finished
2023-03-19T01:29:22.863Z	INFO	consistency/crc32.go:243	get sink checksum finished
2023-03-19T01:29:22.863Z	ERROR	consistency/crc32.go:260	crc32 mismatch	{"table": "workload.sbtest6", "source": 4022943676, "sink": 1133977101}

[2023/03/19 00:31:49.643 +00:00] [INFO] [middleware.go:48] [/api/v2/changefeeds/redo-apply-basic-real-time-sync/pause] [status=200] [method=POST] [path=/api/v2/changefeeds/redo-apply-basic-real-time-sync/pause] [query=] [ip=127.0.0.1] [user-agent=Go-http-client/1.1] [client-version=v6.7.0-alpha] [] [duration=372.451184ms]
[2023/03/19 00:31:50.042 +00:00] [INFO] [processor.go:940] ["processor closing ..."] [namespace=default] [changefeed=redo-apply-basic-real-time-sync]
[2023/03/19 00:31:50.042 +00:00] [INFO] [processor.go:945] ["Processor try to close sink manager"] [namespace=default] [changefeed=redo-apply-basic-real-time-sync]
[2023/03/19 00:31:50.042 +00:00] [INFO] [manager.go:872] ["Closing sink manager"] [namespace=default] [changefeed=redo-apply-basic-real-time-sync]
[2023/03/19 00:31:50.042 +00:00] [INFO] [table_sink_impl.go:145] ["Stopping table sink"] [namespace=default] [changefeed=redo-apply-basic-real-time-sync] [span={table_id:95,start_key:7480000000000000ff5f5f720000000000fa,end_key:7480000000000000ff5f5f730000000000fa}] [checkpointTs=440188384381763591]
[2023/03/19 00:31:50.042 +00:00] [INFO] [table_sink_impl.go:153] ["Table sink stopped"] [namespace=default] [changefeed=redo-apply-basic-real-time-sync] [span={table_id:95,start_key:7480000000000000ff5f5f720000000000fa,end_key:7480000000000000ff5f5f730000000000fa}] [checkpointTs=440188384381763591] [duration=35.307µs]
[2023/03/19 00:31:50.042 +00:00] [INFO] [table_sink_wrapper.go:206] ["Sink is closed"] [span={table_id:95,start_key:7480000000000000ff5f5f720000000000fa,end_key:7480000000000000ff5f5f730000000000fa}] [namespace=default] [changefeed=redo-apply-basic-real-time-sync]
[2023/03/19 00:31:50.043 +00:00] [INFO] [table_sink_impl.go:145] ["Stopping table sink"] [namespace=default] [changefeed=redo-apply-basic-real-time-sync] [span={table_id:96,start_key:7480000000000000ff605f720000000000fa,end_key:7480000000000000ff605f730000000000fa}] [checkpointTs=440188384525680641]
[2023/03/19 00:31:50.043 +00:00] [INFO] [table_sink_impl.go:153] ["Table sink stopped"] [namespace=default] [changefeed=redo-apply-basic-real-time-sync] [span={table_id:96,start_key:7480000000000000ff605f720000000000fa,end_key:7480000000000000ff605f730000000000fa}] [checkpointTs=440188384525680641] [duration=29.219µs]
[2023/03/19 00:31:50.043 +00:00] [INFO] [table_sink_wrapper.go:206] ["Sink is closed"] [span={table_id:96,start_key:7480000000000000ff605f720000000000fa,end_key:7480000000000000ff605f730000000000fa}] [namespace=default] [changefeed=redo-apply-basic-real-time-sync]
[2023/03/19 00:31:50.043 +00:00] [INFO] [table_sink_impl.go:145] ["Stopping table sink"] [namespace=default] [changefeed=redo-apply-basic-real-time-sync] [span={table_id:98,start_key:7480000000000000ff625f720000000000fa,end_key:7480000000000000ff625f730000000000fa}] [checkpointTs=440188384604585995]
[2023/03/19 00:31:50.043 +00:00] [INFO] [table_sink_impl.go:153] ["Table sink stopped"] [namespace=default] [changefeed=redo-apply-basic-real-time-sync] [span={table_id:98,start_key:7480000000000000ff625f720000000000fa,end_key:7480000000000000ff625f730000000000fa}] [checkpointTs=440188384604585995] [duration=24.834µs]
[2023/03/19 00:31:50.043 +00:00] [INFO] [table_sink_wrapper.go:206] ["Sink is closed"] [span={table_id:98,start_key:7480000000000000ff625f720000000000fa,end_key:7480000000000000ff625f730000000000fa}] [namespace=default] [changefeed=redo-apply-basic-real-time-sync]
[2023/03/19 00:31:50.044 +00:00] [INFO] [worker.go:152] ["Transaction dmlSink worker exits as canceled"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=15]
[2023/03/19 00:31:50.045 +00:00] [INFO] [manager.go:287] ["Background GC is stooped because context is canceled"] [namespace=default] [changefeed=redo-apply-basic-real-time-sync]
[2023/03/19 00:31:50.045 +00:00] [WARN] [encoding_worker.go:128] ["redo fileWorkerGroup closed with error"] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/cdc/redo/writer/memory.(*encodingWorkerGroup).runWorker\n\tgithub.com/pingcap/tiflow/cdc/redo/writer/memory/encoding_worker.go:153\ngithub.com/pingcap/tiflow/cdc/redo/writer/memory.(*encodingWorkerGroup).Run.func2\n\tgithub.com/pingcap/tiflow/cdc/redo/writer/memory/encoding_worker.go:135\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1598"]
[2023/03/19 00:31:50.045 +00:00] [WARN] [file_worker.go:142] ["redo file workers closed with error"] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/cdc/redo/writer/memory.(*fileWorkerGroup).bgWriteLogs\n\tgithub.com/pingcap/tiflow/cdc/redo/writer/memory/file_worker.go:193\ngithub.com/pingcap/tiflow/cdc/redo/writer/memory.(*fileWorkerGroup).Run.func2\n\tgithub.com/pingcap/tiflow/cdc/redo/writer/memory/file_worker.go:148\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1598"]
[2023/03/19 00:31:50.045 +00:00] [ERROR] [manager.go:503] ["redo manager fails to close writer"] [namespace=default] [changefeed=redo-apply-basic-real-time-sync] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/cdc/redo/writer/memory.(*encodingWorkerGroup).runWorker\n\tgithub.com/pingcap/tiflow/cdc/redo/writer/memory/encoding_worker.go:153\ngithub.com/pingcap/tiflow/cdc/redo/writer/memory.(*encodingWorkerGroup).Run.func2\n\tgithub.com/pingcap/tiflow/cdc/redo/writer/memory/encoding_worker.go:135\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1598"]
[2023/03/19 00:31:50.045 +00:00] [INFO] [manager.go:508] ["redo manager closed"] [namespace=default] [changefeed=redo-apply-basic-real-time-sync]
[2023/03/19 00:31:50.046 +00:00] [INFO] [worker.go:152] ["Transaction dmlSink worker exits as canceled"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=2]
[2023/03/19 00:31:50.046 +00:00] [WARN] [mysql.go:800] ["execute DMLs with error, retry later"] [error="[CDC:ErrMySQLTxnError]MySQL txn error: sql: transaction has already been committed or rolled back"] [errorVerbose="[CDC:ErrMySQLTxnError]MySQL txn error: sql: transaction has already been committed or rolled back\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1.3\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:773\ngithub.com/pingcap/tiflow/cdc/sink/metrics.(*Statistics).RecordBatchExecution\n\tgithub.com/pingcap/tiflow/cdc/sink/metrics/statistics.go:74\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:728\ngithub.com/pingcap/tiflow/pkg/retry.run\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:715\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).Flush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:218\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).doFlush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:214\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).runBackgroundLoop.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:163\nruntime.goexit\n\truntime/asm_amd64.s:1598"] [duration=48.029416ms] [query=COMMIT] [count=267] [startTs="[440188384643907590]"] [changefeed=default.redo-apply-basic-real-time-sync]
[2023/03/19 00:31:50.046 +00:00] [WARN] [worker.go:215] ["Transaction dmlSink backend flush fail"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=1] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/retry.run\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:90\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:715\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).Flush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:218\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).doFlush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:214\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).runBackgroundLoop.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:163\nruntime.goexit\n\truntime/asm_amd64.s:1598"]
[2023/03/19 00:31:50.046 +00:00] [WARN] [worker.go:178] ["Transaction dmlSink worker exits unexceptedly"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=1]
[2023/03/19 00:31:50.046 +00:00] [INFO] [worker.go:152] ["Transaction dmlSink worker exits as canceled"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=6]
[2023/03/19 00:31:50.047 +00:00] [ERROR] [mysql.go:806] ["execute DMLs with error, can not retry"] [error="[CDC:ErrMySQLTxnError]MySQL txn error: context canceled"] [errorVerbose="[CDC:ErrMySQLTxnError]MySQL txn error: context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).multiStmtExecute\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:644\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1.3\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:742\ngithub.com/pingcap/tiflow/cdc/sink/metrics.(*Statistics).RecordBatchExecution\n\tgithub.com/pingcap/tiflow/cdc/sink/metrics/statistics.go:74\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:728\ngithub.com/pingcap/tiflow/pkg/retry.run\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:715\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).Flush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:218\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).doFlush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:214\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).runBackgroundLoop.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:167\nruntime.goexit\n\truntime/asm_amd64.s:1598"] [duration=30.027µs] [query="INSERT INTO `workload`.`sbtest8` (`id`,`k`,`c`,`pad`) VALUES (?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,"] [count=135] [changefeed=default.redo-apply-basic-real-time-sync]
[2023/03/19 00:31:50.047 +00:00] [WARN] [mysql.go:648] ["failed to rollback txn"] [error="invalid connection"]
[2023/03/19 00:31:50.047 +00:00] [WARN] [worker.go:215] ["Transaction dmlSink backend flush fail"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=3] [error="[CDC:ErrMySQLTxnError]MySQL txn error: context canceled"] [errorVerbose="[CDC:ErrMySQLTxnError]MySQL txn error: context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).multiStmtExecute\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:644\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1.3\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:742\ngithub.com/pingcap/tiflow/cdc/sink/metrics.(*Statistics).RecordBatchExecution\n\tgithub.com/pingcap/tiflow/cdc/sink/metrics/statistics.go:74\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:728\ngithub.com/pingcap/tiflow/pkg/retry.run\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:715\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).Flush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:218\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).doFlush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:214\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).runBackgroundLoop.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:167\nruntime.goexit\n\truntime/asm_amd64.s:1598"]
[2023/03/19 00:31:50.047 +00:00] [WARN] [worker.go:178] ["Transaction dmlSink worker exits unexceptedly"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=3]
[2023/03/19 00:31:50.047 +00:00] [WARN] [mysql.go:800] ["execute DMLs with error, retry later"] [error="[CDC:ErrMySQLTxnError]MySQL txn error: sql: transaction has already been committed or rolled back"] [errorVerbose="[CDC:ErrMySQLTxnError]MySQL txn error: sql: transaction has already been committed or rolled back\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1.3\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:773\ngithub.com/pingcap/tiflow/cdc/sink/metrics.(*Statistics).RecordBatchExecution\n\tgithub.com/pingcap/tiflow/cdc/sink/metrics/statistics.go:74\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:728\ngithub.com/pingcap/tiflow/pkg/retry.run\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:715\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).Flush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:218\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).doFlush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:214\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).runBackgroundLoop.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:163\nruntime.goexit\n\truntime/asm_amd64.s:1598"] [duration=45.001648ms] [query=COMMIT] [count=269] [startTs="[440188384617693195]"] [changefeed=default.redo-apply-basic-real-time-sync]
[2023/03/19 00:31:50.047 +00:00] [WARN] [worker.go:215] ["Transaction dmlSink backend flush fail"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=5] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/retry.run\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:90\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:715\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).Flush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:218\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).doFlush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:214\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).runBackgroundLoop.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:163\nruntime.goexit\n\truntime/asm_amd64.s:1598"]
[2023/03/19 00:31:50.048 +00:00] [WARN] [worker.go:178] ["Transaction dmlSink worker exits unexceptedly"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=5]
[2023/03/19 00:31:50.048 +00:00] [INFO] [worker.go:152] ["Transaction dmlSink worker exits as canceled"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=11]
[2023/03/19 00:31:50.048 +00:00] [INFO] [worker.go:152] ["Transaction dmlSink worker exits as canceled"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=4]
[2023/03/19 00:31:50.048 +00:00] [INFO] [worker.go:152] ["Transaction dmlSink worker exits as canceled"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=0]
[2023/03/19 00:31:50.048 +00:00] [INFO] [worker.go:152] ["Transaction dmlSink worker exits as canceled"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=7]
[2023/03/19 00:31:50.049 +00:00] [INFO] [worker.go:152] ["Transaction dmlSink worker exits as canceled"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=8]
[2023/03/19 00:31:50.049 +00:00] [INFO] [client.go:1115] ["stream to store closed"] [namespace=default] [changefeed=redo-apply-basic-real-time-sync] [addr=upstream-tikv-2.upstream-tikv-peer.cdc-testbed-tps-1658689-1-113.svc:20160] [storeID=5]
[2023/03/19 00:31:50.049 +00:00] [INFO] [worker.go:152] ["Transaction dmlSink worker exits as canceled"] [changefeedID=default.redo-apply-basic-real-time-sync] [workerID=9]
[2023/03/19 00:31:50.049 +00:00] [ERROR] [mysql.go:806] ["execute DMLs with error, can not retry"] [error="[CDC:ErrMySQLTxnError]MySQL txn error: context canceled"] [errorVerbose="[CDC:ErrMySQLTxnError]MySQL txn error: context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).multiStmtExecute\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:644\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1.3\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:742\ngithub.com/pingcap/tiflow/cdc/sink/metrics.(*Statistics).RecordBatchExecution\n\tgithub.com/pingcap/tiflow/cdc/sink/metrics/statistics.go:74\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:728\ngithub.com/pingcap/tiflow/pkg/retry.run\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:715\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql.(*mysqlBackend).Flush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/mysql.go:218\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).doFlush\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:214\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn.(*worker).runBackgroundLoop.func1\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/txn/worker.go:163\nruntime.goexit\n\truntime/asm_amd64.s:1598"] [duration=50.164µs] [query="INSERT INTO `workload`.`sbtest1` (`id`,`k`,`c`,`pad`) VALUES (?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,?,?,?),(?,"] [count=269] [changefeed=default.redo-apply-basic-real-time-sync]

Versions of the cluster

TiCDC version (execute cdc version):
["Welcome to Change Data Capture (CDC)"] [release-version=v6.7.0-alpha] [git-hash=41b40aaff95f8754eea7ec792135debe70b8e4da] [git-branch=heads/refs/tags/v6.7.0-alpha] [utc-build-time="2023-03-18 11:32:23"] [go-version="go version go1.20.2 linux/amd64"]
@fubinzh fubinzh added area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug. labels Mar 20, 2023
@fubinzh fubinzh closed this as completed Mar 20, 2023
@fubinzh fubinzh reopened this Mar 20, 2023
@fubinzh
Copy link
Author

fubinzh commented Mar 20, 2023

/found automation

@ti-chi-bot ti-chi-bot added the found/automation Bugs found by automation cases label Mar 20, 2023
@fubinzh
Copy link
Author

fubinzh commented Mar 20, 2023

/severity critical

@nongfushanquan
Copy link
Contributor

/assign @CharlesCheung96

@CharlesCheung96 CharlesCheung96 mentioned this issue Mar 21, 2023
Merged
ti-chi-bot pushed a commit that referenced this issue Mar 22, 2023
@CharlesCheung96
redo(ticdc): fix resolvedTs regression in applier's memory quota reco…
def41a6 
…rd (#8620)

close #8591
This was referenced Mar 22, 2023
Merged
Merged
ti-chi-bot added a commit that referenced this issue Mar 23, 2023
@ti-chi-bot
redo(ticdc): fix resolvedTs regression in applier's memory quota reco…
4fd7039 
…rd (#8620) (#8640)

close #8591
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this issue Apr 3, 2023
@CharlesCheung96 @3AceShowHand
redo(ticdc): fix resolvedTs regression in applier's memory quota reco…
1dc3fc3 
…rd (pingcap#8620)

close pingcap#8591
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Apr 4, 2023
@CharlesCheung96
redo(ticdc): fix resolvedTs regression in applier's memory quota reco…
da64a1f 
…rd (pingcap#8620)

close pingcap#8591
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Apr 4, 2023
@CharlesCheung96
redo(ticdc): fix resolvedTs regression in applier's memory quota reco…
b6db2f3 
…rd (pingcap#8620)

close pingcap#8591
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Apr 4, 2023
@CharlesCheung96
redo(ticdc): fix resolvedTs regression in applier's memory quota reco…
4b69c62 
…rd (pingcap#8620)

close pingcap#8591
ti-chi-bot added a commit that referenced this issue Apr 4, 2023
@ti-chi-bot
redo(ticdc): fix resolvedTs regression in applier's memory quota reco…
e9e5c2e 
…rd (#8620) (#8639)

close #8591
@nongfushanquan nongfushanquan mentioned this issue Apr 18, 2023
Merged
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CharlesCheung96 CharlesCheung96

Labels
affects-6.5 affects-7.0 area/ticdc Issues or PRs related to TiCDC. found/automation Bugs found by automation cases severity/critical type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

redo(ticdc): fix resolvedTs regression in applier's memory quota record CharlesCheung96/tiflow
5 participants
@fubinzh @VelocityLight @CharlesCheung96 @ti-chi-bot @nongfushanquan