kafka consumer missing data cause data inconsistency when the brokers failed frequently

[fubinzh]

What did you do?

1. create kafka changefeed

/cdc  cli  changefeed  create \"--server=127.0.0.1:8301\" \"--sink-uri=kafka://downstream-kafka.cdc-testbed
-tps-1807799-1-20:9092/broker_fail?max-message-bytes=1048576&replication-factor=2&enable-tidb-extension=true&protocol=canal-json\" \"--changefeed-id=kafka-random-broker-fail-canal-json\"

2. run workload and inject kafka broker failure (case: kafka_random_broker_fail).
   "sysbench --db-driver=mysql --mysql-host=nslookup upstream-tidb.cdc-testbed-tps-1807799-1-20 | awk -F: '{pr int $2}' | awk 'NR==5' | sed s/[[:space:]]//g --mysql-port=4000 --mysql-user=root --mysql-db=workload --tables=100 --table-size=100000 --create_secondary=off --time=1800 --debug=true --threads=100 --mysql-ignore-errors=2013,1213,1105,1205,8022,8027,8028,9004,9007,1062 oltp_update_non_index run"
3. send finishmark
4. run kafka consumer to consumer kafka log to downstream mysql
5. do data consistency check when finishmark is sent to downstream mysql

What did you expect to see?

data should be consistent

What did you see instead?

data inconsistecy seen, 3 out of 100 tables are inconsistent.

Versions of the cluster

cdc version

Release Version: v7.2.0-alpha
Git Commit Hash: 98bd965630ff2f6efd105810a790a2c5e8e389e4
Git Branch: heads/refs/tags/v7.2.0-alpha
UTC Build Time: 2023-06-13 11:32:27
Go Version: go version go1.20.3 linux/amd64
Failpoint Build: false

[fubinzh]

Kafka consumer log:
cdc_kafka_consumer.log

[fubinzh]

case kafka_controller_fail also fails due to consistency check failure. https://tcms.pingcap.net/dashboard/executions/plan/1808271

[fubinzh]

/severity critical

[fubinzh]

/found automation

[3AceShowHand]

[2023/06/14 03:39:17.771 +00:00] [PANIC] [main.go:346] ["Error from consumer: %v"] [error="dial tcp 10.233.78.133:9092: connect: connection refused"] [stack="main.main.func1\n\tgithub.com/pingcap/tiflow/cmd/kafka-consumer/main.go:346"]

[3AceShowHand]

After investigation, the data loss comes from the kafka consumer part, not the cdc.

[fubinzh]

/remove-severity critical

[fubinzh]

/severity major

[3AceShowHand]

/severity moderate

[nongfushanquan]

#9694
all other issues of the consumer will link to the above issue
/close

[ti-chi-bot]

@nongfushanquan: Closing this issue.

In response to this:

#9694
all other issues of the consumer will link to the above issue
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.