mq_sink: unexpected behavior of max-message-bytes

[liuzix]

What did you do?

Assume the Kafka broker has 1M as message.max.bytes. I use TiCDC with Kafka sink and Open Protocol, where I set max-message-bytes=10K.

What did you expect to see?

If there is no row whose serialized size is more than 1M then everything works normally. If there is, I would need a detailed error message helping me identify the problematic row (and table).

The expected behavior is that: as long as CDC has max-message-bytes <= 1M and there is no row larger than 1M, everything should work.

What did you see instead?

Any message larger than 10K fails to be delivered.

The root cause is that (1) max-message-bytes is used both as an advisory to the Open Protocol encoder AND as a hard-limit for the kafka producer (sarama) (2) we have failed to take into account the producer overhead by the kafka client.

Problematic Scenarios

a.

|<------ CDC's limit = Broker's Limit = 10M ----->|<-overflow->|
| row A = 5M | row B = 4M | row C = 1000K| overhead = 100K|

The batch should have been reduced, but was not.

b.
|<--------------------------- Broker's Limit = 10M ------------------------>|
|<---- CDC's Limit = 5M --->|<--overflow-->|
|<------------- row A = 6M ----------------------->|<--still space here->|

The message should have been sent, but was not.

Versions of the cluster

v5.0.1

[cx2671206]

@liuzix Want to know is there a difference between set max-message-bytes=10K and max-message-bytes=1MB in this case? Curious why you set it to 10K when kafka size limit is 1MB

If Kafka limit is 1MB, and max-message-bytes=10KB, what's the expected behavior for event with size larger than 10KB?

[liuzix]

@liuzix Want to know is there a difference between set max-message-bytes=10K and max-message-bytes=1MB in this case? Curious why you set it to 10K when kafka size limit is 1MB

If Kafka limit is 1MB, and max-message-bytes=10KB, what's the expected behavior for event with size larger than 10KB?

If a single event has to take up say 5MB, we have to send it out in a batch containing only itself. Otherwise, data loss would occur.

[yiwu-arbug]

For the particular case we didn't find "Event does not fit into max-message-bytes. Adjust relevant configurations to avoid service interruptions." log line. @lance6716 confirmed we rule out case b, and the issue is likely caused by case a.

[lance6716]

For the particular case we didn't find "Event does not fit into max-message-bytes. Adjust relevant configurations to avoid service interruptions." log line. @lance6716 confirmed we rule out case b, and the issue is likely caused by case a.

now for a new log, case b is also triggered.

[3AceShowHand]

Consider we set max-message-size as 100, and now the message's size is 99, so the log will not trigger.

But when sending the message to the sarama producer, it will check the message's size, 99 + 36 = 135, 135 > 100, so reject it.