Batch acknowledge can lead to message being never acknowledged (until consumer restart)

[Lanayx]

Describe the bug
Issue can happen when acknowledge request fails due to temporary network failure. The root of the issue is that isDuplicate method depends on lastCumulativeAck field which is updated before the acknowledgment request succeeds and cumulativeAckFlushRequired is updated before that. Therefore on failure lastCumulativeAck will stay at messageId that wasn't actually acked and unacked messages will be rejected on redelivery as duplicates.

To Reproduce
Steps to reproduce the behavior:

let say we have 10 messages in a queue

1. Do batch acknowledge for the last message (to acknowledge all)
2. Temporary network failure happens and acknowledgement flush fails, while lastCumulativeAck becomes set to 10th message
3. Consumer reconnects and all 10 messages are redelivered
4. All 10 messages will be rejected as duplicates and won't be consumed again until consumer restart. On consumer restart they will be redelivered and reprocessed.

Expected behavior
10 messages will be reconsumed normally or even better - reacknowledged transparently without consuming by the application

Additional context
Idea:
As pulsar doesn't support acknowledge guarantees , this prevents having "effectively once delivery", since consumer is expected to process the messages that were unable to be acked twice. Transparent reacknowledgment of such messages on redelivery (for both normal and cumulative acknowledgment) can fix this issue for the most cases.

[sijie]

A simple fix will be resetting the consumer state when a consumer reconnects.

Just want to clarify one thing about "effectively once delivery", the effectively-once delivery is achieved by using readers or consumers with external checkpoint or offset management mechanisms. Acknowledgment without transactions doesn't give you "effectively-once" delivery semantic.

[Lanayx]

@sijie while simple fix will fix the issue, from application perspective it will be much better if the acks that were unable to be sent while reconnecting will be sent automatically once connection is re-established than to suddenly face duplicate messages (as reconnect is hidden to the application as well as acknowledge failure).

[sijie]

@Lanayx understood. The main problem of this is we lack the response of ack command. So it is hard for the client to know whether the ack is sent successfully or not.

[Lanayx]

@sijie Right, but at the same time if connection failed during the flush - we definitely know that acks were not sent, so AcknowledgmentsGroupingTracker state can be reverted to the state before flush just like connection was missing in the first place

[lhotari]

@Lanayx understood. The main problem of this is we lack the response of ack command. So it is hard for the client to know whether the ack is sent successfully or not.

@sijie It seems that ack response was added by #8996 and that is included in Pulsar 2.8.0 . Would that help resolve this issue (#7683) ? /cc @congbobo184 @codelipenghui

[tisonkun]

Closed as stale and may be resolved. Please create a new issue if it's still relevant to the maintained versions.