SQS message not deleted after successful acknowledgement

[otbe]

Hi,

its me again :)

We observe a strange behaviour in one of our applications. We consume SQS messages from a FIFO queue, handle them and that's it. We use immediate acknowledgement but also tried manual acknowledgement and the behaviour is the same.

Sometimes (5 times a day while processing ~10k messages) it happens that a message is successfully handled, acknowledged/deleted but SQS delivers it again after the visibility timeout. We turned on debug logs for io.awspring and can see that the message was successful acknowledged.

See also this picture

There's no other log around this time. We created a Cloudtrail for this queue and there's no DeleteMessageBatch call for this specific message id. So its look like it was not really deleted from an AWS point of view. For other messages we see SendMessage and somewhen later DeleteMessageBatch

What makes this even more weird:

• the same application with similar message volumes runs in 5 different AWS accounts (but same region), it only happens in one of them
• it only happens for a very specific type of message not random for all of them

Any idea what could cause this from a awspring perspective?

Thank you :)

[otbe]

Followup question: was looking around in the code and I wonder what happens when the deleteMessageBatch call is successful but contains error. There's no log line about this right?

[tomazfernandes]

Thanks for the heads up @otbe, good catch. We could definitely add logging there to detect failed acknowledgements within an otherwise successful acknowledgement batch.

Would you be up for opening an issue, and perhaps contributing the change?

Another idea down the line could be surfacing the result in AcknowledgementResultCallback, likely via Message Headers, but logging looks like a solid first step.

Let me know your thoughts, thanks.

[otbe]

Yes that would help a lot at least for debugging purposes. We now implemented a custom acknowledgement/delete method that a) logs this and b) also retries it.

[tomazfernandes]

Created this issue @otbe, feel free to contribute a PR if you'd like.
#1497