Support for sending and receiving large message in SQS

[vijayrcse]

Type: Feature

Is your feature request related to a problem? Please describe.
We have a use case where our message size is larger than 254kb. is there a support in spring-cloud-sqs-messaging to send and receive large size messages using SQS extended client library

[tomazfernandes]

Hi @vijayrcse, thanks for bringing this up.

It should be possible to send larger messages by using S3 to store the payload, and sending only the reference in the message

There's a java-extended-client-lib that has this feature built in, as described here

Unfortunately it implements the blocking SqsClient, which is not supported by the framework. It might be possible to create an adapter or wrapper to use it with the SqsTemplate.

Let me know how it works for you.

Thanks.

[NightFox7]

Hello !
I am also interested. I cannot find a way to make the extended client work the spring cloud aws sqs.
Can anyone provide some tips please ?
Thanks in advance;

[tomazfernandes]

Hi @NightFox7, if anyone is interested in creating a prototype of an adapter to use the client I linked above, or a separate implementation, I'd be happy to take a look.

[vijayrcse]

We used SQS Extended client library to publish the message to the queue. We didn't use SQStemplate of spring cloud aws
However on the consumer side, we still use the SQSListener to consumer the message as string and process. If the message size is greater than 254kb, the message will have a link to the S3 bucket. Within the listener implementation we check for a specific header( it will be only available if the message is in greater than 254 kb ) and based on that we have conditional logic to retrieve the actual message from s3 reference.

[NightFox7]

Ok I see. Thank you @vijayrcse

[outcomes-josef-salyer]

@vijayrcse do you have an example of your workaround that you can share?

[vijayrcse]

hi @outcomes-josef-salyer , we referred only java-extended-client-lib. You will find example code snippets in AWS documentation

[agarcia-te]

Unfortunately it implements the blocking SqsClient, which is not supported by the framework.

@tomazfernandes I believe async support has been added to the amazon-sqs-java-extended-client-lib, is this what you mean? https://github.com/awslabs/amazon-sqs-java-extended-client-lib/releases/tag/2.1.0

https://github.com/awslabs/amazon-sqs-java-extended-client-lib/blob/07d988c424dea7e4e7d128b217182e1414310560/src/main/java/com/amazon/sqs/javamessaging/AmazonSQSExtendedAsyncClient.java#L78

[tomazfernandes]

Looks like it @agarcia-te!

I wonder what we we'd need to do here to support that though, maybe the user can simply declare such client as a bean and the framework would use it?

[jorgenota]

Hi! This feature would be very useful.

I guess we need at least be able to configure in the same application the ability to use S3 for large messages for some queues but NOT for others. I'm not sure if it would be a problem for users just to enable sending large messages for all or none, but it would give you more control doing it at a more fine grained level.