Skip to content

Commit 02d3428

Browse files
committed
Adjustments to doc; improve on FIFO explanation
1 parent bc46e7e commit 02d3428

File tree

1 file changed

+31
-17
lines changed

1 file changed

+31
-17
lines changed

queues/sqs.md

Lines changed: 31 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -27,10 +27,6 @@ sqs:
2727
# Default: empty
2828
region: us-west-1
2929

30-
# AWS session token.
31-
# Default: empty (optional; local testing only)
32-
session_token: test
33-
3430
# AWS SQS endpoint.
3531
# This parameter *is only required* if your SQS endpoint is *not* hosted on the AWS infrastructure.
3632
# Always leave this empty if your queue is on AWS SQS.
@@ -90,7 +86,6 @@ You may also use this option if your service *does* support dynamic IAM, but you
9086
sqs:
9187
key: ASIAIOSFODNN7EXAMPLE
9288
secret: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
93-
session_token: AQoDYXdzEJr... # optional; usually not required
9489
# If in a different region *or* if outside AWS, provide the correct region.
9590
region: eu-west-1
9691
```
@@ -166,13 +161,29 @@ jobs:
166161

167162
{% endcode %}
168163

164+
## FIFO Queues
165+
166+
AWS supports [FIFO](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fifo-queues.html)
167+
queues, which guarantee order of delivery. In order for RoadRunner to correctly handle this scenario, some
168+
configuration is required.
169+
170+
1. You should set `retain_failed_jobs` to `true` on the pipeline, so a failed job does not get re-queued at the back of
171+
the queue. This would break the FIFO order.
172+
2. You must have either `num_workers` set to `1` for the worker pool **or** `prefetch` set to `1` for the pipeline. If
173+
you have multiple workers and fetch multiple messages at a time (`prefetch > 1`), RoadRunner may not process the jobs in
174+
correct order.
175+
169176
## Configuration Options
170177

171178
### Prefetch
172179

173-
`prefetch` - Number of jobs to prefetch from the queue until ACK/NACK.
180+
`prefetch` - The number of jobs to request from SQS at a time. This sets the `MaxNumberOfMessages` parameter on the
181+
[`ReceiveMessage`](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html) call
182+
to the queue. Additionally, it limits the number of queues that may be retained on the priority queue in RoadRunner for
183+
the pipeline. If you set this to zero or omit it, the default value of 1 will be used.
174184

175-
Default: `10`
185+
Default: `1`
186+
Maximum: `10`
176187

177188
### Visibility Timeout
178189

@@ -182,8 +193,8 @@ job may run multiple times. If you do not provide a value for this parameter or
182193
attribute of the queue is used, which defaults to 30 seconds. For more information, see the documentation on
183194
[visibility timeouts](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html).
184195

185-
Max value is `43200` seconds (12 hours).
186196
Default: `0`
197+
Maximum: `43200` (12 hours)
187198

188199
### Error Visibility Timeout
189200

@@ -192,17 +203,18 @@ failed) jobs will have their visibility timeout changed to. Note that you cannot
192203
more than the maximum value in cases where it fails multiple times, so setting this value to the maximum is error-prone.
193204
This parameter is ignored if `retain_failed_jobs` is `false`, and it must be larger than zero to apply.
194205

195-
Max value is `43200` seconds (12 hours).
196206
Default: `0`
207+
Maximum: `43200` (12 hours)
197208

198209
### Retain Failed Jobs
199210

200-
`retain_failed_jobs` - If enabled, jobs will not be deleted and requeued if they fail. Instead, RoadRunner will
211+
`retain_failed_jobs` - If enabled, jobs will **not** be deleted and re-queued if they fail. Instead, RoadRunner will
201212
simply let them be processed again after `visibility_timeout` has passed. If you set `error_visibility_timeout` and
202213
enable this feature, RoadRunner will change the timeout of the job to the value of `error_visibility_timeout`. This lets
203214
you customize the timeout for errors specifically. If you enable this feature, you can configure SQS to automatically
204215
move jobs that fail multiple times to a
205216
[dead-letter queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html).
217+
Always enable this feature if you consume from a FIFO queue.
206218

207219
Default: `false`
208220

@@ -211,20 +223,22 @@ Default: `false`
211223
`wait_time_seconds` - The duration (in seconds) for which the call waits for a message to arrive in the queue before
212224
returning. If a message is available, the call returns sooner than `wait_time_seconds`. If no messages are available and
213225
the wait time expires, the call returns successfully with an empty list of messages. Please note that this parameter
214-
cannot be explicitly configured to use zero, as zero will apply the queue defaults.
226+
cannot be explicitly configured to use zero, as zero will apply the queue defaults. In most cases, you should set this
227+
to a non-zero value.
215228

216229
Default: `0`
230+
Maximum: `20`
217231

218232
{% hint style="warning" %}
219-
### Shor vs. Long Polling
233+
### Short vs. Long Polling
220234

221235
By default, SQS and RoadRunner is configured to use **short polling**. Please review the documentation on the
222236
differences between
223237
[short and long polling](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html)
224238
and make sure that your queue and RoadRunner is configured correctly. Long polling will *usually* be a cost-saving
225239
feature with no practical impact on performance or functionality. Remember that not providing a value (or zero) for
226-
`wait_time_seconds` will cause your queue polling to be based on the `ReceiveMessageWaitTimeSeconds` attribute
227-
configured on the queue.
240+
`wait_time_seconds` will cause your queue wait time to be based on the `ReceiveMessageWaitTimeSeconds` attribute
241+
configured on the queue, which also defaults to zero.
228242
{% endhint %}
229243

230244
### Queue
@@ -247,7 +261,7 @@ More info:
247261
[link](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html#SQS-SendMessage-request-MessageGroupId)
248262

249263
### Skip Queue Declaration
250-
"https://sqs.eu-central-1.amazonaws.com"
264+
251265
`skip_queue_declaration` - By default, RR tries to create the queue (using the `queue` name) if it does not exist. Set
252266
this option to `true` if the queue already exists.
253267

@@ -284,8 +298,8 @@ Attributes are only set if RoadRunner creates the queue. Attributes of existing
284298

285299
### Tags
286300

287-
`tags` - Tags don't have any semantic meaning. Amazon SQS interprets tags as character. Tags are only set if RR creates
288-
the queue. Existing queues are not modified.
301+
`tags` - Tags don't have any semantic meaning. Amazon SQS interprets tags as character. Tags are only set if RoadRunner
302+
creates the queue. Existing queues are **not** modified.
289303

290304
{% hint style="info" %}
291305
This functionality is rarely used and slows down queues. Please see

0 commit comments

Comments
 (0)