Skip to content

Improve SQS docs #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Improve SQS docs #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .gitbook.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
root: ./

​structure:
structure:
readme: README.md
summary: SUMMARY.md​
summary: SUMMARY.md
180 changes: 119 additions & 61 deletions queues/sqs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,72 +1,115 @@
# SQS Driver

[Amazon SQS Simple Queue Service](https://aws.amazon.com/sqs/) is an alternative
queue server also developed by Amazon and is also part of the AWS
service infrastructure. If you prefer to use the "cloud" option, you can use the
[prebuilt documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configuring.html)
for its installation.

In addition to the ability to use this queue server within AWS, you can also use the
can also use the local installation of this system on your own servers. If you prefer
this option, you can use the [softwaremill's implementation](https://github.com/softwaremill/elasticmq)
of the Amazon SQS server.

After you have created the SQS server, you need to specify the following
connection settings in the `sqs` configuration settings. Unlike AMQP and Beanstalk,
SQS requires more values to set up a connection and will be different from what we are used to.
we're used to.
[Amazon Simple Queue Service (SQS)](https://aws.amazon.com/sqs/) is an alternative
queue server developed by Amazon, provided as part of the AWS Cloud infrastructure. You can find the documentation
[here](https://docs.aws.amazon.com/sqs/).

While this service is usually deployed on AWS, you can also self-host any compatible version, such as the
community-developed [ElasticMQ](https://github.com/softwaremill/elasticmq).

## Configuration

Specify the following connection settings in the `sqs` configuration settings.

{% code title=".rr.yaml" %}

```yaml
sqs:
# Required AccessKey ID.
# AWS AccessKey ID.
# Default: empty
key: access-key

# Required secret access key.
# AWS Access Key Secret.
# Default: empty
secret: api-secret

# Required AWS region.
# AWS region.
# Default: empty
region: us-west-1

# Required AWS session token.
# Default: empty
# AWS session token.
# Default: empty (optional; local testing only)
session_token: test

# Required AWS SQS endpoint to connect.
# Default: http://127.0.0.1:9324
endpoint: http://127.0.0.1:9324
# AWS SQS endpoint.
# This parameter *is only required* if your SQS endpoint is *not* hosted on the AWS infrastructure.
# Always leave this empty if your queue is on AWS SQS.
# For ElasticMQ, the default is http://127.0.0.1:9324
# Default: empty
endpoint: ""
```
{% endcode %}

### Examples

Because the options can be mixed in ways that don't make sense, here are a few examples of how to use the driver
in various scenarios. In all examples, the **queue name** is provided in the `jobs.pipelines.pipeline_name.config.queue`
property below.

#### AWS SQS with dynamic IAM in same region

If you access your AWS SQS queue from inside AWS via a service that supports dynamic IAM roles (such as EC2), and the
queue is in the same region.

{% code title=".rr.yaml" %}
```yaml
sqs:
# No parameters are required. Normally, if your IAM role has the required permissions and your service endpoint
# is in the same region, the environment variables associated with your instance/task will resolve all the
# configuration for you. You must provide the parent sqs key to enable SQS, even if you want to use the defaults.
```
{% endcode %}

{% hint style="info" %}
Please note that although each of the sections contains default values, it is marked as "required". This means that in
almost all cases they are required to be specified in order to correctly configure the driver.
{% endhint %}
#### AWS SQS in different region with dynamic IAM

After you have configured the connection - you should configure the queue that will use this connection:
If you access your AWS SQS queue from inside AWS via a service that supports dynamic IAM roles (such as EC2), and the
queue is in a different region.

{% hint style="info" %}
You may also skip the whole `sqs` configuration section (global, not the pipeline) to use the AWS IAM credentials if
the RR is inside the EC2 machine. RR will try to detect env automatically by making a http request to
the `http://169.254.169.254/latest/dynamic/instance-identity/` as
pointer [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html).
However, if you want to use credentials from the `sqs` section, you might specify them and IAM credentials will be overridden.
{% endhint %}
{% code title=".rr.yaml" %}
```yaml
sqs:
# Only region is required
region: eu-central-1
```
{% endcode %}

#### AWS SQS without dynamic IAM

If you access your AWS SQS queue from outside AWS, or from a service that does not support IAM, you must provide
an access key and secret. If you access the service from outside AWS, you must also provide `region`.
You may also use this option if your service *does* support dynamic IAM, but you want to use explicit access keys.

{% code title=".rr.yaml" %}
```yaml
sqs:
key: ASIAIOSFODNN7EXAMPLE
secret: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
session_token: AQoDYXdzEJr... # optional; usually not required
# If in a different region *or* if outside AWS, provide the correct region.
region: eu-west-1
```
{% endcode %}

#### Self-hosted (ElasticMQ)

If you self-host your SQS-compatible queue, only the endpoint is required.

{% code title=".rr.yaml" %}
```yaml
sqs:
endpoint: http://127.0.0.1:9324
```
{% endcode %}

After you have configured the driver, you should configure the queue that will use this connection.

{% code title=".rr.yaml" %}

```yaml
version: "3"

sqs:
# SQS connection configuration...
# SQS connection configuration. See above example.

jobs:
pipelines:
Expand Down Expand Up @@ -118,53 +161,67 @@ jobs:

{% endcode %}

## Configuration options

**Here is a detailed description of each of the SQS-specific options:**
## Configuration Options

### Prefetch

`prefetch` - Number of jobs to prefetch from the SQS until ACK/NACK.
Default: `10`.
`prefetch` - Number of jobs to prefetch from the queue until ACK/NACK.

Default: `10`

### Visibility timeout
### Visibility Timeout

`visibility_timeout` - The duration (in seconds) that the received messages are hidden from subsequent retrieve requests
after being retrieved by a `ReceiveMessage` request. Max value is `43200` seconds (12 hours). Default: `0`.
after being retrieved by a `ReceiveMessage` request. Max value is `43200` seconds (12 hours).

### Wait time seconds
Default: `0`

### Wait Time

`wait_time_seconds` - The duration (in seconds) for which the call waits for a message to arrive in the queue before
returning. If a message is available, the call returns sooner than WaitTimeSeconds. If no messages are available and the
wait time expires, the call returns successfully with an empty list of messages.
returning. If a message is available, the call returns sooner than `wait_time_seconds`. If no messages are available and
the wait time expires, the call returns successfully with an empty list of messages.

Default: `5`.
Default: `5`

### Queue

`queue` - SQS internal queue name. Can contain alphanumeric characters, hyphens (`-`), and underscores (`_`).
`queue` - The name of the queue. May only contain alphanumeric characters, hyphens (`-`), and underscores (`_`). On AWS,
this will form the last part of the queue URL, i.e. `https://sqs.<region>.amazonaws.com/<account id>/<queue>`.

Default value is `default` string.
Default: `default`

### Message Group ID

`message_group_id` - Message group ID is required for FIFO queues. Messages that belong to the same message group are processed in a FIFO manner.
`message_group_id` - Message group ID is required for FIFO queues. Messages that belong to the same message group are
processed in a FIFO manner.
More info: [link](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html#SQS-SendMessage-request-MessageGroupId)

### Skip queue declaration
### Skip Queue Declaration

`skip_queue_declaration` - By default, RR tries to create the queue (using the `queue` name) if it does not exist. Set
this option to `true` if the queue already exists.

Default: `false`

{% hint style="info" %}
For queue creation to work, the credentials or the IAM role used must have the
[required permissions](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-api-permissions-reference.html),
such as `sqs:CreateQueue` and `sqs:SetQueueAttributes`.
{% endhint %}

`skip_queue_declaration` - By default, RR tries to declare the queue by default and then gets the queue URL. Set this
option to `true` if the user already declared the queue to only get its URL.
### Consume All

### Consume all
`consume_all` - By default, RR consumes only messages in the queue with the `Jobs` structure. Set this option to `true`
if you want to consume all messages in the queue.

`consume_all` - By default, RR supports only `Jobs` structures from the queue. Set this option to true if you want to
also consume the raw payloads.
Default: `false`

### Attributes

`attributes` - List of
the [AWS SQS attributes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SetQueueAttributes.html).
`attributes` - A list of
[AWS SQS attributes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SetQueueAttributes.html)
to configure for the queue. Attributes are only set if RR creates the queue. Existing queues are not modified.

{% code title=".rr.yaml" %}

Expand All @@ -181,9 +238,10 @@ attributes:

### Tags

`tags` - Tags don't have any semantic meaning. Amazon SQS interprets tags as character.
`tags` - Tags don't have any semantic meaning. Amazon SQS interprets tags as character. Tags are only set if RR creates
the queue. Existing queues are not modified.

{% hint style="info" %}
This functionality is rarely used and slows down the work of
queues: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-queue-tags.html
This functionality is rarely used and slows down queues. Please see
[this link](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-queue-tags.html).
{% endhint %}