[AWS] Clarify the role of "Default AWS Region"

[zmoog]

The AWS package allows for collecting logs & metrics from a rich selection of AWS services.

All integrations in the package share the global settings (access/secret keys, role, region, etc) to access the AWS API using the AWS SDK.

The "Default AWS Region" is essential in one of these settings. Its behavior is documented in the package UI (see screenshot below), but it does not have a dedicated space in the AWS package docs.

The use of us-east-1 as the default region creates authentication issues that are hard to troubleshoot when users deploy the Elastic Agent on AWS partitions like aws-us-gov or aws-cn.

We need to:

• Review the current package information on the UI and docs.
• Identify the gaps we need to address.
• Make changes to improve the OOTB user experience, including users from aws-us-gov, aws-cn, and other existing partitions.

[zmoog]

I worked with an aws-us-gov user trying to ingest CloudTrail logs using S3+SQS. They were using an IAM role.

Without setting the proper value in the "Default AWS Region", Filebeat failed to assume the IAM role, even if I could perform the aws sts assume-role command with the AWS CLI.

I guess defaulting to us-east-1 is a Filebeat specific.

We need to make this work OOTB and/or make it obvious to users that they need to set the region.

[zmoog]

I set up a few resource in my AWS test account the reproduce tis scenario.

Setup

Resource	Region
Elastic Agent	eu-west-1
SQS queue	eu-north-1

SQS queue details

url: https://sqs.eu-north-1.amazonaws.com/123/mbranca-integrations-issues-12749

For authentication, I will use one the the frequently used setup:

• The EC2 uses the IAM role mbranca-elastic-agent-ec2-role with permission to assume the role. mbranca-elastic-agent .
• The IAM role mbranca-elastic-agent has the permission to receive SQS messages, download S3 objects, and so on.

The IAM role mbranca-elastic-agent-ec2-role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::123:role/mbranca-elastic-agent"
    }
  ]
}

The IAM role mbranca-elastic-agent-ec2-role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeRegions",
        "cloudwatch:GetMetricData",
        "cloudwatch:ListMetrics",
        "iam:ListAccountAliases",
        "rds:ListTagsForResource",
        "s3:GetObject",
        "sns:ListTopics",
        "sqs:ChangeMessageVisibility",
        "sqs:DeleteMessage",
        "sqs:ListQueues",
        "sqs:ReceiveMessage",
        "sts:AssumeRole",
        "sts:GetCallerIdentity",
        "tag:GetResources"
      ],
      "Resource": "*"
    }
  ]
}

Validation (CLI)

With this step, we'll log into the EC2 to validate the setup using the AWS CLI.

get-caller-identity

Check the EC2 assumed to role mbranca-elastic-agent-ec2-role for us:

$ aws sts get-caller-identity
{
    "UserId": "AROAZEDJODE3MRTNYEEYL:i-068d57696579bcac7",
    "Account": "627286350134",
    "Arn": "arn:aws:sts::123:assumed-role/mbranca-elastic-agent-ec2-role/i-068d57696579bcac7"
}

✅ The EC2 successfully assumed the IAM role mbranca-elastic-agent-ec2-role.

assume-role

Using the permissions from mbranca-elastic-agent-ec2-role, let's try to assume the IAM role mbranca-elastic-agent:

export ROLE_ARN="arn:aws:iam::627286350134:role/mbranca-elastic-agent"

credentials=$(aws sts assume-role --role-arn $ROLE_ARN --role-session-name my-session)

export AWS_ACCESS_KEY_ID=$(echo $credentials | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo $credentials | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo $credentials | jq -r '.Credentials.SessionToken')

echo $AWS_ACCESS_KEY_ID
ASIA*********

✅ We successfully assumed the IAM role mbranca-elastic-agent and set the AWS environment variables.

receive-message

aws sqs receive-message --queue-url https://sqs.eu-north-1.amazonaws.com/123/mbranca-integrations-issues-12749 --visibility-timeout 0
{
    "Messages": [
        {
            "MessageId": "25959b8a-9190-4e4d-b917-fec4b636c602",
            "ReceiptHandle": "whatever",
            "MD5OfBody": "46d61f942534210f9010f71eac266ca5",
            "Body": "👻 booooo"
        }
    ]
}

✅ We successfully received a SQS message from the queue. The setup is OK.

Agent

I created an Agent Policy named "#12749" with the following settings:

role_arn: arn:aws:iam::123:role/mbranca-elastic-agent
sqs_queue: https://sqs.eu-north-1.amazonaws.com/123/mbranca-integrations-issues-12749

And deployed and Elastic Agent 8.16.3 in the EC2 using Docker:

docker run \
    --rm \
    --name="elastic-agent-elastic-integrations-12749-8.16.3" \
    -e FLEET_ENROLL=true \
    -e FLEET_URL="https://[redacted].elastic-cloud.com:443" \
    -e FLEET_ENROLLMENT_TOKEN="[redacted]" \
    -i docker.elastic.co/beats/elastic-agent:8.16.3

I checked the logs and I found an error from the add_cloud_metadata processor:

[elastic_agent.filebeat][warn] error fetching cluster name metadata: error fetching EC2 Tags: operation error EC2: DescribeTags, https response error StatusCode: 403, RequestID: 27bfd052-5829-459c-ab24-5d8f126c1846, api error UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::123:assumed-role/mbranca-elastic-agent-ec2-role/i-068d57696579bcac7 is not authorized to perform: ec2:DescribeTags because no identity-based policy allows the ec2:DescribeTags action.

However, the input seems to start correctly, picking the AWS region from the SQS queue URL:

# However, the input was able to pick the AWS region from the SQS queue and start correctly:

23:09:37.378 elastic_agent.filebeat aws-s3-default [elastic_agent.filebeat][info] Input 'aws-s3' starting
23:09:37.378 elastic_agent.filebeat aws-s3-default [elastic_agent.filebeat][info] registering
23:09:37.378 elastic_agent.filebeat aws-s3-default [elastic_agent.filebeat][info] AWS api_timeout is set to 2m0s.
23:09:37.378 elastic_agent.filebeat aws-s3-default [elastic_agent.filebeat][info] AWS region is set to eu-north-1.
23:09:37.378 elastic_agent.filebeat aws-s3-default [elastic_agent.filebeat][info] AWS SQS visibility_timeout is set to 5m0s.
23:09:37.378 elastic_agent.filebeat aws-s3-default [elastic_agent.filebeat][info] AWS SQS number_of_workers is set to 5.

[jsevidal13]

Is there a way to perform a similar test in AWS GovCloud? I'd be happy to help if we I can login with a test account there. :)

[zmoog]

Is there a way to perform a similar test in AWS GovCloud? I'd be happy to help if we I can login with a test account there. :)

I have minimal access to GovCloud. I will try to see if I can and report back.