Support rabbit_peer_discovery_aws to work with instance metadata serv…

[thuandb]

…ice v2 (IMDSv2).

IMDSv2 uses session-oriented requests. With session-oriented requests, a session token is retrieved first
then used in subsequent GET requests for instance metadata values such as instance-id, credentials, etc.

Details could be found here https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html

Proposed Changes

The following change is an implementation to the support aws_peer_discover_aws plugin to work with EC2 instance metadata service version 2 (IMDSv2). IMDSv2 is a new version of the instance metadata service which adds defense in depth against open firewalls, reverse proxies, and SSRF vulnerabilities. With IMDSv2, every request submitted to the instance metadata service is now protected by session authentication. Specifically, querying IMDSv2 is done in 2 steps:

• Start a session with an HTTP PUT request to IMDSv2. IMDSv2 then returns a secret token.
• Use the secret token to make any HTTP GET request to retrieve interested metadata values such as availability zone, instance role, instance ID, credentials, etc.

This change is backward compatible. If for some reason, it fails to obtain a secret token from IMDSv2, rabbit_peer_discovery_aws plugin will fallback to use IMDSv1.

In addition, a new configuration flag aws_prefer_imdsv2 is also introduced to allow users explicitly enable or disable using IMDSv1. Per AWS security recommendation, IMDSv2 is preferable. Specifically, if the configuration flag a new configuration flag: aws_prefer_imdsv2 is not set or set to true, the rabbitmq_peer_discovery_aws plugin will attempt to retrieve a secret token first. If a secret token is received successfully, it will be used in subsequent requests submitted to the instance metadata service. On the other hand, if aws_prefer_imdsv2 is set to false, the rabbitmq_peer_discovery_aws plugin will use IMDSv1 for instance metadata queries.

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating
the PR. If you're unsure about any of them, don't hesitate to ask on the
mailing list. We're here to help! This is simply a reminder of what we are
going to look for before merging your code.

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• All tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have tested this change with AWS and with both IMDSv2 and failure scenario to fallback to IMDSv1 as well as explicitly turn off IMDSv2 preference.
• I have added necessary documentation (if appropriate)
• Any dependent changes have been merged and published in related repositories

Further Comments

IMDSv2 secret token retrieval result (both success and failure) is cached and reused. This is to guarantee that there is at most 1 HTTP PUT request is used to request IMDSv2 secret token per session which included multiple HTTP GET requests to query metadata values: instance-id, availability zone, instance role, and credentials.

[thuandb]

Submitted by accident. I will need to do some clean up and resubmit.

[michaelklishin]

Can we namespace this the same way other AWS peer discovery keys are namespaced? For example, cluster_formation.aws.prefer_imdsv2 instead of aws_prefer_imdsv2.

[thuandb]

Hi Micheal, thank you for your quick feedback.

I was thinking that this new aws_prefer_imdsv2 flag is used in the generic rabbitmq_aws application which the rabbit_peer_discovery_aws plugin relies on. It is independent from peer discovery plugins and/or cluster formation.

For example, in future there might be new plugins need interaction with AWS for other purposes such as publishing metrics/logs directly to CloudWatch, and It could be run from a single instance RabbitMQ fleets and/or clusters may not use the rabbit_peer_discovery_aws plugin.

Please feel free to suggest a better namespace for this new flag. And in case cluster_formation.aws.prefer_imdsv2 still better, please let me know.

P/S: I accidentally submitted this PR request while preparing a private one for being reviewed internally first, hence I have closed it. I will submit a new PR request once it is approved internally and will make sure your comments here is addressed.

[michaelklishin]

I would not object to aws.prefer_imdsv2 or amws.preferred_imds = v2 or similar. I have a slight preference towards the latter but recognize that many tools use boolean settings such as prefer X (e.g. the IPv6 vs. IPv4 preference on the JVM).

[thuandb]

Sure. I will change to aws.prefer_imdsv2 = true | false then. Thanks again.