Skip to content

Support rabbit_peer_discovery_aws to work with instance metadata serv… #2952

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Support rabbit_peer_discovery_aws to work with instance metadata serv… #2952

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions deps/rabbit/priv/schema/rabbit.schema
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1733,6 +1733,19 @@ end}.
end
}.


% ===============================
% AWS section
% ===============================

%% Whether or not to prefer IMDSv2 when querying instance metadata service (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html).
%% If not set or set to true, IMDSv2 will be preferred to use first. If fails, IMDSv1 will be used.
%% {aws_prefer_imdsv2, false}

{mapping, "aws_prefer_imdsv2", "rabbit.aws_prefer_imdsv2",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we namespace this the same way other AWS peer discovery keys are namespaced? For example, cluster_formation.aws.prefer_imdsv2 instead of aws_prefer_imdsv2.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Micheal, thank you for your quick feedback.

I was thinking that this new aws_prefer_imdsv2 flag is used in the generic rabbitmq_aws application which the rabbit_peer_discovery_aws plugin relies on. It is independent from peer discovery plugins and/or cluster formation.

For example, in future there might be new plugins need interaction with AWS for other purposes such as publishing metrics/logs directly to CloudWatch, and It could be run from a single instance RabbitMQ fleets and/or clusters may not use the rabbit_peer_discovery_aws plugin.

Please feel free to suggest a better namespace for this new flag. And in case cluster_formation.aws.prefer_imdsv2 still better, please let me know.

P/S: I accidentally submitted this PR request while preparing a private one for being reviewed internally first, hence I have closed it. I will submit a new PR request once it is approved internally and will make sure your comments here is addressed.

Copy link
Collaborator

@michaelklishin michaelklishin Apr 5, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not object to aws.prefer_imdsv2 or amws.preferred_imds = v2 or similar. I have a slight preference towards the latter but recognize that many tools use boolean settings such as prefer X (e.g. the IPv6 vs. IPv4 preference on the JVM).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I will change to aws.prefer_imdsv2 = true | false then. Thanks again.

[{datatype, {enum, [true, false]}}]}.


% ===============================
% Validators
% ===============================
Expand Down
2 changes: 2 additions & 0 deletions deps/rabbitmq_aws/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ configuration or to impact configuration behavior:
``rabbitmq_aws:set_region/1`` | Manually specify the AWS region to make requests to.
``rabbitmq_aws:set_credentials/2`` | Manually specify the request credentials to use.
``rabbitmq_aws:refresh_credentials/0`` | Refresh the credentials from the environment, filesystem, or EC2 Instance Metadata service.
``rabbitmq_aws:ensure_imdsv2_token_valid/0`` | Make sure IMDSv2 token is acctive and valid.
``rabbitmq_aws:api_get_request/2`` | Perform an AWS service API request.
``rabbitmq_aws:get/2`` | Perform a GET request to the API specifying the service and request path.
``rabbitmq_aws:get/3`` | Perform a GET request specifying the service, path, and headers.
``rabbitmq_aws:post/4`` | Perform a POST request specifying the service, path, headers, and body.
Expand Down
18 changes: 18 additions & 0 deletions deps/rabbitmq_aws/include/rabbitmq_aws.hrl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,18 @@

-define(INSTANCE_CREDENTIALS, "iam/security-credentials").
-define(INSTANCE_METADATA_BASE, "latest/meta-data").
-define(INSTANCE_ID, "instance-id").

-define(TOKEN_URL, "latest/api/token").

-define(METADATA_TOKEN_TLL_HEADER, "X-aws-ec2-metadata-token-ttl-seconds").

% AWS IMDSv2 is session-based and instance metadata service requests which are only needed for loading/refreshing credentials.
% We dont need to have long-live metadata token. In fact, we only need the token is valid for a sufficient period to successfully
% load/refresh credentials. 60 seconds is more than enough for that goal.
-define(METADATA_TOKEN_TLL_SECONDS, 60).

-define(METADATA_TOKEN, "X-aws-ec2-metadata-token").

-type access_key() :: nonempty_string().
-type secret_access_key() :: nonempty_string().
Expand All @@ -41,11 +53,17 @@
-type sc_error() :: {error, Reason :: atom()}.
-type security_credentials() :: sc_ok() | sc_error().

-record(imdsv2token, { token :: security_token() | undefined,
expiration :: expiration() | undefined}).

-type imdsv2token() :: #imdsv2token{}.

-record(state, {access_key :: access_key() | undefined,
secret_access_key :: secret_access_key() | undefined,
expiration :: expiration() | undefined,
security_token :: security_token() | undefined,
region :: region() | undefined,
imdsv2_token:: imdsv2token() | undefined,
error :: atom() | string() | undefined}).
-type state() :: #state{}.

Expand Down
134 changes: 106 additions & 28 deletions deps/rabbitmq_aws/src/rabbitmq_aws.erl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@
request/5, request/6, request/7,
set_credentials/2,
has_credentials/0,
set_region/1]).
set_region/1,
ensure_imdsv2_token_valid/0,
api_get_request/2]).

%% gen-server exports
-export([start_link/0,
Expand Down Expand Up @@ -77,6 +79,14 @@ post(Service, Path, Body, Headers) ->
refresh_credentials() ->
gen_server:call(rabbitmq_aws, refresh_credentials).

-spec refresh_credentials(state()) -> ok | error.
%% @doc Manually refresh the credentials from the environment, filesystem or EC2
%% Instance metadata service.
%% @end
refresh_credentials(State) ->
rabbit_log:debug("Refreshing AWS credentials..."),
{_, NewState} = load_credentials(State),
set_credentials(NewState).

-spec request(Service :: string(),
Method :: method(),
Expand Down Expand Up @@ -120,6 +130,9 @@ request(Service, Method, Path, Body, Headers, HTTPOptions) ->
request(Service, Method, Path, Body, Headers, HTTPOptions, Endpoint) ->
gen_server:call(rabbitmq_aws, {request, Service, Method, Headers, Path, Body, HTTPOptions, Endpoint}).

-spec set_credentials(state()) -> ok.
set_credentials(NewState) ->
gen_server:call(rabbitmq_aws, {set_credentials, NewState}).

-spec set_credentials(access_key(), secret_access_key()) -> ok.
%% @doc Manually set the access credentials for requests. This should
Expand All @@ -137,6 +150,20 @@ set_credentials(AccessKey, SecretAccessKey) ->
set_region(Region) ->
gen_server:call(rabbitmq_aws, {set_region, Region}).

-spec set_imdsv2_token(imdsv2token()) -> ok.
%% @doc Manually set the Imdsv2Token to perform instance metadata service requests.
%% @end
set_imdsv2_token(Imdsv2Token) ->
gen_server:call(rabbitmq_aws, {set_imdsv2_token, Imdsv2Token}).


-spec get_imdsv2_token() -> imdsv2token().
%% @doc return the current Imdsv2Token to perform instance metadata service requests.
%% @end
get_imdsv2_token() ->
{ok, Imdsv2Token}=gen_server:call(rabbitmq_aws, get_imdsv2_token),
Imdsv2Token.


%%====================================================================
%% gen_server functions
Expand All @@ -158,15 +185,8 @@ terminate(_, _) ->
code_change(_, _, State) ->
{ok, State}.


handle_call(Msg, _From, #state{region = undefined}) ->
%% Delay initialisation until a RabbitMQ plugin require the AWS backend
{ok, Region} = rabbitmq_aws_config:region(),
{_, State} = load_credentials(#state{region = Region}),
handle_msg(Msg, State);
handle_call(Msg, _From, State) ->
handle_msg(Msg, State).

handle_msg(Msg, State).

handle_cast(_Request, State) ->
{noreply, State}.
Expand Down Expand Up @@ -196,15 +216,29 @@ handle_msg({set_credentials, AccessKey, SecretAccessKey}, State) ->
expiration = undefined,
error = undefined}};

handle_msg({set_credentials, NewState}, State) ->
{reply, ok, State#state{access_key = NewState#state.access_key,
secret_access_key = NewState#state.secret_access_key,
security_token = NewState#state.security_token,
expiration = NewState#state.expiration,
error = NewState#state.error}};

handle_msg({set_region, Region}, State) ->
{reply, ok, State#state{region = Region}};

handle_msg({set_imdsv2_token, Imdsv2Token}, State) ->
{reply, ok, State#state{imdsv2_token = Imdsv2Token}};

handle_msg(has_credentials, State) ->
{reply, has_credentials(State), State};

handle_msg(get_imdsv2_token, State) ->
{reply, {ok, State#state.imdsv2_token}, State};

handle_msg(_Request, State) ->
{noreply, State}.


-spec endpoint(State :: state(), Host :: string(),
Service :: string(), Path :: string()) -> string().
%% @doc Return the endpoint URL, either by constructing it with the service
Expand Down Expand Up @@ -296,15 +330,17 @@ load_credentials(#state{region = Region}) ->
access_key = AccessKey,
secret_access_key = SecretAccessKey,
expiration = Expiration,
security_token = SecurityToken}};
security_token = SecurityToken,
imdsv2_token = undefined}};
{error, Reason} ->
error_logger:error_msg("Could not load AWS credentials from environment variables, AWS_CONFIG_FILE, AWS_SHARED_CREDENTIALS_FILE or EC2 metadata endpoint: ~p. Will depend on config settings to be set.~n.", [Reason]),
{error, #state{region = Region,
error = Reason,
access_key = undefined,
secret_access_key = undefined,
expiration = undefined,
security_token = undefined}}
security_token = undefined,
imdsv2_token = undefined}}
end.


Expand Down Expand Up @@ -377,23 +413,8 @@ perform_request_has_creds(false, State, _, _, _, _, _, _, _) ->
%% @end
perform_request_creds_expired(false, State, Service, Method, Headers, Path, Body, Options, Host) ->
perform_request_with_creds(State, Service, Method, Headers, Path, Body, Options, Host);
perform_request_creds_expired(true, State, Service, Method, Headers, Path, Body, Options, Host) ->
perform_request_creds_refreshed(load_credentials(State), Service, Method, Headers, Path, Body, Options, Host).


-spec perform_request_creds_refreshed({ok, State :: state()} | {error, State :: state()},
Service :: string(), Method :: method(),
Headers :: headers(), Path :: path(), Body :: body(),
Options :: http_options(), Host :: string() | undefined)
-> {Result :: result(), NewState :: state()}.
%% @doc If it's been determined that there are credentials but they have expired,
%% check to see if the credentials could be loaded and either make the request
%% or return an error.
%% @end
perform_request_creds_refreshed({ok, State}, Service, Method, Headers, Path, Body, Options, Host) ->
perform_request_with_creds(State, Service, Method, Headers, Path, Body, Options, Host);
perform_request_creds_refreshed({error, State}, _, _, _, _, _, _, _) ->
perform_request_creds_error(State).
perform_request_creds_expired(true, State, _, _, _, _, _, _, _) ->
perform_request_creds_error(State#state{error = "Credentials expired!"}).


-spec perform_request_with_creds(State :: state(), Service :: string(), Method :: method(),
Expand Down Expand Up @@ -470,3 +491,60 @@ sign_headers(#state{access_key = AccessKey,
uri = URI,
headers = Headers,
body = Body}).

-spec expired_imdsv2_token(imdsv2token()) -> boolean().
%% @doc Determine whether an Imdsv2Token has expired or not.
%% @end
expired_imdsv2_token(Imdsv2Token) ->
case Imdsv2Token of
undefined -> rabbit_log:debug("AWS Imdsv2 token has not been obtained yet."),
true;
{_ ,_, undefined} -> rabbit_log:debug("AWS Imdsv2 token has expired."),
true;
{_, _, Expiration} -> Now = calendar:datetime_to_gregorian_seconds(local_time()),
Now >= Expiration
end.

-spec ensure_imdsv2_token_valid() -> imdsv2token().
ensure_imdsv2_token_valid() ->
Imdsv2Token=get_imdsv2_token(),
case expired_imdsv2_token(Imdsv2Token) of
true -> Value=rabbitmq_aws_config:load_imdsv2_token(),
Expiration=calendar:datetime_to_gregorian_seconds(local_time()) + ?METADATA_TOKEN_TLL_SECONDS,
set_imdsv2_token(#imdsv2token{token = Value,
expiration = Expiration}),
Value;
_ -> rabbit_log:debug("Imdsv2 token is still valid."),
Imdsv2Token#imdsv2token.token
end.

-spec ensure_credentials_valid() -> ok.
%% @doc Invoked before each AWS service API request checking to see if credentials available
%% or whether the current credentials have expired.
%% If they haven't, move on performing the request, otherwise try and refresh the
%% credentials before performing the request.
%% @end
ensure_credentials_valid() ->
rabbit_log:debug("Making sure AWS credentials is available and still valid."),
{ok, State}=gen_server:call(rabbitmq_aws, get_state),
case has_credentials(State) of
true -> case expired_credentials(State#state.expiration) of
true -> refresh_credentials(State);
_ -> ok
end;
_ -> refresh_credentials(State)
end.


-spec api_get_request(string(), path()) -> result().
%% @doc Invoke an API call to an AWS service.
%% @end
api_get_request(Service, Path) ->
rabbit_log:debug("Invoking AWS request {Service: ~p; Path: ~p}...", [Service, Path]),
ensure_credentials_valid(),
case get(Service, Path) of
{ok, {_Headers, Payload}} -> rabbit_log:debug("AWS request: ~s~nResponse: ~p", [Path, Payload]),
{ok, Payload};
{error, {credentials, _}} -> {error, credentials};
{error, Message, _} -> {error, Message}
end.
Loading