Support the EC2 Metadata Server

[sjperkins]

TODO

• Support expiration of credentials. EC2 Metadata server returns access key, secret key and a session token that expires.
• Handle more failure cases in the Mock

[sjperkins]

It might be simplest to add an expiration date to the AwsCredentials struct, default-initialised toabsl::InfiniteFuture.

/// Holds S3 credentials
///
/// Contains the access key, secret key and session token.
/// An empty access key implies anonymous access,
/// while the presence of a session token implies the use of
/// short-lived STS credentials
/// https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html
struct AwsCredentials {
  /// AWS_ACCESS_KEY_ID
  std::string access_key;
  /// AWS_SECRET_KEY_ID
  std::string secret_key;
  /// AWS_SESSION_TOKEN
  std::string session_token;
  bool IsAnonymous() const { return access_key.empty(); }
};

[sjperkins]

Some of the reference url's were useful for identifying edge cases, but may provide too much detail.

[laramiel]

include any directly named types. I notice at least, in this file:

string
absl/time
absl/strings/str_cat
tensorstore/json_binding

[sjperkins]

Done, I think the existing tensorstore/internal/json_binding/* imports covers the last case.

[laramiel]

There are a lot more includes missing.

Look at the using declarations, for example. Anything that you spell out, such as 'tensorstore::Result' needs an import in each file where it is spelled (include-what-you-use); we don't rely on transitive includes for any of those cases. We only permit transitive includes for non-spelled field access and auto type use.

json_binding is different, though. Those includes are needed to satisfy the need for type-specializations to be visible where they are used, such as bindings for absl::Time.

[sjperkins]

Ah, thanks for pointing this out. I've gone through the file on a IWYU basis and adjusted the #includes accordingly.

[sjperkins]

I suspect that, except for Code, all of these should be optional to encourage success in json parsing, when Code != "Success".

A sample from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials

{
  "Code" : "Success",
  "LastUpdated" : "2012-04-26T16:39:16Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "ASIAIOSFODNN7EXAMPLE",
  "SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "Token" : "token",
  "Expiration" : "2017-05-17T15:09:54Z"
}

I'll try find the schema.

[sjperkins]

I've made all members optional except except Code and added a test case for the Code == SomethingOtherThanSuccess case.

[laramiel]

It seems like we may want to convert this from Result<AwsCredentials> GetCredentials() to Future<AwsCredentials> GetCredentials().

[sjperkins]

Could do so. I guess this would imply returning a Future in the AwsCredentialProvider interface too?

[laramiel]

Those are separate issues, but perhaps. Also, defer that work if you want until we get this done.

[sjperkins]

It seems like we may want to convert this from Result<AwsCredentials> GetCredentials() to Future<AwsCredentials> GetCredentials().

I guess this so that the GetCredentials here can be converted into chained futures so as to not block the thread on which futures are submitted?

[laramiel]

Another include-order detail. If there is a .h file which directly correlates with the .cc file (a header with the declarations and the same name), then we include it before anything else.

[laramiel]

A note on ordering here. I think that doing finishing this prior to #119 is the way to go, actually. Then I think that it might better indicate if the refactoring there pulls it's weight--I suspect that some of it isn't very useful.

[laramiel]

I'm preparing a submit for this. I've added the following in aws_credential_provider.h

class EC2MetadataCredentialProvider : public AwsCredentialProvider {
 public:
   /// ... public members ...
 private:
  std::shared_ptr<internal_http::HttpTransport> transport_;
  absl::Mutex mutex_;
  AwsCredentials credentials_ ABSL_GUARDED_BY(mutex_);
  absl::Time timeout_ ABSL_GUARDED_BY(mutex_) = absl::InfinitePast();
};

// Returns whether the EC2 Metadata Server is available.
bool IsEC2MetadataServiceAvailable(internal_http::HttpTransport& transport);

And in aws_credential_provider.cc:

// Obtain a metadata token for communicating with the api server.
Result<absl::Cord> GetEC2ApiToken(internal_http::HttpTransport& transport) {
  auto token_request = HttpRequestBuilder("POST", kTokenUrl)
                           .AddHeader(kTokenTtlHeader)
                           .BuildRequest();

  TENSORSTORE_ASSIGN_OR_RETURN(
      auto token_response,
      transport
          .IssueRequest(token_request, {}, absl::Seconds(1), kConnectTimeout)
          .result());

  TENSORSTORE_RETURN_IF_ERROR(HttpResponseCodeToStatus(token_response));
  return std::move(token_response.payload);
}

}  // namespace

// Returns whether the EC2 Metadata Server is available.
bool IsEC2MetadataServiceAvailable(internal_http::HttpTransport& transport) {
  return GetEC2ApiToken(transport).ok();
}

// ... comment ...
Result<AwsCredentials> EC2MetadataCredentialProvider::GetCredentials() {
  absl::MutexLock l(&mutex_);
  if (absl::Now() + absl::Seconds(60) < timeout_) {
    return credentials_;
  }
  auto default_timeout = absl::Now() + kDefaultTimeout;

  // Obtain an API token for communicating with the EC2 Metadata server
  TENSORSTORE_ASSIGN_OR_RETURN(auto api_token, GetEC2ApiToken(*transport_));

[sjperkins]

9ec5bc2 might be useful as it follows the go logic. It also handles the case of .../iam/security-credentials returning multiple iam roles separated by newline -- the first one is chosen.

[laramiel]

Ok, now you can rebase and move the ec2 metadata server work to #119

[sjperkins]

Ok, now you can rebase and move the ec2 metadata server work to #119

Thanks. Will take a look.