[Python][C++] Add Profile support to S3FileSystem

[yaxxie-piramidal]

Describe the enhancement requested

pyarrow.fs.S3FileSystem works fine if you're willing to pull temporary credentials and create this client manually

This comes with inconvenience such as having to recreate this client periodically as the temporary credentials expire.

In particular it is something that is necessary to do when set up using multiple profiles, as currently there is no way to select a profile and use the underlying SDK machinery to use/refresh the credentials given the profile.

Please add support for this

Component(s)

Python

[raulcd]

Thanks for opening the issue. Is this related to the following?

• [C++] S3FileSystem enable automatic temporary credential refreshing for AWS Instance Profile #18888

If not, could you add details of AWS documentation for this specific underlying SDK and the feature. This documentation will be a necessary resource if someone can work on this in the future. I am not that familiar with all the AWS S3 authentication methods.

[yaxxie-piramidal]

Thanks for adding this; that issue is certainly related, but not the same. I'm asking for a "profile" argument to be exposed.

When you do aws configure you can have multiple named profiles set up, and you can swap between them by supplying the profile argument

What I was doing was assuming that pyarrow already taps into the refreshing credentials machinery available (which from that issue it appears not to be the case), and suggesting that we can expose the profile parameter and get to tap into that machinery.

Would this resource be enough?

[raulcd]

Yes, from the looks of cpp/src/arrow/filesystem/s3fs.cc it does seem that the ProfileConfigFileAWSCredentialsProvider is not implemented as a possible S3Options.credentials_provider:

arrow/cpp/src/arrow/filesystem/s3fs.cc

Lines 237 to 273 in 4592486

	void S3Options::ConfigureDefaultCredentials() {
	credentials_provider =
	std::make_shared<Aws::Auth::DefaultAWSCredentialsProviderChain>();
	credentials_kind = S3CredentialsKind::Default;
	}
	void S3Options::ConfigureAnonymousCredentials() {
	credentials_provider = std::make_shared<Aws::Auth::AnonymousAWSCredentialsProvider>();
	credentials_kind = S3CredentialsKind::Anonymous;
	}
	void S3Options::ConfigureAccessKey(const std::string& access_key,
	const std::string& secret_key,
	const std::string& session_token) {
	credentials_provider = std::make_shared<Aws::Auth::SimpleAWSCredentialsProvider>(
	ToAwsString(access_key), ToAwsString(secret_key), ToAwsString(session_token));
	credentials_kind = S3CredentialsKind::Explicit;
	}
	void S3Options::ConfigureAssumeRoleCredentials(
	const std::string& role_arn, const std::string& session_name,
	const std::string& external_id, int load_frequency,
	const std::shared_ptr<Aws::STS::STSClient>& stsClient) {
	credentials_provider = std::make_shared<Aws::Auth::STSAssumeRoleCredentialsProvider>(
	ToAwsString(role_arn), ToAwsString(session_name), ToAwsString(external_id),
	load_frequency, stsClient);
	credentials_kind = S3CredentialsKind::Role;
	}
	void S3Options::ConfigureAssumeRoleWithWebIdentityCredentials() {
	// The AWS SDK uses environment variables AWS_DEFAULT_REGION,
	// AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_SESSION_NAME
	// to configure the required credentials
	credentials_provider =
	std::make_shared<Aws::Auth::STSAssumeRoleWebIdentityCredentialsProvider>();
	credentials_kind = S3CredentialsKind::WebIdentity;
	}