Skip to content

[WIP] Remove repositories.s3 settings and deprecated settings #23276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[WIP] Remove repositories.s3 settings and deprecated settings #23276

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
60f2d0e
WIP Remove deprecated settings and document
dadoonet Jan 30, 2017
49f087d
Disable test for now
dadoonet Jan 30, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions docs/reference/migration/migrate_6_0/plugins.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,30 @@ region inside the repository settings. Instead, specify the full endpoint if a c
s3 location is needed, or rely on the default behavior which automatically locates
the region of the configured bucket.

* Global repositories settings you were able to set in elasticsearch config file under `repositories.s3`
name space have been removed. This includes `repositories.s3.bucket`, `repositories.s3.server_side_encryption`,
`repositories.s3.buffer_size`, `repositories.s3.max_retries`, `repositories.s3.use_throttle_retries`,
`repositories.s3.chunk_size`, `repositories.s3.compress`, `repositories.s3.storage_class`, `repositories.s3.canned_acl`,
`repositories.s3.base_path` and `repositories.s3.path_style_access`.
You must set those settings per repository instead. Respectively `bucket`, `server_side_encryption`, `buffer_size`,
`max_retries`, `use_throttle_retries`, `chunk_size`, `compress`, `storage_class`, `canned_acl`, `base_path` and
`path_style_access`.
See {plugins}/repository-s3-repository.html[S3 Repository settings].

* S3 plugin used now named configurations. You will basically define all the S3 clients you want to use by naming
them under `s3.client.xxx` prefix where `xxx` is your named configuration (use `default` as the name if you want to have
one applied by default).

As a consequence, the following settings have been removed: `cloud.aws.access_key`, `cloud.aws.secret_key`,
`cloud.aws.protocol`, `cloud.aws.proxy.host`, `cloud.aws.proxy.port`, `cloud.aws.proxy.username`,
`cloud.aws.proxy.password`, `cloud.aws.signer`, `cloud.aws.read_timeout`.
Also their S3 specific equivalent have been removed: `cloud.aws.s3.access_key`, `cloud.aws.s3.secret_key`,
`cloud.aws.s3.protocol`, `cloud.aws.s3.proxy.host`, `cloud.aws.s3.proxy.port`, `cloud.aws.s3.proxy.username`,
`cloud.aws.s3.proxy.password`, `cloud.aws.s3.signer`, `cloud.aws.s3.read_timeout`.
Also `repositories.s3.access_key`, `repositories.s3.secret_key`, `repositories.s3.endpoint` and
`repositories.s3.protocol`.


==== Azure Repository plugin

* The container an azure repository is configured with will no longer be created automatically.
Expand Down
135 changes: 0 additions & 135 deletions plugins/repository-s3/src/main/java/org/elasticsearch/cloud/aws/AwsS3Service.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,145 +19,10 @@

package org.elasticsearch.cloud.aws;

import java.util.Locale;
import java.util.function.Function;

import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.services.s3.AmazonS3;
import org.elasticsearch.common.component.LifecycleComponent;
import org.elasticsearch.common.settings.SecureString;
import org.elasticsearch.common.settings.Setting;
import org.elasticsearch.common.settings.Setting.Property;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.unit.TimeValue;

public interface AwsS3Service extends LifecycleComponent {

// Legacy global AWS settings (shared between discovery-ec2 and repository-s3)
// Each setting starting with `cloud.aws` also exists in discovery-ec2 project. Don't forget to update
// the code there if you change anything here.
/**
* cloud.aws.access_key: AWS Access key. Shared with discovery-ec2 plugin
*/
Setting<SecureString> KEY_SETTING = new Setting<>("cloud.aws.access_key", "", SecureString::new,
Property.NodeScope, Property.Filtered, Property.Deprecated, Property.Shared);
/**
* cloud.aws.secret_key: AWS Secret key. Shared with discovery-ec2 plugin
*/
Setting<SecureString> SECRET_SETTING = new Setting<>("cloud.aws.secret_key", "", SecureString::new,
Property.NodeScope, Property.Filtered, Property.Deprecated, Property.Shared);
/**
* cloud.aws.protocol: Protocol for AWS API: http or https. Defaults to https. Shared with discovery-ec2 plugin
*/
Setting<Protocol> PROTOCOL_SETTING = new Setting<>("cloud.aws.protocol", "https",
s -> Protocol.valueOf(s.toUpperCase(Locale.ROOT)), Property.NodeScope, Property.Deprecated, Property.Shared);
/**
* cloud.aws.proxy.host: In case of proxy, define its hostname/IP. Shared with discovery-ec2 plugin
*/
Setting<String> PROXY_HOST_SETTING = Setting.simpleString("cloud.aws.proxy.host",
Property.NodeScope, Property.Deprecated, Property.Shared);
/**
* cloud.aws.proxy.port: In case of proxy, define its port. Defaults to 80. Shared with discovery-ec2 plugin
*/
Setting<Integer> PROXY_PORT_SETTING = Setting.intSetting("cloud.aws.proxy.port", 80, 0, 1<<16,
Property.NodeScope, Property.Deprecated, Property.Shared);
/**
* cloud.aws.proxy.username: In case of proxy with auth, define the username. Shared with discovery-ec2 plugin
*/
Setting<SecureString> PROXY_USERNAME_SETTING = new Setting<>("cloud.aws.proxy.username", "", SecureString::new,
Property.NodeScope, Property.Deprecated, Property.Shared);
/**
* cloud.aws.proxy.password: In case of proxy with auth, define the password. Shared with discovery-ec2 plugin
*/
Setting<SecureString> PROXY_PASSWORD_SETTING = new Setting<>("cloud.aws.proxy.password", "", SecureString::new,
Property.NodeScope, Property.Filtered, Property.Deprecated, Property.Shared);
/**
* cloud.aws.signer: If you are using an old AWS API version, you can define a Signer. Shared with discovery-ec2 plugin
*/
Setting<String> SIGNER_SETTING = Setting.simpleString("cloud.aws.signer",
Property.NodeScope, Property.Deprecated, Property.Shared);
/**
* cloud.aws.read_timeout: Socket read timeout. Shared with discovery-ec2 plugin
*/
Setting<TimeValue> READ_TIMEOUT = Setting.timeSetting("cloud.aws.read_timeout",
TimeValue.timeValueMillis(ClientConfiguration.DEFAULT_SOCKET_TIMEOUT), Property.NodeScope, Property.Deprecated, Property.Shared);

/**
* Defines specific s3 settings starting with cloud.aws.s3.
* NOTE: These are legacy settings. Use the named client configs in {@link org.elasticsearch.repositories.s3.S3Repository}.
*/
interface CLOUD_S3 {
/**
* cloud.aws.s3.access_key: AWS Access key specific for S3 API calls. Defaults to cloud.aws.access_key.
* @see AwsS3Service#KEY_SETTING
*/
Setting<SecureString> KEY_SETTING =
new Setting<>("cloud.aws.s3.access_key", AwsS3Service.KEY_SETTING, SecureString::new,
Property.NodeScope, Property.Filtered, Property.Deprecated);
/**
* cloud.aws.s3.secret_key: AWS Secret key specific for S3 API calls. Defaults to cloud.aws.secret_key.
* @see AwsS3Service#SECRET_SETTING
*/
Setting<SecureString> SECRET_SETTING =
new Setting<>("cloud.aws.s3.secret_key", AwsS3Service.SECRET_SETTING, SecureString::new,
Property.NodeScope, Property.Filtered, Property.Deprecated);
/**
* cloud.aws.s3.protocol: Protocol for AWS API specific for S3 API calls: http or https. Defaults to cloud.aws.protocol.
* @see AwsS3Service#PROTOCOL_SETTING
*/
Setting<Protocol> PROTOCOL_SETTING =
new Setting<>("cloud.aws.s3.protocol", AwsS3Service.PROTOCOL_SETTING, s -> Protocol.valueOf(s.toUpperCase(Locale.ROOT)),
Property.NodeScope, Property.Deprecated);
/**
* cloud.aws.s3.proxy.host: In case of proxy, define its hostname/IP specific for S3 API calls. Defaults to cloud.aws.proxy.host.
* @see AwsS3Service#PROXY_HOST_SETTING
*/
Setting<String> PROXY_HOST_SETTING =
new Setting<>("cloud.aws.s3.proxy.host", AwsS3Service.PROXY_HOST_SETTING, Function.identity(),
Property.NodeScope, Property.Deprecated);
/**
* cloud.aws.s3.proxy.port: In case of proxy, define its port specific for S3 API calls. Defaults to cloud.aws.proxy.port.
* @see AwsS3Service#PROXY_PORT_SETTING
*/
Setting<Integer> PROXY_PORT_SETTING =
new Setting<>("cloud.aws.s3.proxy.port", AwsS3Service.PROXY_PORT_SETTING,
s -> Setting.parseInt(s, 0, 1<<16, "cloud.aws.s3.proxy.port"), Property.NodeScope, Property.Deprecated);
/**
* cloud.aws.s3.proxy.username: In case of proxy with auth, define the username specific for S3 API calls.
* Defaults to cloud.aws.proxy.username.
* @see AwsS3Service#PROXY_USERNAME_SETTING
*/
Setting<SecureString> PROXY_USERNAME_SETTING =
new Setting<>("cloud.aws.s3.proxy.username", AwsS3Service.PROXY_USERNAME_SETTING, SecureString::new,
Property.NodeScope, Property.Deprecated);
/**
* cloud.aws.s3.proxy.password: In case of proxy with auth, define the password specific for S3 API calls.
* Defaults to cloud.aws.proxy.password.
* @see AwsS3Service#PROXY_PASSWORD_SETTING
*/
Setting<SecureString> PROXY_PASSWORD_SETTING =
new Setting<>("cloud.aws.s3.proxy.password", AwsS3Service.PROXY_PASSWORD_SETTING, SecureString::new,
Property.NodeScope, Property.Filtered, Property.Deprecated);
/**
* cloud.aws.s3.signer: If you are using an old AWS API version, you can define a Signer. Specific for S3 API calls.
* Defaults to cloud.aws.signer.
* @see AwsS3Service#SIGNER_SETTING
*/
Setting<String> SIGNER_SETTING =
new Setting<>("cloud.aws.s3.signer", AwsS3Service.SIGNER_SETTING, Function.identity(),
Property.NodeScope, Property.Deprecated);
/**
* cloud.aws.s3.endpoint: Endpoint.
*/
Setting<String> ENDPOINT_SETTING = Setting.simpleString("cloud.aws.s3.endpoint", Property.NodeScope);
/**
* cloud.aws.s3.read_timeout: Socket read timeout. Defaults to cloud.aws.read_timeout
* @see AwsS3Service#READ_TIMEOUT
*/
Setting<TimeValue> READ_TIMEOUT =
Setting.timeSetting("cloud.aws.s3.read_timeout", AwsS3Service.READ_TIMEOUT, Property.NodeScope, Property.Deprecated);
}

AmazonS3 client(Settings repositorySettings, Integer maxRetries, boolean useThrottleRetries, Boolean pathStyleAccess);
}
73 changes: 0 additions & 73 deletions plugins/repository-s3/src/main/java/org/elasticsearch/cloud/aws/AwsSigner.java
View file
Open in desktop

This file was deleted.

62 changes: 13 additions & 49 deletions plugins/repository-s3/src/main/java/org/elasticsearch/cloud/aws/InternalAwsS3Service.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,20 +97,14 @@ static ClientConfiguration buildConfiguration(Logger logger, Settings repository
// the response metadata cache is only there for diagnostics purposes,
// but can force objects from every response to the old generation.
clientConfiguration.setResponseMetadataCacheSize(0);
Protocol protocol = getConfigValue(repositorySettings, settings, clientName, S3Repository.PROTOCOL_SETTING,
S3Repository.Repository.PROTOCOL_SETTING, S3Repository.Repositories.PROTOCOL_SETTING);
Protocol protocol = getConfigValue(settings, clientName, S3Repository.PROTOCOL_SETTING);
clientConfiguration.setProtocol(protocol);

String proxyHost = getConfigValue(null, settings, clientName,
S3Repository.PROXY_HOST_SETTING, null, CLOUD_S3.PROXY_HOST_SETTING);
String proxyHost = getConfigValue(settings, clientName, S3Repository.PROXY_HOST_SETTING);
if (Strings.hasText(proxyHost)) {
Integer proxyPort = getConfigValue(null, settings, clientName,
S3Repository.PROXY_PORT_SETTING, null, CLOUD_S3.PROXY_PORT_SETTING);
try (SecureString proxyUsername = getConfigValue(null, settings, clientName,
S3Repository.PROXY_USERNAME_SETTING, null, CLOUD_S3.PROXY_USERNAME_SETTING);
SecureString proxyPassword = getConfigValue(null, settings, clientName,
S3Repository.PROXY_PASSWORD_SETTING, null, CLOUD_S3.PROXY_PASSWORD_SETTING)) {

Integer proxyPort = getConfigValue(settings, clientName, S3Repository.PROXY_PORT_SETTING);
try (SecureString proxyUsername = getConfigValue(settings, clientName, S3Repository.PROXY_USERNAME_SETTING);
SecureString proxyPassword = getConfigValue(settings, clientName, S3Repository.PROXY_PASSWORD_SETTING)) {
clientConfiguration
.withProxyHost(proxyHost)
.withProxyPort(proxyPort)
Expand All @@ -125,15 +119,7 @@ static ClientConfiguration buildConfiguration(Logger logger, Settings repository
}
clientConfiguration.setUseThrottleRetries(useThrottleRetries);

// #155: we might have 3rd party users using older S3 API version
String awsSigner = CLOUD_S3.SIGNER_SETTING.get(settings);
if (Strings.hasText(awsSigner)) {
logger.debug("using AWS API signer [{}]", awsSigner);
AwsSigner.configureSigner(awsSigner, clientConfiguration, endpoint);
}

TimeValue readTimeout = getConfigValue(null, settings, clientName,
S3Repository.READ_TIMEOUT_SETTING, null, CLOUD_S3.READ_TIMEOUT);
TimeValue readTimeout = getConfigValue(settings, clientName, S3Repository.READ_TIMEOUT_SETTING);
clientConfiguration.setSocketTimeout((int)readTimeout.millis());

return clientConfiguration;
Expand All @@ -142,11 +128,8 @@ static ClientConfiguration buildConfiguration(Logger logger, Settings repository
public static AWSCredentialsProvider buildCredentials(Logger logger, DeprecationLogger deprecationLogger,
Settings settings, Settings repositorySettings, String clientName) {
AWSCredentialsProvider credentials;
try (SecureString key = getConfigValue(repositorySettings, settings, clientName, S3Repository.ACCESS_KEY_SETTING,
S3Repository.Repository.KEY_SETTING, S3Repository.Repositories.KEY_SETTING);
SecureString secret = getConfigValue(repositorySettings, settings, clientName, S3Repository.SECRET_KEY_SETTING,
S3Repository.Repository.SECRET_SETTING, S3Repository.Repositories.SECRET_SETTING)) {

try (SecureString key = getConfigValue(settings, clientName, S3Repository.ACCESS_KEY_SETTING);
SecureString secret = getConfigValue(settings, clientName, S3Repository.SECRET_KEY_SETTING)) {
if (key.length() == 0 && secret.length() == 0) {
logger.debug("Using instance profile credentials");
credentials = new InstanceProfileCredentialsProvider();
Expand All @@ -160,38 +143,19 @@ public static AWSCredentialsProvider buildCredentials(Logger logger, Deprecation
}

// pkg private for tests
/** Returns the endpoint the client should use, based on the available endpoint settings found. */
/** Returns the endpoint the client should use. */
static String findEndpoint(Logger logger, Settings repositorySettings, Settings settings, String clientName) {
String endpoint = getConfigValue(repositorySettings, settings, clientName, S3Repository.ENDPOINT_SETTING,
S3Repository.Repository.ENDPOINT_SETTING, S3Repository.Repositories.ENDPOINT_SETTING);
if (Strings.isNullOrEmpty(endpoint)) {
// No region has been set so we will use the default endpoint
if (CLOUD_S3.ENDPOINT_SETTING.exists(settings)) {
endpoint = CLOUD_S3.ENDPOINT_SETTING.get(settings);
logger.debug("using explicit s3 endpoint [{}]", endpoint);
}
} else {
logger.debug("using repository level endpoint [{}]", endpoint);
}

String endpoint = getConfigValue(settings, clientName, S3Repository.ENDPOINT_SETTING);
logger.debug("using repository level endpoint [{}]", endpoint);
return endpoint;
}

/**
* Find the setting value, trying first with named configs,
* then falling back to repository and global repositories settings.
*/
private static <T> T getConfigValue(Settings repositorySettings, Settings globalSettings, String clientName,
Setting.AffixSetting<T> configSetting, Setting<T> repositorySetting, Setting<T> globalSetting) {
Setting<T> concreteSetting = configSetting.getConcreteSettingForNamespace(clientName);
if (concreteSetting.exists(globalSettings)) {
return concreteSetting.get(globalSettings);
} else if (repositorySetting == null) {
// no repository setting, just use global setting
return globalSetting.get(globalSettings);
} else {
return S3Repository.getValue(repositorySettings, globalSettings, repositorySetting, globalSetting);
}
private static <T> T getConfigValue(Settings globalSettings, String clientName, Setting.AffixSetting<T> configSetting) {
return configSetting.getConcreteSettingForNamespace(clientName).get(globalSettings);
}

@Override
Expand Down
Loading