Skip to content

Commit 98d2065

Browse files
tasanumajojochuang
authored andcommitted
HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
1 parent 840d02c commit 98d2065

File tree

3 files changed

+56
-2
lines changed

3 files changed

+56
-2
lines changed

hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/ServerSetup.md.vm

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ Configure `etc/hadoop/ssl-server.xml` with proper values, for example:
114114
```
115115

116116
The SSL passwords can be secured by a credential provider. See
117-
[Credential Provider API](../../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html).
117+
[Credential Provider API](../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html).
118118

119119
You need to create an SSL certificate for the HttpFS server. As the `httpfs` Unix user, using the Java `keytool` command to create the SSL certificate:
120120

@@ -131,6 +131,7 @@ The answer to "What is your first and last name?" (i.e. "CN") must be the hostna
131131
Start HttpFS. It should work over HTTPS.
132132

133133
Using the Hadoop `FileSystem` API or the Hadoop FS shell, use the `swebhdfs://` scheme. Make sure the JVM is picking up the truststore containing the public key of the SSL certificate if using a self-signed certificate.
134+
For more information about the client side settings, see [SSL Configurations for SWebHDFS](../hadoop-project-dist/hadoop-hdfs/WebHDFS.html#SSL_Configurations_for_SWebHDFS).
134135

135136
NOTE: Some old SSL clients may use weak ciphers that are not supported by the HttpFS server. It is recommended to upgrade the SSL client.
136137

hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@ In the REST API, the prefix "`/webhdfs/v1`" is inserted in the path and a query
104104

105105
swebhdfs://<HOST>:<HTTP_PORT>/<PATH>
106106

107+
See also: [SSL Configurations for SWebHDFS](#SSL_Configurations_for_SWebHDFS)
107108

108109
### HDFS Configuration Options
109110

@@ -164,6 +165,56 @@ The following properties control OAuth2 authentication.
164165
| `dfs.webhdfs.oauth2.refresh.token.expires.ms.since.epoch` | (required if using ConfRefreshTokenBasedAccessTokenProvider) Access token expiration measured in milliseconds since Jan 1, 1970. *Note this is a different value than provided by OAuth providers and has been munged as described in interface to be suitable for a client application* |
165166
| `dfs.webhdfs.oauth2.credential` | (required if using ConfCredentialBasedAccessTokenProvider). Credential used to obtain initial and subsequent access tokens. |
166167

168+
SSL Configurations for SWebHDFS
169+
-------------------------------------------------------
170+
171+
To use SWebHDFS FileSystem (i.e. using the swebhdfs protocol), a SSL configuration
172+
file needs to be specified on the client side. This must specify 3 parameters:
173+
174+
| SSL property | Description |
175+
|:---- |:---- |
176+
| `ssl.client.truststore.location` | The local-filesystem location of the trust-store file, containing the certificate for the NameNode. |
177+
| `ssl.client.truststore.type` | (Optional) The format of the trust-store file. |
178+
| `ssl.client.truststore.password` | (Optional) Password for the trust-store file. |
179+
180+
The following is an example SSL configuration file (**ssl-client.xml**):
181+
182+
```xml
183+
<configuration>
184+
<property>
185+
<name>ssl.client.truststore.location</name>
186+
<value>/work/keystore.jks</value>
187+
<description>Truststore to be used by clients. Must be specified.</description>
188+
</property>
189+
190+
<property>
191+
<name>ssl.client.truststore.password</name>
192+
<value>changeme</value>
193+
<description>Optional. Default value is "".</description>
194+
</property>
195+
196+
<property>
197+
<name>ssl.client.truststore.type</name>
198+
<value>jks</value>
199+
<description>Optional. Default value is "jks".</description>
200+
</property>
201+
</configuration>
202+
```
203+
204+
The SSL configuration file must be in the class-path of the client program and the filename needs to be specified in **core-site.xml**:
205+
206+
```xml
207+
<property>
208+
<name>hadoop.ssl.client.conf</name>
209+
<value>ssl-client.xml</value>
210+
<description>
211+
Resource file from which ssl client keystore information will be extracted.
212+
This file is looked up in the classpath, typically it should be in Hadoop
213+
conf/ directory. Default value is "ssl-client.xml".
214+
</description>
215+
</property>
216+
```
217+
167218
Proxy Users
168219
-----------
169220

hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -542,10 +542,12 @@ $H3 Copying Between Versions of HDFS
542542
HftpFileSystem, as webhdfs is available for both read and write operations,
543543
DistCp can be run on both source and destination cluster.
544544
Remote cluster is specified as `webhdfs://<namenode_hostname>:<http_port>`.
545-
(Use the "`swebhdfs://`" scheme when webhdfs is secured with SSL).
546545
When copying between same major versions of Hadoop cluster (e.g. between 2.X
547546
and 2.X), use hdfs protocol for better performance.
548547

548+
$H3 Secure Copy over the wire with distcp
549+
Use the "`swebhdfs://`" scheme when webhdfs is secured with SSL. For more information see [SSL Configurations for SWebHDFS](../hadoop-project-dist/hadoop-hdfs/WebHDFS.html#SSL_Configurations_for_SWebHDFS).
550+
549551
$H3 MapReduce and other side-effects
550552

551553
As has been mentioned in the preceding, should a map fail to copy one of its

0 commit comments

Comments
 (0)