forked from apache/incubator-uniffle
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[apache#1094] docs: split tez client guide into separate file
- Loading branch information
Showing
2 changed files
with
28 additions
and
29 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
# Deploy Tez Client Plugin & Configurations | ||
## Deploy Tez Client Plugin | ||
|
||
1. Append client jar to pacakge which is set by 'tez.lib.uris'. | ||
|
||
In production mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to package which is set by 'tez.lib.uris'. | ||
|
||
In development mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to HADOOP_CLASSPATH. | ||
|
||
2. Update tez-site.xml to enable Uniffle. | ||
|
||
| Property Name |Default| Description | | ||
|----------------------------|---|------------------------------| | ||
| tez.am.launch.cmd-opts |-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC org.apache.tez.dag.app.RssDAGAppMaster| enable remote shuffle service | | ||
| tez.rss.coordinator.quorum |coordinatorIp1:19999,coordinatorIp2:19999|coordinator address| | ||
|
||
Note that the RssDAGAppMaster will automatically disable slow start (i.e., `tez.shuffle-vertex-manager.min-src-fraction=1`, `tez.shuffle-vertex-manager.max-src-fraction=1`). | ||
|
||
## Tez Specialized Configurations | ||
|
||
| Property Name | Default | Description | | ||
|--------------------------------|---------|-------------------------------------------------------------------------| | ||
| tez.rss.avoid.recompute.succeeded.task | false | Whether to avoid recompute succeeded task when node is unhealthy or black-listed | | ||
| tez.rss.client.max.buffer.size | 3k | The max buffer size in map side. Control the size of each segment(WrappedBuffer) in the buffer. | | ||
| tez.rss.client.batch.trigger.num | 50 | The max batch of buffers to send data in map side. Affect the number of blocks sent to the server in each batch, and may affect rss_worker_used_buffer_size | | ||
| tez.rss.client.send.thread.num | 5 | The thread pool size for the client to send data to the server. | |