Skip to content

Commit

Permalink
[apache#1094] docs: split tez client guide into separate file
Browse files Browse the repository at this point in the history
  • Loading branch information
zivali committed Nov 5, 2023
1 parent 76ae3ed commit 18b4ed1
Show file tree
Hide file tree
Showing 2 changed files with 28 additions and 29 deletions.
31 changes: 2 additions & 29 deletions docs/client_guide/client_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,29 +33,9 @@ Refer to the following documents on how to deploy Uniffle client plugins with Sp
|Tez|[Deploy Tez Client Plugin & Configurations](tez_client_guide.md)|


### Deploy Tez Client Plugin
## Common Configuration

1. Append client jar to pacakge which is set by 'tez.lib.uris'.

In production mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to package which is set by 'tez.lib.uris'.

In development mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to HADOOP_CLASSPATH.

2. Update tez-site.xml to enable Uniffle.

| Property Name |Default| Description |
|----------------------------|---|------------------------------|
| tez.am.launch.cmd-opts |-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC org.apache.tez.dag.app.RssDAGAppMaster| enable remote shuffle service |
| tez.rss.coordinator.quorum |coordinatorIp1:19999,coordinatorIp2:19999|coordinator address|

Note that the RssDAGAppMaster will automatically disable slow start (i.e., `tez.shuffle-vertex-manager.min-src-fraction=1`, `tez.shuffle-vertex-manager.max-src-fraction=1`).

## Configuration

The important configuration of client is listed as following.

### Common Setting
These configurations are shared by all types of clients.
The important configuration of client is listed as following. These configurations are shared by all types of clients.

|Property Name|Default| Description |
|---|---|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
Expand Down Expand Up @@ -150,14 +130,7 @@ This experimental feature allows reduce tasks to spill data to remote storage (e
Notice: this feature requires the MEMORY_LOCAL_HADOOP mode.


### Tez Specialized Setting

| Property Name | Default | Description |
|--------------------------------|---------|-------------------------------------------------------------------------|
| tez.rss.avoid.recompute.succeeded.task | false | Whether to avoid recompute succeeded task when node is unhealthy or black-listed |
| tez.rss.client.max.buffer.size | 3k | The max buffer size in map side. Control the size of each segment(WrappedBuffer) in the buffer. |
| tez.rss.client.batch.trigger.num | 50 | The max batch of buffers to send data in map side. Affect the number of blocks sent to the server in each batch, and may affect rss_worker_used_buffer_size |
| tez.rss.client.send.thread.num | 5 | The thread pool size for the client to send data to the server. |

### Netty Setting
| Property Name | Default | Description |
Expand Down
26 changes: 26 additions & 0 deletions docs/client_guide/tez_client_guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Deploy Tez Client Plugin & Configurations
## Deploy Tez Client Plugin

1. Append client jar to pacakge which is set by 'tez.lib.uris'.

In production mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to package which is set by 'tez.lib.uris'.

In development mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to HADOOP_CLASSPATH.

2. Update tez-site.xml to enable Uniffle.

| Property Name |Default| Description |
|----------------------------|---|------------------------------|
| tez.am.launch.cmd-opts |-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC org.apache.tez.dag.app.RssDAGAppMaster| enable remote shuffle service |
| tez.rss.coordinator.quorum |coordinatorIp1:19999,coordinatorIp2:19999|coordinator address|

Note that the RssDAGAppMaster will automatically disable slow start (i.e., `tez.shuffle-vertex-manager.min-src-fraction=1`, `tez.shuffle-vertex-manager.max-src-fraction=1`).

## Tez Specialized Setting

| Property Name | Default | Description |
|--------------------------------|---------|-------------------------------------------------------------------------|
| tez.rss.avoid.recompute.succeeded.task | false | Whether to avoid recompute succeeded task when node is unhealthy or black-listed |
| tez.rss.client.max.buffer.size | 3k | The max buffer size in map side. Control the size of each segment(WrappedBuffer) in the buffer. |
| tez.rss.client.batch.trigger.num | 50 | The max batch of buffers to send data in map side. Affect the number of blocks sent to the server in each batch, and may affect rss_worker_used_buffer_size |
| tez.rss.client.send.thread.num | 5 | The thread pool size for the client to send data to the server. |

0 comments on commit 18b4ed1

Please sign in to comment.