Skip to content

Commit e6bb354

Browse files
zhouyuanbeliefer
authored andcommitted
adding thrift config (apache#5)
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
1 parent 1cc73ae commit e6bb354

File tree

1 file changed

+35
-0
lines changed

1 file changed

+35
-0
lines changed
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
./stop_thrift
2+
SPARK_HOME=/home/sparkuser/tpcds/spark
3+
THRIFTSERVER_START_CMD="${SPARK_HOME}/sbin/start-thriftserver.sh"
4+
THRIFTSERVER_STOP_CMD="${SPARK_HOME}/sbin/stop-thriftserver.sh"
5+
6+
7+
batchsize=20480
8+
9+
10+
runname_base=vanilla_tpcds_power_jenkins
11+
runname="${runname_base}_${1}"
12+
13+
THRIFTSERVER_CONFIG="--name ${runname}
14+
--driver-memory 5g
15+
--num-executors 18
16+
--driver-memory 20g
17+
--executor-memory 20g
18+
--executor-cores 4
19+
--master yarn
20+
--deploy-mode client
21+
--conf spark.executor.memoryOverhead=384M
22+
--conf spark.sql.inMemoryColumnarStorage.batchSize=${batchsize}
23+
--conf spark.sql.execution.arrow.maxRecordsPerBatch=${batchsize}
24+
--conf spark.sql.parquet.columnarReaderBatchSize=${batchsize}
25+
--hiveconf hive.server2.thrift.port=10001
26+
--hiveconf hive.server2.thrift.bind.host=sr614
27+
--conf spark.sql.broadcastTimeout=300
28+
--conf spark.driver.maxResultSize=20g
29+
--conf spark.sql.shuffle.partitions=432
30+
--conf spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=0
31+
--conf spark.sql.optimizer.runtime.bloomFilter.enabled=true
32+
--conf spark.sql.optimizer.runtimeFilter.semiJoinReduction.enabled=true"
33+
34+
$THRIFTSERVER_START_CMD $THRIFTSERVER_CONFIG
35+

0 commit comments

Comments
 (0)