Skip to content

Commit 1eb932a

Browse files
andygroveandygrove
authored
docs: Update TPC-H benchmark results (#1257)
1 parent d7a7812 commit 1eb932a

File tree

9 files changed

+419
-22
lines changed

9 files changed

+419
-22
lines changed

‎README.md‎

Lines changed: 6 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -46,30 +46,23 @@ The following chart shows the time it takes to run the 22 TPC-H queries against
4646
using a single executor with 8 cores. See the [Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html)
4747
for details of the environment used for these benchmarks.
4848

49-
When using Comet, the overall run time is reduced from 615 seconds to 364 seconds, a 1.7x speedup, with query 1
50-
running 9x faster than Spark.
49+
When using Comet, the overall run time is reduced from 640 seconds to 331 seconds, very close to a 2x speedup.
5150

52-
Running the same queries with DataFusion standalone (without Spark) using the same number of cores results in a 3.6x
53-
speedup compared to Spark.
51+
![](docs/source/_static/images/benchmark-results/0.5.0/tpch_allqueries.png)
5452

55-
Comet is not yet achieving full DataFusion speeds in all cases, but with future work we aim to provide a 2x-4x speedup
56-
for a broader set of queries.
53+
Here is a breakdown showing relative performance of Spark and Comet for each TPC-H query.
5754

58-
![](docs/source/_static/images/benchmark-results/0.4.0/tpch_allqueries.png)
59-
60-
Here is a breakdown showing relative performance of Spark, Comet, and DataFusion for each TPC-H query.
61-
62-
![](docs/source/_static/images/benchmark-results/0.4.0/tpch_queries_compare.png)
55+
![](docs/source/_static/images/benchmark-results/0.5.0/tpch_queries_compare.png)
6356

6457
The following charts shows how much Comet currently accelerates each query from the benchmark.
6558

6659
### Relative speedup
6760

68-
![](docs/source/_static/images/benchmark-results/0.4.0/tpch_queries_speedup_rel.png)
61+
![](docs/source/_static/images/benchmark-results/0.5.0/tpch_queries_speedup_rel.png)
6962

7063
### Absolute speedup
7164

72-
![](docs/source/_static/images/benchmark-results/0.4.0/tpch_queries_speedup_abs.png)
65+
![](docs/source/_static/images/benchmark-results/0.5.0/tpch_queries_speedup_abs.png)
7366

7467
These benchmarks can be reproduced in any environment using the documentation in the
7568
[Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html). We encourage

‎docs/source/_static/images/benchmark-results/0.5.0/tpch_allqueries.png‎

27.5 KB
Loading

‎docs/source/_static/images/benchmark-results/0.5.0/tpch_queries_compare.png‎

32.3 KB
Loading

‎docs/source/_static/images/benchmark-results/0.5.0/tpch_queries_speedup_abs.png‎

37.3 KB
Loading

‎docs/source/_static/images/benchmark-results/0.5.0/tpch_queries_speedup_rel.png‎

47.1 KB
Loading

‎docs/source/contributor-guide/benchmark-results/0.5.0/comet-tpch.json‎

Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
{
2+
"engine": "datafusion-comet",
3+
"benchmark": "tpch",
4+
"data_path": "/mnt/bigdata/tpch/sf100/",
5+
"query_path": "/home/andy/git/apache/datafusion-benchmarks/tpch/queries",
6+
"spark_conf": {
7+
"spark.comet.explain.native.enabled": "false",
8+
"spark.eventLog.enabled": "true",
9+
"spark.executor.extraClassPath": "/home/andy/git/apache/datafusion-comet/spark/target/comet-spark-spark3.4_2.12-0.5.0-SNAPSHOT.jar",
10+
"spark.comet.explainFallback.enabled": "false",
11+
"spark.comet.exec.replaceSortMergeJoin": "true",
12+
"spark.comet.exec.shuffle.enabled": "true",
13+
"spark.memory.offHeap.enabled": "true",
14+
"spark.comet.exec.shuffle.compression.level": "1",
15+
"spark.executor.memory": "16g",
16+
"spark.app.name": "comet benchmark derived from tpch",
17+
"spark.comet.batchSize": "8192",
18+
"spark.app.startTime": "1736802464855",
19+
"spark.comet.exec.shuffle.fallbackToColumnar": "true",
20+
"spark.serializer.objectStreamReset": "100",
21+
"spark.driver.host": "10.0.0.118",
22+
"spark.comet.exec.shuffle.enableFastEncoding": "true",
23+
"spark.submit.deployMode": "client",
24+
"spark.driver.port": "33103",
25+
"spark.comet.scan.impl": "native_comet",
26+
"spark.driver.extraClassPath": "/home/andy/git/apache/datafusion-comet/spark/target/comet-spark-spark3.4_2.12-0.5.0-SNAPSHOT.jar",
27+
"spark.executor.cores": "8",
28+
"spark.comet.explain.verbose.enabled": "false",
29+
"spark.driver.extraJavaOptions": "-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false",
30+
"spark.shuffle.manager": "org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager",
31+
"spark.comet.exec.enabled": "true",
32+
"spark.sql.warehouse.dir": "file:/home/andy/git/personal/research/benchmarks/spark-standalone/spark-warehouse",
33+
"spark.comet.scan.enabled": "true",
34+
"spark.app.submitTime": "1736802464584",
35+
"spark.executor.id": "driver",
36+
"spark.master": "spark://woody:7077",
37+
"spark.comet.exec.shuffle.mode": "auto",
38+
"spark.sql.extensions": "org.apache.comet.CometSparkSessionExtensions",
39+
"spark.driver.memory": "8G",
40+
"spark.repl.local.jars": "file:///home/andy/git/apache/datafusion-comet/spark/target/comet-spark-spark3.4_2.12-0.5.0-SNAPSHOT.jar",
41+
"spark.app.initial.jar.urls": "spark://10.0.0.118:33103/jars/comet-spark-spark3.4_2.12-0.5.0-SNAPSHOT.jar",
42+
"spark.app.id": "app-20250113140745-0058",
43+
"spark.rdd.compress": "True",
44+
"spark.executor.extraJavaOptions": "-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false",
45+
"spark.executor.instances": "1",
46+
"spark.cores.max": "8",
47+
"spark.comet.enabled": "true",
48+
"spark.submit.pyFiles": "",
49+
"spark.comet.exec.sortMergeJoinWithJoinFilter.enabled": "false",
50+
"spark.comet.exec.shuffle.compression.codec": "lz4",
51+
"spark.jars": "file:///home/andy/git/apache/datafusion-comet/spark/target/comet-spark-spark3.4_2.12-0.5.0-SNAPSHOT.jar",
52+
"spark.memory.offHeap.size": "16g",
53+
"spark.comet.columnar.shuffle.batch.size": "8192"
54+
},
55+
"1": [
56+
12.59755539894104,
57+
10.855465650558472,
58+
11.160947799682617,
59+
11.323237657546997,
60+
11.410452365875244
61+
],
62+
"2": [
63+
6.155406475067139,
64+
5.539891719818115,
65+
5.698071002960205,
66+
5.684133529663086,
67+
5.742799758911133
68+
],
69+
"3": [
70+
16.097025156021118,
71+
14.982890367507935,
72+
14.998259544372559,
73+
15.659432649612427,
74+
15.878185749053955
75+
],
76+
"4": [
77+
10.319517850875854,
78+
10.0553297996521,
79+
10.136846780776978,
80+
9.925675392150879,
81+
10.140193462371826
82+
],
83+
"5": [
84+
26.09030055999756,
85+
25.57556390762329,
86+
26.102373600006104,
87+
26.540887117385864,
88+
26.162983655929565
89+
],
90+
"6": [
91+
2.691145658493042,
92+
2.5986382961273193,
93+
2.659151792526245,
94+
2.6488683223724365,
95+
2.6785433292388916
96+
],
97+
"7": [
98+
15.326677560806274,
99+
15.57035493850708,
100+
16.023503065109253,
101+
16.015883207321167,
102+
15.79127025604248
103+
],
104+
"8": [
105+
27.72478675842285,
106+
27.45163321495056,
107+
27.935590267181396,
108+
27.86525869369507,
109+
28.016165733337402
110+
],
111+
"9": [
112+
39.186867237091064,
113+
39.73552465438843,
114+
40.866581439971924,
115+
40.73869442939758,
116+
40.89244842529297
117+
],
118+
"10": [
119+
14.022773742675781,
120+
14.476953029632568,
121+
14.305155515670776,
122+
14.187727451324463,
123+
14.57831335067749
124+
],
125+
"11": [
126+
5.223851919174194,
127+
4.722897291183472,
128+
4.844727277755737,
129+
4.803720474243164,
130+
4.822873592376709
131+
],
132+
"12": [
133+
4.974349021911621,
134+
5.013054132461548,
135+
5.0682995319366455,
136+
5.1071436405181885,
137+
5.142468452453613
138+
],
139+
"13": [
140+
9.769477128982544,
141+
9.743404626846313,
142+
9.935744285583496,
143+
9.966437339782715,
144+
9.854998588562012
145+
],
146+
"14": [
147+
5.320314168930054,
148+
5.26824426651001,
149+
5.269179344177246,
150+
5.322073698043823,
151+
5.292902708053589
152+
],
153+
"15": [
154+
9.532674789428711,
155+
9.520610570907593,
156+
9.538906335830688,
157+
9.553953886032104,
158+
9.65409803390503
159+
],
160+
"16": [
161+
5.146467924118042,
162+
4.716687440872192,
163+
4.863113164901733,
164+
4.725494384765625,
165+
4.653785228729248
166+
],
167+
"17": [
168+
30.45087242126465,
169+
30.785797119140625,
170+
30.950777530670166,
171+
31.04833745956421,
172+
31.12831139564514
173+
],
174+
"18": [
175+
27.549716472625732,
176+
27.610363960266113,
177+
27.41417407989502,
178+
27.633289098739624,
179+
27.72838020324707
180+
],
181+
"19": [
182+
5.9813477993011475,
183+
6.041543483734131,
184+
6.087557554244995,
185+
6.106397390365601,
186+
6.011293888092041
187+
],
188+
"20": [
189+
10.53919005393982,
190+
10.382107019424438,
191+
10.370867729187012,
192+
10.376642942428589,
193+
10.48800802230835
194+
],
195+
"21": [
196+
42.36113142967224,
197+
42.296979904174805,
198+
42.56899857521057,
199+
42.587459564208984,
200+
42.86927652359009
201+
],
202+
"22": [
203+
3.755877733230591,
204+
3.523585319519043,
205+
3.5420711040496826,
206+
3.605468273162842,
207+
3.6084585189819336
208+
]
209+
}

0 commit comments

Comments
 (0)