Skip to content

Commit fd56924

Browse files
yaooqinndongjoon-hyun
yaooqinn
authored and
dongjoon-hyun
committed
[SPARK-31129][SQL][TESTS] Fix IntervalBenchmark and DateTimeBenchmark
### What changes were proposed in this pull request? This PR aims to recover `IntervalBenchmark` and `DataTimeBenchmark` due to banning intervals as output. ### Why are the changes needed? This PR recovers the benchmark suite. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Manually, re-run the benchmark. Closes #27885 from yaooqinn/SPARK-31111-2. Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit fbc9dc7) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 4bcba6f commit fd56924

File tree

6 files changed

+491
-489
lines changed

6 files changed

+491
-489
lines changed

‎sql/core/benchmarks/DateTimeBenchmark-jdk11-results.txt

Lines changed: 217 additions & 217 deletions
Large diffs are not rendered by default.

‎sql/core/benchmarks/DateTimeBenchmark-results.txt

Lines changed: 217 additions & 217 deletions
Large diffs are not rendered by default.

‎sql/core/benchmarks/IntervalBenchmark-jdk11-results.txt

Lines changed: 26 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,29 @@
1-
OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
2-
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
1+
Java HotSpot(TM) 64-Bit Server VM 11.0.5+10-LTS on Mac OS X 10.15.3
2+
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
33
cast strings to intervals: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
44
------------------------------------------------------------------------------------------------------------------------
5-
prepare string w/ interval 756 831 78 1.3 756.4 1.0X
6-
prepare string w/o interval 622 637 15 1.6 622.4 1.2X
7-
1 units w/ interval 506 532 22 2.0 506.2 1.5X
8-
1 units w/o interval 472 477 9 2.1 471.6 1.6X
9-
2 units w/ interval 699 710 11 1.4 698.5 1.1X
10-
2 units w/o interval 670 674 6 1.5 669.9 1.1X
11-
3 units w/ interval 1431 1437 7 0.7 1431.3 0.5X
12-
3 units w/o interval 1418 1429 11 0.7 1418.2 0.5X
13-
4 units w/ interval 1687 1692 8 0.6 1686.8 0.4X
14-
4 units w/o interval 1679 1688 7 0.6 1679.4 0.5X
15-
5 units w/ interval 1862 1864 3 0.5 1861.9 0.4X
16-
5 units w/o interval 1847 1864 15 0.5 1846.9 0.4X
17-
6 units w/ interval 2067 2081 12 0.5 2066.9 0.4X
18-
6 units w/o interval 2070 2071 2 0.5 2069.6 0.4X
19-
7 units w/ interval 2458 2468 13 0.4 2457.7 0.3X
20-
7 units w/o interval 2450 2453 3 0.4 2450.1 0.3X
21-
8 units w/ interval 2833 2838 8 0.4 2832.6 0.3X
22-
8 units w/o interval 2830 2839 8 0.4 2829.8 0.3X
23-
9 units w/ interval 2873 2880 6 0.3 2873.4 0.3X
24-
9 units w/o interval 2860 2863 3 0.3 2860.1 0.3X
25-
10 units w/ interval 3252 3257 5 0.3 3252.2 0.2X
26-
10 units w/o interval 3212 3220 8 0.3 3211.6 0.2X
27-
11 units w/ interval 3369 3376 6 0.3 3368.5 0.2X
28-
11 units w/o interval 3384 3395 15 0.3 3384.2 0.2X
5+
prepare string w/ interval 448 469 20 2.2 447.6 1.0X
6+
prepare string w/o interval 405 409 4 2.5 404.6 1.1X
7+
1 units w/ interval 321 328 6 3.1 321.4 1.4X
8+
1 units w/o interval 303 307 4 3.3 303.1 1.5X
9+
2 units w/ interval 445 458 12 2.2 444.6 1.0X
10+
2 units w/o interval 416 424 10 2.4 416.2 1.1X
11+
3 units w/ interval 1006 1012 8 1.0 1006.4 0.4X
12+
3 units w/o interval 1240 1249 8 0.8 1239.6 0.4X
13+
4 units w/ interval 1295 1418 106 0.8 1295.4 0.3X
14+
4 units w/o interval 1172 1188 15 0.9 1171.6 0.4X
15+
5 units w/ interval 1326 1335 11 0.8 1325.6 0.3X
16+
5 units w/o interval 1309 1336 44 0.8 1308.7 0.3X
17+
6 units w/ interval 1441 1464 29 0.7 1441.0 0.3X
18+
6 units w/o interval 1350 1369 17 0.7 1350.1 0.3X
19+
7 units w/ interval 1606 1669 99 0.6 1605.6 0.3X
20+
7 units w/o interval 1546 1557 12 0.6 1546.3 0.3X
21+
8 units w/ interval 1771 1875 120 0.6 1770.6 0.3X
22+
8 units w/o interval 1775 1789 13 0.6 1775.2 0.3X
23+
9 units w/ interval 2126 2757 849 0.5 2126.4 0.2X
24+
9 units w/o interval 2053 2070 21 0.5 2053.3 0.2X
25+
10 units w/ interval 2209 2243 30 0.5 2209.1 0.2X
26+
10 units w/o interval 2400 2702 365 0.4 2400.2 0.2X
27+
11 units w/ interval 2616 2699 72 0.4 2616.5 0.2X
28+
11 units w/o interval 3218 3380 195 0.3 3218.4 0.1X
2929

‎sql/core/benchmarks/IntervalBenchmark-results.txt

Lines changed: 26 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,29 @@
1-
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
2-
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
1+
Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.3
2+
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
33
cast strings to intervals: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
44
------------------------------------------------------------------------------------------------------------------------
5-
prepare string w/ interval 648 721 94 1.5 648.3 1.0X
6-
prepare string w/o interval 562 596 49 1.8 562.3 1.2X
7-
1 units w/ interval 568 590 21 1.8 568.5 1.1X
8-
1 units w/o interval 522 538 20 1.9 521.7 1.2X
9-
2 units w/ interval 751 754 3 1.3 751.5 0.9X
10-
2 units w/o interval 716 723 6 1.4 716.1 0.9X
11-
3 units w/ interval 1402 1411 11 0.7 1401.6 0.5X
12-
3 units w/o interval 1381 1387 5 0.7 1381.2 0.5X
13-
4 units w/ interval 1591 1595 6 0.6 1591.2 0.4X
14-
4 units w/o interval 1582 1585 3 0.6 1582.3 0.4X
15-
5 units w/ interval 1747 1749 2 0.6 1747.3 0.4X
16-
5 units w/o interval 1738 1746 10 0.6 1737.7 0.4X
17-
6 units w/ interval 1929 1931 3 0.5 1929.1 0.3X
18-
6 units w/o interval 1919 1922 2 0.5 1919.0 0.3X
19-
7 units w/ interval 2345 2354 8 0.4 2345.0 0.3X
20-
7 units w/o interval 2334 2336 2 0.4 2334.1 0.3X
21-
8 units w/ interval 2533 2546 16 0.4 2533.0 0.3X
22-
8 units w/o interval 2519 2521 1 0.4 2519.4 0.3X
23-
9 units w/ interval 2885 2889 5 0.3 2884.5 0.2X
24-
9 units w/o interval 2804 2813 12 0.4 2803.9 0.2X
25-
10 units w/ interval 3041 3060 16 0.3 3041.3 0.2X
26-
10 units w/o interval 3031 3043 15 0.3 3031.2 0.2X
27-
11 units w/ interval 3270 3280 9 0.3 3269.9 0.2X
28-
11 units w/o interval 3273 3280 7 0.3 3272.6 0.2X
5+
prepare string w/ interval 389 410 21 2.6 388.7 1.0X
6+
prepare string w/o interval 340 360 18 2.9 340.5 1.1X
7+
1 units w/ interval 378 389 16 2.6 377.8 1.0X
8+
1 units w/o interval 346 350 5 2.9 346.2 1.1X
9+
2 units w/ interval 444 457 11 2.3 444.2 0.9X
10+
2 units w/o interval 455 464 12 2.2 455.1 0.9X
11+
3 units w/ interval 942 964 20 1.1 941.5 0.4X
12+
3 units w/o interval 927 1020 93 1.1 927.3 0.4X
13+
4 units w/ interval 1114 1127 17 0.9 1113.9 0.3X
14+
4 units w/o interval 1100 1105 4 0.9 1100.3 0.4X
15+
5 units w/ interval 1180 1244 57 0.8 1180.1 0.3X
16+
5 units w/o interval 1135 1141 6 0.9 1135.2 0.3X
17+
6 units w/ interval 1284 1316 48 0.8 1284.0 0.3X
18+
6 units w/o interval 1276 1357 122 0.8 1276.1 0.3X
19+
7 units w/ interval 1609 1636 32 0.6 1609.1 0.2X
20+
7 units w/o interval 1551 1578 36 0.6 1550.9 0.3X
21+
8 units w/ interval 1787 1874 129 0.6 1787.1 0.2X
22+
8 units w/o interval 1751 1767 15 0.6 1750.6 0.2X
23+
9 units w/ interval 1960 2065 141 0.5 1959.7 0.2X
24+
9 units w/o interval 1885 1908 39 0.5 1885.1 0.2X
25+
10 units w/ interval 2178 2185 11 0.5 2177.9 0.2X
26+
10 units w/o interval 2150 2255 164 0.5 2150.1 0.2X
27+
11 units w/ interval 2457 2542 139 0.4 2456.7 0.2X
28+
11 units w/o interval 2557 2770 188 0.4 2556.7 0.2X
2929

‎sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DateTimeBenchmark.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -114,12 +114,12 @@ object DateTimeBenchmark extends SqlBasedBenchmark {
114114
}
115115
runBenchmark("Parsing") {
116116
val n = 1000000
117-
val timestampStrExpr = "concat('2019-01-27 11:02:01.', cast(mod(id, 1000) as string))"
117+
val timestampStrExpr = "concat('2019-01-27 11:02:01.', rpad(mod(id, 1000), 3, '0'))"
118118
val pattern = "'yyyy-MM-dd HH:mm:ss.SSS'"
119119
run(n, "to timestamp str", timestampStrExpr)
120120
run(n, "to_timestamp", s"to_timestamp($timestampStrExpr, $pattern)")
121121
run(n, "to_unix_timestamp", s"to_unix_timestamp($timestampStrExpr, $pattern)")
122-
val dateStrExpr = "concat('2019-01-', cast(mod(id, 25) as string))"
122+
val dateStrExpr = "concat('2019-01-', lpad(mod(id, 25), 2, '0'))"
123123
run(n, "to date str", dateStrExpr)
124124
run(n, "to_date", s"to_date($dateStrExpr, 'yyyy-MM-dd')")
125125
}

‎sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/IntervalBenchmark.scala

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,9 @@ object IntervalBenchmark extends SqlBasedBenchmark {
4444
spark
4545
.range(0, cardinality, 1, 1)
4646
.select(exprs: _*)
47-
.noop()
47+
.queryExecution
48+
.toRdd
49+
.foreach(_ => ())
4850
}
4951
}
5052

0 commit comments

Comments
 (0)