Skip to content

Commit 3227968

Browse files
MaxGekkMaxGekk
committed
Merge remote-tracking branch 'remotes/origin/master' into special-values-under-config
2 parents b8ed08b + 051e691 commit 3227968

File tree

151 files changed

+3248
-1195
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

151 files changed

+3248
-1195
lines changed

‎.github/workflows/master.yml‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,35 @@ jobs:
2626
- name: Set up JDK ${{ matrix.java }}
2727
uses: actions/setup-java@v1
2828
with:
29-
version: ${{ matrix.java }}
29+
java-version: ${{ matrix.java }}
3030
- name: Build with Maven
3131
run: |
32-
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
32+
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
3333
export MAVEN_CLI_OPTS="--no-transfer-progress"
3434
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -P${{ matrix.hadoop }} -Phadoop-cloud -Djava.version=${{ matrix.java }} package
35+
36+
37+
lint:
38+
runs-on: ubuntu-latest
39+
name: Linters
40+
steps:
41+
- uses: actions/checkout@master
42+
- uses: actions/setup-java@v1
43+
with:
44+
java-version: '11'
45+
- uses: actions/setup-python@v1
46+
with:
47+
python-version: '3.x'
48+
architecture: 'x64'
49+
- name: Scala
50+
run: ./dev/lint-scala
51+
- name: Java
52+
run: ./dev/lint-java
53+
- name: Python
54+
run: |
55+
pip install flake8 sphinx numpy
56+
./dev/lint-python
57+
- name: License
58+
run: ./dev/check-license
59+
- name: Dependencies
60+
run: ./dev/test-dependencies.sh

‎LICENSE-binary‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -396,6 +396,8 @@ org.codehaus.janino:commons-compiler
396396
org.codehaus.janino:janino
397397
jline:jline
398398
org.jodd:jodd-core
399+
com.github.wendykierp:JTransforms
400+
pl.edu.icm:JLargeArrays
399401

400402

401403
BSD 3-Clause
@@ -433,8 +435,12 @@ is distributed under the 3-Clause BSD license.
433435
MIT License
434436
-----------
435437

436-
org.spire-math:spire-macros_2.12
437-
org.spire-math:spire_2.12
438+
org.typelevel:spire_2.12
439+
org.typelevel:spire-macros_2.12
440+
org.typelevel:spire-platform_2.12
441+
org.typelevel:spire-util_2.12
442+
org.typelevel:algebra_2.12:jar
443+
org.typelevel:cats-kernel_2.12
438444
org.typelevel:machinist_2.12
439445
net.razorvine:pyrolite
440446
org.slf4j:jcl-over-slf4j
@@ -496,11 +502,6 @@ Eclipse Public License (EPL) 2.0
496502
jakarta.annotation:jakarta-annotation-api https://projects.eclipse.org/projects/ee4j.ca
497503
jakarta.ws.rs:jakarta.ws.rs-api https://github.com/eclipse-ee4j/jaxrs-api
498504

499-
Mozilla Public License (MPL) 1.1
500-
--------------------------------
501-
502-
com.github.rwl:jtransforms https://sourceforge.net/projects/jtransforms/
503-
504505

505506
Python Software Foundation License
506507
----------------------------------

‎build/mvn‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ _DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
2222
# Preserve the calling directory
2323
_CALLING_DIR="$(pwd)"
2424
# Options used during compilation
25-
_COMPILE_JVM_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"
25+
_COMPILE_JVM_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g"
2626

2727
# Installs any application tarball given a URL, the expected tarball name,
2828
# and, optionally, a checkable binary path to determine if the binary has

‎common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -252,7 +252,7 @@ private static <T> Predicate<? super T> getPredicate(
252252

253253
return (value) -> set.contains(indexValueForEntity(getter, value));
254254
} else {
255-
HashSet<Comparable> set = new HashSet<>(values.size());
255+
HashSet<Comparable<?>> set = new HashSet<>(values.size());
256256
for (Object key : values) {
257257
set.add(asKey(key));
258258
}

‎common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVTypeInfo.java‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ interface Accessor {
124124

125125
Object get(Object instance) throws ReflectiveOperationException;
126126

127-
Class getType();
127+
Class<?> getType();
128128
}
129129

130130
private class FieldAccessor implements Accessor {
@@ -141,7 +141,7 @@ public Object get(Object instance) throws ReflectiveOperationException {
141141
}
142142

143143
@Override
144-
public Class getType() {
144+
public Class<?> getType() {
145145
return field.getType();
146146
}
147147
}
@@ -160,7 +160,7 @@ public Object get(Object instance) throws ReflectiveOperationException {
160160
}
161161

162162
@Override
163-
public Class getType() {
163+
public Class<?> getType() {
164164
return method.getReturnType();
165165
}
166166
}

‎common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -293,9 +293,8 @@ public void close() {
293293
}
294294
connectionPool.clear();
295295

296-
if (workerGroup != null) {
296+
if (workerGroup != null && !workerGroup.isShuttingDown()) {
297297
workerGroup.shutdownGracefully();
298-
workerGroup = null;
299298
}
300299
}
301300
}

‎common/network-common/src/test/java/org/apache/spark/network/TransportClientFactorySuite.java‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,4 +217,11 @@ public Iterable<Map.Entry<String, String>> getAll() {
217217
assertFalse(c1.isActive());
218218
}
219219
}
220+
221+
@Test(expected = IOException.class)
222+
public void closeFactoryBeforeCreateClient() throws IOException, InterruptedException {
223+
TransportClientFactory factory = context.createClientFactory();
224+
factory.close();
225+
factory.createClient(TestUtils.getLocalHost(), server1.getPort());
226+
}
220227
}

‎core/benchmarks/CoalescedRDDBenchmark-jdk11-results.txt‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
================================================================================================
2+
Coalesced RDD , large scale
3+
================================================================================================
4+
5+
OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
6+
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
7+
Coalesced RDD: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
8+
------------------------------------------------------------------------------------------------------------------------
9+
Coalesce Num Partitions: 100 Num Hosts: 1 455 623 151 0.2 4547.4 1.0X
10+
Coalesce Num Partitions: 100 Num Hosts: 5 340 409 65 0.3 3397.1 1.3X
11+
Coalesce Num Partitions: 100 Num Hosts: 10 292 363 95 0.3 2923.3 1.6X
12+
Coalesce Num Partitions: 100 Num Hosts: 20 307 320 11 0.3 3069.8 1.5X
13+
Coalesce Num Partitions: 100 Num Hosts: 40 333 368 55 0.3 3329.1 1.4X
14+
Coalesce Num Partitions: 100 Num Hosts: 80 286 338 63 0.3 2862.5 1.6X
15+
Coalesce Num Partitions: 500 Num Hosts: 1 769 837 59 0.1 7693.5 0.6X
16+
Coalesce Num Partitions: 500 Num Hosts: 5 427 461 31 0.2 4268.5 1.1X
17+
Coalesce Num Partitions: 500 Num Hosts: 10 372 389 27 0.3 3722.2 1.2X
18+
Coalesce Num Partitions: 500 Num Hosts: 20 347 365 31 0.3 3468.5 1.3X
19+
Coalesce Num Partitions: 500 Num Hosts: 40 335 336 1 0.3 3347.3 1.4X
20+
Coalesce Num Partitions: 500 Num Hosts: 80 329 360 49 0.3 3294.5 1.4X
21+
Coalesce Num Partitions: 1000 Num Hosts: 1 1254 1292 47 0.1 12538.6 0.4X
22+
Coalesce Num Partitions: 1000 Num Hosts: 5 518 553 47 0.2 5177.0 0.9X
23+
Coalesce Num Partitions: 1000 Num Hosts: 10 394 432 42 0.3 3937.3 1.2X
24+
Coalesce Num Partitions: 1000 Num Hosts: 20 341 381 44 0.3 3414.4 1.3X
25+
Coalesce Num Partitions: 1000 Num Hosts: 40 313 358 48 0.3 3134.9 1.5X
26+
Coalesce Num Partitions: 1000 Num Hosts: 80 335 360 38 0.3 3347.0 1.4X
27+
Coalesce Num Partitions: 5000 Num Hosts: 1 3937 4066 156 0.0 39375.0 0.1X
28+
Coalesce Num Partitions: 5000 Num Hosts: 5 1413 1453 40 0.1 14133.4 0.3X
29+
Coalesce Num Partitions: 5000 Num Hosts: 10 826 861 49 0.1 8255.2 0.6X
30+
Coalesce Num Partitions: 5000 Num Hosts: 20 542 609 58 0.2 5423.3 0.8X
31+
Coalesce Num Partitions: 5000 Num Hosts: 40 410 470 64 0.2 4101.0 1.1X
32+
Coalesce Num Partitions: 5000 Num Hosts: 80 352 427 69 0.3 3515.3 1.3X
33+
Coalesce Num Partitions: 10000 Num Hosts: 1 7101 7151 54 0.0 71007.4 0.1X
34+
Coalesce Num Partitions: 10000 Num Hosts: 5 2540 2582 59 0.0 25396.2 0.2X
35+
Coalesce Num Partitions: 10000 Num Hosts: 10 1378 1432 48 0.1 13781.4 0.3X
36+
Coalesce Num Partitions: 10000 Num Hosts: 20 829 867 66 0.1 8286.8 0.5X
37+
Coalesce Num Partitions: 10000 Num Hosts: 40 573 630 49 0.2 5730.2 0.8X
38+
Coalesce Num Partitions: 10000 Num Hosts: 80 438 449 9 0.2 4382.5 1.0X
39+
40+

‎core/benchmarks/KryoBenchmark-jdk11-results.txt‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
================================================================================================
2+
Benchmark Kryo Unsafe vs safe Serialization
3+
================================================================================================
4+
5+
OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
6+
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
7+
Benchmark Kryo Unsafe vs safe Serialization: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
8+
------------------------------------------------------------------------------------------------------------------------
9+
basicTypes: Int with unsafe:true 324 329 2 3.1 324.0 1.0X
10+
basicTypes: Long with unsafe:true 353 355 1 2.8 353.0 0.9X
11+
basicTypes: Float with unsafe:true 336 338 1 3.0 336.4 1.0X
12+
basicTypes: Double with unsafe:true 347 347 1 2.9 346.5 0.9X
13+
Array: Int with unsafe:true 4 4 0 265.4 3.8 86.0X
14+
Array: Long with unsafe:true 6 7 0 157.3 6.4 51.0X
15+
Array: Float with unsafe:true 4 4 0 268.8 3.7 87.1X
16+
Array: Double with unsafe:true 6 7 0 157.5 6.3 51.0X
17+
Map of string->Double with unsafe:true 52 52 1 19.3 51.8 6.3X
18+
basicTypes: Int with unsafe:false 357 358 1 2.8 357.2 0.9X
19+
basicTypes: Long with unsafe:false 387 388 0 2.6 387.4 0.8X
20+
basicTypes: Float with unsafe:false 356 357 1 2.8 356.0 0.9X
21+
basicTypes: Double with unsafe:false 371 372 1 2.7 371.0 0.9X
22+
Array: Int with unsafe:false 24 24 0 41.3 24.2 13.4X
23+
Array: Long with unsafe:false 37 38 0 26.8 37.4 8.7X
24+
Array: Float with unsafe:false 11 11 0 94.9 10.5 30.8X
25+
Array: Double with unsafe:false 18 18 0 55.2 18.1 17.9X
26+
Map of string->Double with unsafe:false 55 55 0 18.2 55.1 5.9X
27+
28+

‎core/benchmarks/KryoSerializerBenchmark-jdk11-results.txt‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
================================================================================================
2+
Benchmark KryoPool vs old"pool of 1" implementation
3+
================================================================================================
4+
5+
OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
6+
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
7+
Benchmark KryoPool vs old"pool of 1" implementation: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
8+
------------------------------------------------------------------------------------------------------------------------
9+
KryoPool:true 6524 9149 NaN 0.0 13047076.5 1.0X
10+
KryoPool:false 12855 16469 663 0.0 25709170.6 0.5X
11+
12+

0 commit comments

Comments
 (0)