Skip to content

Commit 01721b7

Browse files
committed
Use snappy instead of lz4 from jar conflict issues in 2.3+
1 parent 93df015 commit 01721b7

File tree

2 files changed

+3
-1
lines changed

2 files changed

+3
-1
lines changed

build.sbt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ name := "spark-testing-base"
44

55
publishMavenStyle := true
66

7-
version := "0.10.0"
7+
version := "0.11.0"
88

99
sparkVersion := "2.4.0"
1010

src/main/2.0/scala/com/holdenkarau/spark/testing/DataFrameSuiteBase.scala

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,8 @@ trait DataFrameSuiteBaseLike extends SparkContextProvider
7575
val localWarehousePath = new File(tempDir, "wharehouse").getCanonicalPath
7676
def newBuilder() = {
7777
val builder = SparkSession.builder()
78+
// Long story with lz4 issues in 2.3+
79+
builder.config("spark.io.compression.codec", "snappy")
7880
// We have to mask all properties in hive-site.xml that relates to metastore
7981
// data source as we used a local metastore here.
8082
if (enableHiveSupport) {

0 commit comments

Comments
 (0)