[SPARK-54571][CORE][SQL] Use LZ4 safeDecompressor by pan3793 · Pull Request #53454 · apache/spark

pan3793 · 2025-12-12T08:44:57Z

What changes were proposed in this pull request?

Previously, lz4-java was upgraded to 1.10.1 to address CVEs,

while this casues significant performance drop, see the benchmark report at

[SPARK-54693][CORE][TESTS] Add LZ4TPCDSDataBenchmark #53453

this PR follows the suggestion to migrate to safeDecompressor.

This PR also upgrades DB2 JDBC driver to a special version 12.1.3.0_special_74723 provided by DB2 team that bundles lz4-java 1.10.1 which addresses NoSuchMethodError issue (it only affects test).

Update: I opened SPARK-55136 (#53920) to process DB2 JDBC driver independently, please reivew that first.

Why are the changes needed?

Mitigate performance regression.

Does this PR introduce any user-facing change?

No, except for performance.

How was this patch tested?

GHA for functionality, benchmark for performance.

TL;DR - my test results show lz4-java 1.10.1 is about 10~15% slower on lz4 compression than 1.8.0, and is about ~5% slower on lz4 decompression even with migrating to suggested safeDecompressor

Was this patch authored or co-authored using generative AI tooling?

No.

pan3793 · 2025-12-14T19:30:39Z

the test failure is caused by com.ibm.db2:jcc includes unshaded old lz4 classes

pan3793 · 2025-12-15T02:39:33Z

cc @dbtsai @huaxingao, com.ibm.db2:jcc includes unshaded old lz4 classes, which causes sql/hive/docker-it modules test failure after this patch

java.lang.NoSuchMethodError: 'net.jpountz.lz4.LZ4BlockInputStream$Builder net.jpountz.lz4.LZ4BlockInputStream.newBuilder()'
 	at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:156)
 	...

I checked all versions available in Maven Central, all of them have the same issue.
https://mvnrepository.com/artifact/com.ibm.db2/jcc

~~I don't find the public contact info of the IBM DB2 JDBC driver team, not sure what's the next step, temporarily purge the dependency and disable DB2 tests? Or any better ideas?~~

Update: contacted the DB2 JDBC driver's author, new release that bundles the latest lz4-java is working in progress

Update: DB2 team provides a special JDBC driver 12.1.3.0_special_74723 that bundles lz4-java 1.10.1 which addressed NoSuchMethodError issue.

github-actions · 2026-01-22T06:30:25Z

JIRA Issue Information

=== Improvement SPARK-54571 ===
Summary: Use LZ4 safeDecompressor
Assignee: None
Status: Open
Affected: ["4.2.0"]

This comment was automatically generated by GitHub Actions

pan3793 · 2026-01-22T07:59:49Z

pom.xml

    <postgresql.version>42.7.7</postgresql.version>
-    <db2.jcc.version>11.5.9.0</db2.jcc.version>
+    <!-- A special version that bundles lz4-java 1.10.1 -->
+    <db2.jcc.version>12.1.3.0_special_74723</db2.jcc.version>


this dep is only used for testing

pan3793 · 2026-01-22T12:34:41Z

cc @dbtsai @mridulm @dongjoon-hyun @LuciferYang

LuciferYang · 2026-01-22T12:38:25Z

pom.xml

    <postgresql.version>42.7.7</postgresql.version>
-    <db2.jcc.version>11.5.9.0</db2.jcc.version>
+    <!-- A special version that bundles lz4-java 1.10.1 -->
+    <db2.jcc.version>12.1.3.0_special_74723</db2.jcc.version>


Can we upgrade it first in a separate pr？

okay, let me upgrade it first

@LuciferYang FYI, I opened SPARK-55136 (#53920) for it

dbtsai · 2026-02-11T06:18:17Z

@viirya can you take a look? Thanks.

viirya · 2026-02-11T07:42:15Z

Hmm, so that's said, in lz4-java 1.10.1, fastDecompressor is paradoxically much slower than safeDecompressor?

pan3793 · 2026-02-11T07:48:11Z

@viirya yeah, it was mentioned in https://sites.google.com/sonatype.com/vulnerabilities/cve-2025-12183

viirya · 2026-02-11T08:23:47Z

core/src/main/scala/org/apache/spark/io/CompressionCodec.scala

+    LZ4BlockInputStream.newBuilder()
+      .withDecompressor(lz4Factory.safeDecompressor())
+      .withChecksum(xxHashFactory.newStreamingHash32(defaultSeed).asChecksum)
+      .withStopOnEmptyBlock(disableConcatenationOfByteStream)


Is this .withStopOnEmptyBlock(disableConcatenationOfByteStream) a correct mapping to previous behavior?

yes, exactly, you can check the source code at

https://github.com/yawkat/lz4-java/blob/33d180cb70c4d93c80fb0dc3ab3002f457e93840/src/java/net/jpountz/lz4/LZ4BlockInputStream.java#L69

public LZ4BlockInputStream(InputStream in, LZ4FastDecompressor fastDecompressor, Checksum checksum, boolean stopOnEmptyBlock) { this(in, fastDecompressor, null, checksum, stopOnEmptyBlock); }

viirya · 2026-02-11T08:25:17Z

pom.xml

+    <!-- A special version that bundles lz4-java 1.10.1 -->
+    <db2.jcc.version>12.1.3.0_special_74723</db2.jcc.version>


Does it mean we need to update this dependency when upgrading lz4-java?

@viirya, ideally, we should, but I'm afraid we can't, we can discuss the Db2 JDBC issue in the dedicated PR #53920

Yicong-Huang · 2026-02-26T00:49:00Z

@pan3793 thanks for the great effort! I was not aware of this PR and opened the other one due to performance regression. Do we know what are the current blockers from merging this PR?

cc @dbtsai @ueshin @dongjoon-hyun

pan3793 · 2026-02-26T00:57:33Z

@Yicong-Huang, you can find all related issues/background in the PR description

…17, Scala 2.13, split 1 of 1)

…21, Scala 2.13, split 1 of 1)

dongjoon-hyun · 2026-02-26T04:58:21Z

core/benchmarks/LZ4TPCDSDataBenchmark-results.txt

 Compression:                              Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 ------------------------------------------------------------------------------------------------------------------------
-Compression 4 times                                2605           2611           8          0.0   651243236.8       1.0X
+Compression 4 times                                2613           2621          12          0.0   653232793.3       1.0X


~~Although safeDecompressor shows slower performance, but it's in the marginal scope. There might be some improvements at lz4-java 1.10.3.~~

~~Given the above, could you update the PR description?~~

nvm. I forgot that this PR is about Decompression (lz4Factory.safeDecompressor).

dongjoon-hyun · 2026-02-26T05:06:31Z

BTW, I updated the PR description to include the following because the previous benchmark result was generated based on that.

[SPARK-55189][BUILD] Upgrade lz4-java to 1.10.3 #53971

pan3793 · 2026-02-26T05:39:19Z

@dongjoon-hyun, the benchmark results in this PR are generated by lz4-java 1.10.1 a few months ago, I will regenerate that once we resolve the Db2 JDBC driver issue.

dongjoon-hyun · 2026-02-26T05:44:02Z

What I meant (by the following) was that the result in the current master branch. The - part.

the previous benchmark result was generated based on that.

- Decompression 4 times                              2219           2220           1          0.0   554762743.5       1.0X

For the new result, yes. We need to update it, of course.

pan3793 · 2026-02-26T05:47:47Z

I see

github-actions bot added the CORE label Dec 12, 2025

This was referenced Dec 12, 2025

[SPARK-54693][CORE][TESTS] Add LZ4TPCDSDataBenchmark #53453

Closed

[WIP][SPARK-54571][CORE] Use LZ4 safeDecompressor #53290

Closed

pan3793 changed the title ~~WIP [SPARK-54571] Use safeDecompressor~~ WIP [SPARK-54571] Use LZ4 safeDecompressor Dec 12, 2025

github-actions bot added the SQL label Dec 17, 2025

pan3793 force-pushed the SPARK-54571 branch from 8d20d9f to 6bf027c Compare December 17, 2025 07:11

pan3793 force-pushed the SPARK-54571 branch from c91c457 to 101a1ca Compare January 22, 2026 06:30

pan3793 changed the title ~~WIP [SPARK-54571] Use LZ4 safeDecompressor~~ [WIP][SPARK-54571] Use LZ4 safeDecompressor Jan 22, 2026

github-actions bot added the BUILD label Jan 22, 2026

pan3793 changed the title ~~[WIP][SPARK-54571] Use LZ4 safeDecompressor~~ [SPARK-54571] Use LZ4 safeDecompressor Jan 22, 2026

pan3793 changed the title ~~[SPARK-54571] Use LZ4 safeDecompressor~~ [SPARK-54571][CORE][SQL] Use LZ4 safeDecompressor Jan 22, 2026

pan3793 commented Jan 22, 2026

View reviewed changes

pan3793 marked this pull request as ready for review January 22, 2026 08:00

pan3793 force-pushed the SPARK-54571 branch from a04e58b to d293139 Compare January 22, 2026 08:04

LuciferYang reviewed Jan 22, 2026

View reviewed changes

pan3793 mentioned this pull request Jan 22, 2026

[SPARK-55136][BUILD] Bump Db2 JDBC driver 12.1.3.0_special_74723 #53920

Open

pan3793 requested a review from LuciferYang January 23, 2026 03:10

viirya reviewed Feb 11, 2026

View reviewed changes

pan3793 mentioned this pull request Feb 26, 2026

[SPARK-54571][CORE] Use LZ4 safeDecompressor #54496

Closed

safeDecompressor

5adab03

pan3793 and others added 4 commits February 26, 2026 09:53

Switch to LZ4 safeDecompressor in Statistics

980ad39

Bump DB2 JDBC driver 12.1.3.0_special_74723

737e17b

Benchmark results for org.apache.spark.io.LZ4TPCDSDataBenchmark (JDK …

b01343b

…17, Scala 2.13, split 1 of 1)

Benchmark results for org.apache.spark.io.LZ4TPCDSDataBenchmark (JDK …

63d6c50

…21, Scala 2.13, split 1 of 1)

pan3793 force-pushed the SPARK-54571 branch from d293139 to 63d6c50 Compare February 26, 2026 01:53

dongjoon-hyun reviewed Feb 26, 2026

View reviewed changes

		<!-- A special version that bundles lz4-java 1.10.1 -->
		<db2.jcc.version>12.1.3.0_special_74723</db2.jcc.version>

Conversation

pan3793 commented Dec 12, 2025 • edited by dongjoon-hyun Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

pan3793 commented Dec 14, 2025

Uh oh!

pan3793 commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 22, 2026

JIRA Issue Information

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Jan 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dbtsai commented Feb 11, 2026

Uh oh!

viirya commented Feb 11, 2026

Uh oh!

pan3793 commented Feb 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yicong-Huang commented Feb 26, 2026

Uh oh!

pan3793 commented Feb 26, 2026

Uh oh!

dongjoon-hyun Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 commented Feb 26, 2026

Uh oh!

dongjoon-hyun commented Feb 26, 2026

Uh oh!

pan3793 commented Feb 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pan3793 commented Dec 12, 2025 •

edited by dongjoon-hyun

Loading

pan3793 commented Dec 15, 2025 •

edited

Loading

dongjoon-hyun Feb 26, 2026 •

edited

Loading

dongjoon-hyun commented Feb 26, 2026 •

edited

Loading