Repository navigation
[Feature Request]: Support Apache Spark 4.x in Beam Spark Runner #36841
Description
What would you like to happen?
I would like Beam's Spark Runner to officially support Apache Spark 4.0. Spark 4.0 brings multiple improvements and new features, including performance enhancements, security updates, and updated APIs (e.g., improvements in shuffle, catalog features, Kubernetes operator, and structured streaming). As major platforms (such as Dataproc, EMR, and internal platforms) start adopting Spark 4.0, supporting this version will enable users to migrate without friction and leverage new Spark functionalities. Current Beam documentation lists support up to Spark 3.x, and 4.0 support is not mentioned. Attempting to run the Spark Runner on Spark 4.0 results in compatibility issues (such as dependency conflicts and API changes). Official CI matrix coverage, documentation, and dependency updates for 4.0 are needed for safe production use. This would help unblock organizations intending to move to Spark 4.0 and keep Beam competitive as Spark evolves.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
Activity
liferoad commentedon Nov 24, 2025
Spark 4.0 requires Java 17+.
Fix Spark 4.x compilation errors for Beam Spark Runner
Abacn commentedon Dec 1, 2025
We can set applyJavaNature(..., requireJavaVersion: JavaVersion.VERSION_17) for BeamModulePlugin.
nia-potato commentedon Mar 4, 2026
hi, any updates on this? we are currently blocked by this issue. thanks
tkaymak commentedon Apr 14, 2026
@Abacn is this issue still valid to be targeted for the 3.0 release? Can I help here?
docs: add Spark 4 runner entry to CHANGES.md
docs: add Spark 4 runner entry to CHANGES.md
[Spark Runner] Hoist Spark 3 structured-streaming to shared base, ado…
docs: add Spark 4 runner entry to CHANGES.md
12 remaining items
[Spark][#36841] Translate stateless streaming pipelines on the Spark …
tkaymak commentedon Sep 11, 2026
Slice 4, the stateless streaming translator, merged as #40090. The Spark 4 runner now runs unbounded reads, stateless ParDo, Window.Assign, Flatten and Reshuffle in streaming mode, everything else fails at translation with a message pointing here.
Found on the way:
cancel()did not cancel Spark jobs and could stop a shared SparkSession, #40101, fix in #40103.Remaining, cut from #39576 in this order: state and timer internals, the transformWithState processor, the GroupByKey and stateful ParDo translators, then PAssert support. No CHANGES.md entry until windowed GroupByKey lands.
Abacn commentedon Sep 14, 2026
My personal suggest is to setup ValidateRunner test suite as long as there is meaningful tests can pass. Then the addition is verified to functioning in pipeline level when checking in
tkaymak commentedon Sep 15, 2026
Thanks for the suggestion, I looked into wiring
ValidatesRunnerfor streaming:ValidatesRunnertests in sdks/java/core useCreate.oforPAssert, which requireGroupByKey(Slice 5c) andImpulse(Slice 5d), so zero SDKValidatesRunnertests can pass on master yet.My plan is to introduce
validatesStructuredStreamingRunnerStreamingalongside Slice 5c and 5d, so standard SDK tests start validating the runner end to end as soon asGroupByKeyandPAssertland.[Spark][apache#36841] Add an opt-in Dataset-based backend for portabl…
[Spark][apache#36841] Add an opt-in Dataset-based backend for portabl…
[Spark][apache#36841] Prune dead DStream helpers and add streaming li…
[Spark][#36841] Prune dead DStream helpers and add streaming lifecycl…
[Spark][apache#36841] Select the Dataset backend by flag and add comp…
[Spark][#36841] Add an opt-in Dataset-based backend for portable pipe…
tkaymak commentedon Sep 25, 2026
Status update on slice 5b. Stateful ParDo, meaning state plus event time timers, now runs on the Spark 4 Structured Streaming runner through
transformWithState. It is coded and tested locally. The PR is not opened yet.There is one known blocker for 5c. Spark has no watermark hold. So output from an event time timer is emitted after the watermark has already passed the timer timestamp. That is harmless when only stateless ParDos follow. A downstream GroupByKey would drop that output as late. Before 5c supports a GroupByKey after a stateful ParDo, it needs a way to hold back output timestamps, or it has to reject that pipeline shape.
[Spark 4] Support stateful ParDo in Structured Streaming via transfor…
[Spark 4] Support stateful ParDo in Structured Streaming via transfor…
[Spark 4] Filter late data against the key's previous watermark in th…
tkaymak commentedon Oct 12, 2026
Status update. Slice 5b, stateful ParDo via
transformWithState, merged in #40281. The blocker from my last comment is resolved by #40447: late data is now filtered against the key's previous watermark, so timer output behind the shared watermark survives into a downstream stateful operator. The bound is per key rather than per batch, tracked in #40446. Next is windowed GroupByKey and Combine on the same processor, then Impulse and PAssert so a streaming ValidatesRunner suite can run.