Repository navigation
Proposal: Flink support #1264
Description
Auron just support Spark(for batch) now. But in terms of capability, it can be used to support Flink (for streaming).
For many user scenarios in Flink, processing at the second or even minute level is perfectly acceptable. Examples include mini-batch processing for unbounded aggregates or minute-level window aggregations.
Auron will build Flink capabilities in three phases.
Phase One: Stateless Operator Support(Calc Operator)
- Support for various operator expressions: such as + - * /
- Support for Flink built-in functions
- Support for user-defined Flink functions
Phase Two: Stateful Operator Support(Agg Operator)
- Support for Flink built-in agg functions
- Support for user-defined Flink agg functions
- Support for flink count distinct
Phase Three: Stateful Operator Support(Join Operator)
- Support Regular Join
- Support Window Join
Supporting Flink in Auron is a massive and complex undertaking. We welcome community members to collaborate and discuss together.
Activity
x-tong commentedon Sep 22, 2025
I want to follow up on this feature.
Tartarus0zm commentedon Sep 23, 2025
We warmly welcome you! We will start building the foundational framework in the next week or two, and there will be a lot of work ahead that we will need to collaborate on with community partners.
github-actions commentedon Oct 24, 2025
This issue is stale because it has been open for 30 days with no activity.
Tartarus0zm commentedon Nov 3, 2025
Phase 1 Implementation Plan for Flink SQL on Auron We welcome everyone to share their insights and suggestions.
https://docs.google.com/document/d/1z3Noy4C9Lv54zEuL9MfGXohrDsIUh9zBpxOkmulRM6s/edit?usp=sharing
github-actions commentedon Dec 4, 2025
This issue is stale because it has been open for 30 days with no activity.
github-actions commentedon Dec 19, 2025
This issue was closed because it has been inactive for 14 days since being marked as stale.
Tartarus0zm commentedon Dec 22, 2025
Work related to Flink integration will continue to use this issue, so I have reopened it.
x-tong commentedon Jan 6, 2026
Should we split some task lists?
30 remaining items
auron-flink-plannerjar that replacesflink-table-planner.jar#2291argoyal2212 commentedon Sep 2, 2026
Hi @Tartarus0zm @x-tong @cxzl25 @weiqingy, thanks for driving this. I went through the current Flink integration code and noticed the three phases above (Calc/Agg/Join) all target streaming.
DefaultPlannerFactorystill routesBATCHmode to Flink BatchPlanner which is unmodified.Is Flink batch support intentionally out of scope here, or just not prioritized yet? A couple of things I can suggest which could be reasonably scoped in another track:
CommonExecCalcbase between batch/stream in Flink itselfHappy to start small with a
BatchExecCalcshadow PR if there is interest.Tartarus0zm commentedon Sep 7, 2026
hi @argoyal2212 you raised a very critical issue.
From what I know, most companies use Flink for stream processing and Spark for batch processing — not many actually run batch jobs on Flink in production (we tried it once but eventually gave up). So when I designed Flink on Auron, I only considered the streaming compute scenario, and then split the work into two phases based on the challenges. The current issue covers phase one — the implementation of stateless operators. Since the Auron and Spark integration is already quite mature, while the Flink and Auron integration is still in its early stages, I personally feel that using Flink on Auron for batch computing doesn't offer any additional advantage over using Spark on Auron for batch computing — theoretically they're equivalent. So I'd recommend that most users just go directly with Spark on Auron for their batch computing needs.
If you'd like to explore this direction, I think you could start a discussion in the Auron community.
I think it's time we start discussing phase two of Flink on Auron — the stateful operators part.
@weiqingy What do you think?
[AURON apache#1264] Add BatchExecCalc shadow for Flink BATCH mode Calc
[AURON apache#1264] Add BatchExecCalc shadow for Flink BATCH mode Calc
weiqingy commentedon Sep 12, 2026
One concrete data point: we do run Flink batch in production for a convergence use case, so the scenario isn't only theoretical.
@Tartarus0zm +1 to continuing the streaming work as the main direction. Since most of the existing subtasks are close to completion, would it make sense to create the Phase 2 and Phase 3 subtasks for stateful support now, giving contributors something concrete to pick up?
Meanwhile, @argoyal2212, would you be interested in sharing a short proposal or summary of how Flink batch could leverage the existing streaming implementation? There appears to be considerable room for reuse: our shadowed
StreamExecCalcand Flink'sBatchExecCalcshare theCommonExecCalcbase, the converter layer is not stream-specific, and parts of the operator layer may also be reusable. The incremental implementation effort may therefore be relatively small, and understanding the actual cost would help us decide how to prioritize it.WDYT?
argoyal2212 commentedon Sep 14, 2026
Thanks @Tartarus0zm @weiqingy for the reply. Happy to help with the streaming work also.
I also dug into the code to answer the reuse question and wrote a short summary here. The planner base (
CommonExecCalc), the native proto plan format and the runtime operator all carry over to batch unchanged. So, each phase (Calc/Agg/Join) ends up being mostly a new shadowExecNodeclass that calls into the existing conversion and runtime code and not new engineering. The native engine, proto plan, and JNI bridge are already done.Given that, I think Calc is a good place to start since it is the smallest, safest and mirrors the
StreamExecCalcpattern directly. Happy to scope a draft PR for that if that is useful.Tartarus0zm commentedon Sep 20, 2026
hi @weiqingy Thank you so much for the addition. I also think it's time to move on to phase two's design and discussion. This involves a lot of state-related implementation work — would you be willing to lead this effort (including writing the AIP, initiating the discussion, calling for the vote, and breaking down the issues)?
Tartarus0zm commentedon Sep 20, 2026
hi @argoyal2212 Very glad to hear you're interested in Flink batch on Auron. I think this is a significant workstream — part of it will overlap with Flink Streaming (such as the Calc operator), and much of it is unrelated to Flink Streaming but highly overlapping with Spark on Auron (such as Hive source & sink, batch agg, batch join, etc. — to some extent the Calc operator is also relevant here). If we're really going to do this, I believe we need to first think through how we want to approach these parts, and only then start coding. Would you be willing to lead this effort (including writing the AIP, initiating the discussion, calling for the vote, and breaking down the issues)?
argoyal2212 commentedon Sep 23, 2026
@Tartarus0zm Happy to lead this. I will split the work into two tracks in the AIP: one reusing the Flink streaming pieces directly (Calc, parts of Agg/Join), and one that is batch specific and closer to Spark on Auron (Hive source/sink, batch agg/join, spill/shuffle), pointing each subtask at the existing code it should build on. I will share it on the dev list once ready.
argoyal2212 commentedon Sep 29, 2026
@Tartarus0zm @weiqingy started a thread : https://lists.apache.org/thread/44sh02dlvwljr9olg52ft2bvfzdsp3q7 with the AIP.