Repository navigation
[Roadmap] Incremental Rust modernization for HugeGraph core modules #3110
Description
Goal
Explore an incremental Rust implementation for selected HugeGraph Server, HStore, and PD components, especially performance- or memory-sensitive paths where Rust can provide measurable safety and efficiency benefits.
This is a parent tracking issue, not a commitment to rewrite the whole server. The current implementation remains the compatibility baseline, and Java/Rust components may coexist behind stable process, protocol, or storage boundaries.
Roadmap at a glance
Toolchain foundations
│
v
┌─ This repository ─────────────────┐
│ Server core -> HStore -> PD │
└───────────────────────────────────┘
│
v
Computer kernels -> Vermeer/OLAP
Each arrow is a review gate, not an automatic rewrite.
Proposed priority
- Server core
- Begin with bounded leaf components, memory management, codecs, or performance-sensitive data structures rather than the whole request path.
- Preserve REST/Gremlin behavior, authentication, observability, and plugin contracts.
- HStore
- Evaluate storage/replication paths only after workload baselines, on-disk and wire compatibility, failure recovery, and rolling-upgrade requirements are explicit.
- PD
- Evaluate scheduling and metadata components after their distributed-system invariants and compatibility contract are covered by repeatable tests.
The sequence may change based on benchmark evidence and maintainer consensus.
Delivery model
proposal/RFC -> isolated component -> differential/fault tests
-> benchmark -> opt-in deployment -> staged adoption
Planning guardrails (required)
Before implementation starts, each child issue must define:
- scope, owner/mentor, and non-goals;
- API, protocol, storage-format, and operational compatibility;
- differential, concurrency, fault-injection, and upgrade/rollback tests;
- reproducible latency, throughput, memory, and recovery benchmarks;
- security/dependency checks and supported platforms;
- observability, deployment, coexistence, migration, and rollback plans;
- explicit acceptance criteria and user/operator/contributor documentation.
No large rewrite PR should start before its design and acceptance criteria are reviewed. Correctness, recoverability, and compatibility take precedence over headline benchmark numbers. Small, reviewable PRs and stable interoperability boundaries are required; an unplanned rewrite PR may be closed or asked to return to the RFC/design stage.
Candidate work items
- Select the first small server-side proof of concept
- Document stable Java/Rust interoperability boundaries
- Build shared differential and compatibility test fixtures
- Establish representative OLTP and storage workload baselines
- Define CI, packaging, security, and release conventions
- Define rolling-upgrade and rollback requirements before HStore/PD work
- Split approved work into newcomer-friendly child issues
This checklist will link to concrete child issues as proposals are accepted.
Project-wide roadmap
- Toolchain: [Roadmap] Incremental Rust modernization for HugeGraph Toolchain hugegraph-toolchain#748
- Server / HStore / PD (this issue): [Roadmap] Incremental Rust modernization for HugeGraph core modules #3110
- Computer / Vermeer: [Roadmap] Incremental Rust modernization for graph computing hugegraph-computer#355
- Documentation hub: [Roadmap] Documentation hub for HugeGraph Rust modernization hugegraph-doc#462
How to participate
Interested in open source, Rust, databases, distributed systems, storage engines, or infrastructure? Comment with the area you want to work on and any relevant experience. RFC review, failure-model analysis, benchmarks, tests, and documentation are all valuable.
First-time contributors can start with the HugeGraph contribution guide and the repository's good first issue / help wanted tasks.
Activity
dpol1 commentedon Jul 25, 2026
I'd be interested in taking the interop/build/test foundation work before proposing a server component.
Fair warning, I'm not a Rust person. My work here has been compatibility and failure handling (#3062). Where I'd be useful is the build, test and licensing plumbing someone has to do anyway.
Poked around a bit first. Few things that might save time for whoever picks this up:
os.detected.classifiertoosx-x86_64on ARM Macs, so anything keyed on that property would need handling there.On picking the first PoC, I'd rather not name a component. A small matrix (test oracle? how many call sites? work per call? does it touch a persisted format?) plus a rule for when a JNI hop pays for itself makes it easy to move on when the first pick doesn't hold up.
Small first thing, no Rust until there's agreement: a compat inventory of the codec classes, a golden fixture corpus in plain Java that can merge on its own, and an interop draft (JNI seam, failure semantics, packaging) as a child issue.
Questions: does this fit the toolchain -> server ordering, or should it wait behind apache/hugegraph-toolchain#748? Park packaging for now, or is the classifier jar shape what you'd want? And is anyone up for mentoring / picking the first component?
imbajin commentedon Jul 26, 2026
@dpol1 Thank you very much for digging into this and for offering to take on the interoperability, build, test, and licensing foundations. The community very much welcomes your participation and these suggestions.
A few updates and clarifications from our side:
The community is already working on reducing some of the historical compatibility and technology-stack burden in #3117. If that proceeds smoothly, we hope to move the project baseline to Java 17 soon. Without the current TinkerPop version constraint, Java 21 would also be feasible, but with the existing dependency relationship we will likely need to wait for TinkerPop 4.0.
hugegraph-commonsno longer needs to remain Java 8 compatible and can move to Java 11.hugegraph-clientis the only component where we may still need to consider older-Java compatibility; the other components do not have that requirement.For the initial Rust work, supporting x86_64 and ARM/aarch64 is sufficient. We do not need to cover other architectures for now. On unsupported or incompatible environments, a graceful fallback to the existing Java implementation is the preferred direction.
The component priority is to start with Toolchain because it has the smallest dependency and compatibility surface, is relatively independent, and has clear potential benefits from a CLI-oriented implementation. This does not mean storage, server, or graph-computing exploration cannot proceed in parallel; those areas simply need more caution because of their compatibility and correctness impact.
We completely agree that the surrounding foundations and evaluation work can start now. Compatibility inventories, golden fixtures, test oracles, differential validation, the PoC selection matrix, and an interoperability draft would all be useful and can proceed in parallel. There is no need to wait before starting the testing and validation work.
We will also discuss and update the specific mentor, initial component, and related coordination details as the proposal moves forward. Those decisions do not need to block the work you outlined here, so please feel free to continue with the research, preparation, compatibility inventory, fixtures, and testing in parallel.
We would also be very happy to have contributors with deeper Rust experience join the design review and validation, so that the approach can be confirmed collectively and evolved incrementally and safely.
Overall, this is a very useful and pragmatic proposal. Thank you again for the careful investigation and for volunteering to help build the foundations.
dpol1 commentedon Jul 27, 2026
Thanks @imbajin , this clarifies the scope. I'll treat Java 17/commons Java 11 as intended target baselines and build against the current ones until they land, x86_64+aarch64 as the initial native targets, and graceful Java fallback as a requirement. The fallback point is useful as a selection criterion too: a candidate only qualifies if keeping the Java implementation alive stays cheap, which rules out a few things I had on my list. I'll start with a child issue containing the compatibility inventory and selection matrix, followed by a Java-only golden-fixture PR. JNI/Cargo/product wiring can wait for component, mentor, and Rust design review.
SebastianGruza commentedon Sep 20, 2026
I'll take the two checklist items that sit before HStore and PD in the gate order: representative OLTP and storage workload baselines and rolling-upgrade and rollback requirements before HStore/PD work. No overlap with #3145: codec fixtures and the PoC matrix stay there; I supply the numbers and the requirements that matrix asks for in its "oracle" and "fallback cost" columns for the storage layer.
The material already exists, it is just scattered: over the last two weeks I measured on a 3-node cluster (chart from hugegraph#221) a Server rolling upgrade under load in three passes, PD behaviour after its disk fills up (#3222), the Store memory profile under the cluster preset, and readiness (#3221), always with an oracle checking that no acknowledged write was lost. The child issue below turns that into a repeatable set: one data generator shared with apache/hugegraph-toolchain#772 (there the loader is measured as a client, here the server and the storage), a fixed OLTP mix, HStore storage metrics, and written upgrade/rollback requirements with a harness that checks them. All Java and scripts, no Rust; it is the reference point without which no HStore or PD PoC has anything to compare against.
Child issue with scope, non-goals and acceptance criteria: #3223.