Skip to content

docs: explain ToplingDB runtime setup (2/4) - #510

Open
imbajin wants to merge 9 commits into
apache:masterfrom
hugegraph:task/topling-split-docs
Open

imbajin wants to merge 9 commits into
apache:masterfrom
hugegraph:task/topling-split-docs

Conversation

@imbajin

@imbajin imbajin commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

Document the optional ToplingDB runtime for Server, PD and Store, matching ASF runtime PR #3275.

  • Explain component-local preparation, provider selection and external persistent directories.
  • Show init-store.sh before the first startup of a fresh standalone Server graph, in both English and Chinese.
  • Preserve existing routes and historical articles; register the guide for the development version only.

Validation: bilingual commands checked against the runtime scripts; repository link validation passed. This content check does not replace runtime or full HStore acceptance.

Coordinate both merges with #3275.

Conceptual overview

Conceptual documentation overview of optional ToplingDB: matching the local runtime and provider, persistent paths and first initialization. This diagram is not runtime or release acceptance evidence.

Optional ToplingDB runtime and provider consistency, persistent paths and initialization

- retain historical interfaces and public routes
- explain unreleased runtime and recovery boundaries
- register bilingual latest guides in the route map

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The isolated-data example for the standalone Server moves only rocksdb.data_path, so the WAL stays inside the generated -topling distribution directory, where re-running the documented generator deletes it. The new guide is also missing from both SUMMARY.md indexes. Evidence: static review of the diff at b2e9a63 against the implementation in hugegraph/hugegraph#264 at 9af9d14 (RocksDBOptions.WAL_PATH, bin/rocksdb-server-config.sh server_verify_graph_roots, install-dist/scripts/build-topling-distribution.sh publish and cleanup steps) and apache/hugegraph RocksDBStdSessions write and close paths; content/en|cn/docs/SUMMARY.md and commit 3287b60.

Comment thread content/en/docs/guides/toplingdb.md Outdated
Comment thread content/en/docs/guides/toplingdb.md

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

审查基于文档 head b2e9a63,并与实现 PR264 的 9af9d146659a2c8f290d231be7d0777421585f38 逐项交叉核对。建议修复下述问题后再审,中英文应同步修改。WAL 配置遗漏已有评审指出,本轮不重复新增该行级评论,沿用 #510 (comment) ;本次新增目录保护覆盖范围的行级意见。

评分(每项 20):设计 17、逻辑 14、内容/代码质量 17、测试 12、易用性 12,总分 7.2/10。

优点:保留历史接口和路由、显式限定尚未发布的开发版、仅注册 latest 路由;三组件/JNI 所有权边界、外部产物与 SHA-256 的能力限制、非原子全图恢复、断电持久性和回退备份要求都表述克制。未把镜像标签、/versions 或构建成功说成原生验收。

需要修复:独立目录示例漏掉 WAL,按文档首次初始化会在 JVM 前失败;目录保护的保证超出了当前裸包 PD/Store 实现,不能让读者把未经保护的启动路径当作已拒绝错误数据根。配套实现目前的 Topling Docker 构建也尚未通过,应继续保留交付状态限制并协调合入。

本轮验证:validate-links 通过;Python 3.13 的链接、站点输出、Hugo wrapper 聚焦测试 50 项通过;真实 shell 预检查复现了文档配置退出 1,补充独立 WAL 并创建目录后退出 0。latest-only 路由、中英文命令对齐、三个监控回环地址及默认关闭配置均已核对。

验证边界:未下载或执行外部 JNI,未做真实服务验收;本环境缺 Hugo/WebP,未跑严格 Hugo 构建和浏览器验收。全量 218 项源码测试有 8 errors、1 skipped,错误来自缺工具以及云环境 /tmp/.git 触发输出目录安全检查,不作为本 PR 引入的回归。git diff --check 另提示两个历史博客 _index.md 新增 EOF 空行,可顺手清理。发布合规仍是单独的发版核查事项,本文提醒不能替代许可/声明材料审查。

Comment thread content/cn/docs/guides/toplingdb.md Outdated

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The previously reported WAL and directory-guard mismatches remain on this head; this pass also found the active Docs sidebar omits the guide and replacing the JNI can leave its runtime checksum stale. Evidence: diff b2e9a63 compared with versioning.py, docs_nav.json, and the Topling build and preparation scripts at hugegraph/hugegraph#264 head 9af9d146.

Comment thread content/en/docs/guides/toplingdb.md
Comment thread content/en/docs/guides/toplingdb.md Outdated
Comment thread content/en/docs/guides/toplingdb.md Outdated
- persist both data and WAL outside generated packages
- expose the guide in summaries and active navigation
- document runtime manifest regeneration
- state component ownership and shutdown order
- remove trailing blog blank lines

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The standalone Server steps fail on first use: the Topling launcher check requires the configured data and WAL directories to exist already, and the guide only says to mount or create the parent root. Evidence: static review of the diff at 6c0f2da against the implementation in hugegraph/hugegraph#264 at 36058b0 (bin/preload-topling.sh, bin/rocksdb-server-config.sh server_verify_graph_roots and server_verify_storage_path, bin/verify-rocksdb-provider.sh); running server_verify_graph_roots from that head on this page's properties, with flock stubbed and the paths under a temp root where only the server root exists, fails with 'RocksDB data path must be an existing directory: .../server/data'.

Comment thread content/en/docs/guides/toplingdb.md Outdated
- explain trusted preparation and explicit selection
- preserve external data WAL and Raft paths
- separate packaging and recovery follow-ups
- link the clean three-component implementation

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: At 7d24a52 the EN and CN guides match the paired implementation hugegraph/hugegraph#266 (head 3ec2871): provider keys and files, data/WAL/Raft keys, prepare-topling.sh checks, launcher selection and the fail-before-open provider check. The earlier WAL, mkdir, SUMMARY, sidebar, ownership-guard and stop-sequence threads are addressed. One minor gap remains in the HStore start step. Evidence: read the full diff against merge base e6389aa; checked preload-topling.sh, prepare-topling.sh, the PD/Store/Server launchers, init-store.sh, dump-store.sh, RocksDBRuntime, RocksDBStdSessions.openRocksDB, StoreNodeApplication, HugePDServer, the PD/Store application.yml and application-pd.yml, Store AppConfig and toplingdb.yaml at 3ec2871; git diff --check is clean; all site build checks pass at this head.

Comment thread content/en/docs/guides/toplingdb.md Outdated
@imbajin imbajin changed the title docs: document ToplingDB development integration docs: explain ToplingDB runtime setup (2/4) Oct 5, 2026
- clear the inherited Topling opt-in before Server start
- document rocksdb as an explicit alternative
- align the English and Chinese HStore instructions
- align both guides with the current master
- state Java 17 and Maven requirements
- retain HStore provider environment isolation
- require a Java 17 JDK for preparation
- explain real EasyMigrate configuration checks
- bind launch verification to prepared libraries
- reject competing RocksDB JNI preloads
- describe default and explicit standard startup checks
- retain unrelated preload compatibility
- align both guides with runtime selection behavior

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: At 713667a the EN and CN ToplingDB guides match the paired implementation hugegraph/hugegraph#266 at head a8dd703, and every earlier review thread (WAL path, directory creation, SUMMARY and sidebar entries, ownership-guard scope, stale JNI checksum, stop order, HStore Server provider reset) is addressed; merge timing still needs to be coordinated with #266 as the PR body says. Evidence: read the full diff against merge base e6389aa; checked provider keys in PD conf/application.yml, Store conf/application-pd.yml (included through spring.profiles.include: pd) and Server RocksDBOptions, data/WAL/Raft path keys, bin/prepare-topling.sh (hash, marker, ELF, ldd, EasyMigrate probe, refusal of an existing topling directory, checksum receipt), bin/preload-topling.sh (explicit TOPLINGDB_ROCKSDB_PROVIDER selection, competing JNI rejection, receipt check, conf/toplingdb.yaml default), RocksDBRuntime.verify, the PD/Store assembly descriptors that ship the scripts and toplingdb.yaml, and the shipped profile (auto_start_http: false, memtable_as_log_index: false) at a8dd703; scripts/test_versioning.py and scripts/test_validate_links.py pass (87 tests) under Python 3.12; git diff --check is clean; all site build checks pass on this head.

- describe readable runtime directory and file modes
- explain different preparation and service accounts
- retain parent-directory access responsibility

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: At da23367 the EN and CN ToplingDB guides still match the paired implementation hugegraph/hugegraph#266, now at head 7ee8766. The only change since the previous approval at 713667a is the runtime-permission paragraph, and it matches the new chmod step in bin/prepare-topling.sh (directories 0755, regular files 0644, parent directories untouched). Merge timing still needs to be coordinated with #266, as the PR body says. Evidence: read the full diff against merge base e6389aa and the 713667a..da23367 delta. At 7ee8766 I checked bin/prepare-topling.sh (hash, marker, single JNI entry, ELF header, ldd, Java 17 EasyMigrate probe, refusal of an existing topling directory, checksum receipt, find/chmod before mv -T) and bin/preload-topling.sh (TOPLINGDB_ROCKSDB_PROVIDER selection, competing JNI rejection, receipt check, conf/toplingdb.yaml default). I also checked the PD conf/application.yml and Store conf/application-pd.yml provider keys, the pd.data-path, app.data-path and app.raft-path keys (AppConfig), the RocksDBRuntime.verify call sites, the PD/Store assembly descriptors and the shipped toplingdb.yaml (auto_start_http: false, memtable_as_log_index: false). Since a8dd703, #266 changed only those chmod lines among the runtime scripts and configs. scripts/test_versioning.py and scripts/test_validate_links.py pass (87 tests, Python 3.12), dist/validate-links.sh passes, git diff --check is clean, and all site build checks pass on this head.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: Fresh standalone Server instructions omit backend initialization before the first start. Evidence: the current Server quickstart and paired Topling core PR require bin/init-store.sh first.

Comment thread content/en/docs/guides/toplingdb.md Outdated
- show initialization before the first server start
- align English and Chinese switching steps
- link the matching ASF runtime pull request

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The multi-graph example reuses identical RocksDB data and WAL roots across local graph configurations. Evidence: Static comparison with HugeGraph #3275 at head 96f99de73820af6a4b41f09df1423da6302ab58a shows wrapPath appends only the store name and the graph suffix is applied only during cloning; the inline comment links the exact paths.

| Store data | `conf/application.yml`: `app.data-path` | `/srv/hugegraph/topling/store/data` |
| Store Raft | `conf/application.yml`: `app.raft-path` | `/srv/hugegraph/topling/store/raft` |

For a standalone Server, edit every selected graph configuration, for example `conf/graphs/hugegraph.properties`:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Important: This tells readers to edit every selected graph config but gives the same data and WAL roots for all of them. In paired Server PR #3275, RocksDBStore.wrapPath() appends only the store name, while RocksDBStoreProvider.onCloneConfig() adds a graph suffix only when cloning; the local conf/graphs/*.properties files are loaded independently. Reusing this example across two graphs therefore opens the same RocksDB roots and the second graph cannot open them. Please state that each graph needs unique data and WAL paths, show graph-specific directories, and sync the Chinese guide. Evidence: config scan, local graph loading, path wrapping, clone suffix.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants