Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 25 additions & 4 deletions content/cn/docs/quickstart/toolchain/hugegraph-loader.md
Original file line number Diff line number Diff line change
Expand Up @@ -864,18 +864,39 @@ GRAPH 输入源从另一个 HugeGraph 图(通过 HugeGraph-PD 访问)读取
| `--scheduler-type` | distributed | | 自动创建图(如果不存在)时的任务调度器类型 |
| `--batch-failure-fallback` | true | | 批量插入失败时是否回退至单条插入模式 |

> 参数少于三个时 loader 会直接打印用法并退出,因此只传 `-f struct.json` 是不够的。
映射文件选项 `--file`(`-f`)是必填项,其他选项使用默认值。有效的最简调用为:

```bash
bin/hugegraph-loader.sh --file struct.json
```

启动脚本也支持 JCommander 原生的 `@file` 语法。在 `loader.args` 中把每个选项及其值分别写在单独一行:

```text
--file
struct.json
```

```bash
bin/hugegraph-loader.sh @loader.args
```

这与上面的最简调用等价。参数中的相对路径以 Loader 进程的工作目录为基准。
若参数文件包含凭据,使用 `chmod 0600 loader.args` 把权限限制为 `0600`;`@file` 调用让这些值不出现在进程参数中。
Loader 必须能够创建并写入以映射文件前缀命名的进度目录。
从只读 Kubernetes ConfigMap 挂载映射文件时,先把映射复制到可写卷中再运行 Loader。

##### 3.4.2 断点续导模式

通常情况下,Loader 任务都需要较长时间执行,如果因为某些原因导致导入中断进程退出,而下次希望能从中断的点继续导,这就是使用断点续导的场景。

用户设置命令行参数 --incremental-mode 为 true 即打开了断点续导模式。断点续导的关键在于进度文件,导入进程退出的时候,会把退出时刻的导入进度
记录到进度文件中,进度文件位于 `${struct}` 目录下,文件名形如 `load-progress_${timestamp}` ,${struct} 为映射文件的前缀,${timestamp} 为导入开始
设置 `--incremental-mode` 为 `true` 即可从进度文件续导。Loader 在导入成功或正常停止后关闭时写入进度;意外失败可能阻止该写入。
进度文件位于 `${struct}` 目录下,文件名形如 `load-progress_${timestamp}` ,${struct} 为映射文件的前缀,${timestamp} 为导入开始
的时刻,格式为 `yyyyMMdd-HHmmss`。比如:在 `2019-10-10 12:30:30` 开始的一次导入任务,使用的映射文件为 `struct-example.json`,则进度文件的路径为与 struct-example.json
同级的 `struct-example/load-progress_20191010-123030`。当目录下存在多个进度文件时,续导会读取按文件名排序的最后一个,也就是最新的那个。

> 注意:进度文件的生成与 --incremental-mode 是否打开无关,每次导入结束都会生成一个进度文件。
> 注意:导入成功或正常停止(包括达到错误上限)后会写入进度文件,与 `--incremental-mode` 或 `--print-progress` 是否打开无关。
> 意外失败导致导入中止时,可能不会写入进度文件。

如果数据文件格式都是合法的,是用户自己停止(CTRL + C 或 kill,kill -9 不支持)的导入任务,也就是说没有错误记录的情况下,下一次导入只需要设置为断点续导即可。

Expand Down
36 changes: 30 additions & 6 deletions content/en/docs/quickstart/toolchain/hugegraph-loader.md
Original file line number Diff line number Diff line change
Expand Up @@ -863,18 +863,42 @@ The import process is controlled by commands submitted by the user, and the user
| `--scheduler-type` | distributed | | The task scheduler type when creating graph if not exists |
| `--batch-failure-fallback` | true | | Whether to fallback to single insert when batch insert fails |

> The loader prints its usage and exits when it is given fewer than three arguments, so `-f struct.json` on its own is not enough.
The mapping file option `--file` (`-f`) is required; other options use their defaults. A valid minimal invocation is:

```bash
bin/hugegraph-loader.sh --file struct.json
```

The launcher also accepts JCommander's native `@file` syntax. Put each option and value on separate lines in `loader.args`:

```text
--file
struct.json
```

```bash
bin/hugegraph-loader.sh @loader.args
```

This is equivalent to the minimal invocation above. Relative path values resolve from the Loader process working directory.
If the argument file contains credentials, restrict it to mode `0600` with `chmod 0600 loader.args`.
The `@file` form keeps credential values out of process arguments.
Loader must be able to create and write the mapping-prefix directory used for progress output.
Copy a mapping mounted from a read-only Kubernetes ConfigMap to a writable volume before running Loader.

##### 3.4.2 Breakpoint Continuation Mode

Usually, the Loader task takes a long time to execute. If the import interrupt process exits for some reason, and next time you want to continue the import from the interrupted point, this is the scenario of using breakpoint continuation.

The user sets the command line parameter --incremental-mode to true to open the breakpoint resume mode. The key to breakpoint continuation lies in the progress file. When the import process exits, the import progress at the time of exit will be recorded.
Recorded in the progress file, the progress file is located in the `${struct}` directory, the file name is like `load-progress_${timestamp}`, ${struct} is the prefix of the mapping file, and ${timestamp} is the start of the import
moment, formatted as `yyyyMMdd-HHmmss`. For example, for an import task started at `2019-10-10 12:30:30`, the mapping file used is `struct-example.json`, then the path of the progress file is the same as struct-example.json
Sibling `struct-example/load-progress_20191010-123030`. When the directory holds several progress files, the resumed import reads the last one in name order, which is the most recent one.
Set `--incremental-mode` to `true` to resume from a progress file. Loader records progress when it closes normally after a successful import
or a normal stop; an unexpected failure may prevent this write.
The progress file is in `${struct}` and named `load-progress_${timestamp}`, where `${struct}` is the mapping-file prefix and `${timestamp}` is the
import start time, formatted as `yyyyMMdd-HHmmss`. For example, an import started at `2019-10-10 12:30:30` with mapping file `struct-example.json`
writes `struct-example/load-progress_20191010-123030`. When the directory holds several progress files, the resumed import reads the last one in name
order, which is the most recent one.

> Note: The generation of progress files is independent of whether --incremental-mode is turned on or not, and a progress file is generated at the end of each import.
> Note: A progress file is written after a successful import or a normal stop (including an error limit), regardless of `--incremental-mode`
> or `--print-progress`. An import aborted by an unexpected failure may not write one.

If the data file formats are all legal and the import task is stopped by the user (CTRL + C or kill, kill -9 is not supported), that is to say, if there is no error record, the next import only needs to be set to
Continue for the breakpoint.
Expand Down
Loading