Skip to content

Commit

Permalink
docs(develop-guide): doc for data format and peristence (GreptimeTeam…
Browse files Browse the repository at this point in the history
  • Loading branch information
v0y4g3r authored Oct 12, 2023
1 parent ced0898 commit 7503df8
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 16 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ Second, data of the same column tends to be homogeneous which helps with compres

## Data Persistence

When the size of data buffered in MemTable reaches a threshold, the MemTable will be flushed to a SST file.
GreptimeDB provides a configuration item `storage.flush.global_write_buffer_size`, which is flush threshold of the total memory usage for all MemTables.

When the size of data buffered in MemTables reaches that threshold, GreptimeDB will pick MemTables and flush them to SST files.

## Indexing Data in SST Files

Expand Down
31 changes: 16 additions & 15 deletions docs/zh/v0.4/developer-guide/datanode/data-persistence-indexing.md
Original file line number Diff line number Diff line change
@@ -1,36 +1,37 @@
# Data Persistence and Indexing
# 数据持久化与索引

Similar to all LSMT-like storage engines, data in MemTables is persisted to durable storage, for example, the local disk file system or object storage service. GreptimeDB adopts [Apache Parquet][1] as its persistent file format.
与所有类似LSMT的存储引擎一样,MemTables中的数据被持久化到耐久性存储,例如本地磁盘文件系统或对象存储服务。GreptimeDB采用[Apache Parquet][1]作为其持久文件格式。

## SST File Format
## SST 文件格式

Parquet is an open source columnar format that provides fast data querying and has already been adopted by many projects, such as Delta Lake.
Parquet 是一种提供快速数据查询的开源列式存储格式,已经被许多项目采用,例如 Delta Lake

Parquet has a hierarchical structure like "row groups-columns-data pages". Data in a Parquet file is horizontally partitioned into row groups, in which all values of the same column are stored together to form a data page. Data page is the minimal storage unit. This structure greatly improves performance.
Parquet 具有层次结构,类似于“行组-列-数据页”。Parquet 文件中的数据被水平分区为行组(row group),在其中相同列的所有值一起存储以形成数据页(data pages)。数据页是最小的存储单元。这种结构极大地提高了性能。

First, clustering data by column makes file scanning more efficient, especially when only a few columns are queried, which is very common in analytical systems.
首先,数据按列聚集,这使得文件扫描更加高效,特别是当查询只涉及少数列时,这在分析系统中非常常见。

Second, data of the same column tends to be homogeneous which helps with compression when apply techniques like dictionary and Run-Length Encoding (RLE).
其次,相同列的数据往往是同质的(比如具备近似的值),这有助于在采用字典和 Run-Length EncodingRLE)等技术进行压缩。

![Parquet file format](/parquet-file-format.png)

## Data Persistence
## 数据持久化

When the size of data buffered in MemTable reaches a threshold, the MemTable will be flushed to a SST file.
GreptimeDB 提供了 `storage.flush.global_write_buffer_size` 的配置项来设置全局的 Memtable 大小阈值。当数据库所有 MemTable 中的数据量之和达到阈值时将自动触发持久化操作,将 MemTable 的数据 flush 到 SST 文件中。

## Indexing Data in SST Files

Apache Parquet file format provides inherent statistics in headers of column chunks and data pages, which are used for pruning and skipping.
## 在 SST 文件的索引

Apache Parquet文件格式在列块和数据页的头部提供了内置的统计信息,用于剪枝和跳过。

![Column chunk header](/column-chunk-header.png)

For example, in the above Parquet file, if you want to filter rows where `name` = `Emily`, you can easily skip row group 0 because the max value for `name` field is `Charlie`. This statistical information reduces IO operations.
例如,在上述 Parquet 文件中,如果你想要过滤 `name` 等于 `Emily` 的行,你可以轻松跳过行组 0,因为 `name` 字段的最大值是 `Charlie`。这些统计信息减少了 IO 操作。

Besides Parquet's built-in statistics, our team is working on supporting a separate index file that utilizes some time-series specific indexing techniques to improve scanning performance.
除了 Parquet 内置的统计信息外,我们的团队正在开发支持使用一些时间序列特定的索引技术的单独索引文件,以提高扫描性能。

## Unified Data Access Layer: OpenDAL
## 统一数据访问层:OpenDAL

GreptimeDB uses [OpenDAL][2] to provide a unified data access layer, thus, the storage engine does not need to interact with different storage APIs, and data can be migrated to cloud-based storage like AWS S3 seamlessly.
GreptimeDB使用 [OpenDAL][2] 提供统一的数据访问层,因此,存储引擎无需与不同的存储 API 交互,数据可以无缝迁移到基于云的存储,如AWS S3。

[1]: https://parquet.apache.org
[2]: https://github.com/datafuselabs/opendal

0 comments on commit 7503df8

Please sign in to comment.