Add raw batch put/get/delete/scan #244

sunxiaoguang · 2018-03-22T03:09:10Z

Add batch operations to raw kv, which can significantly reduce gRPC cpu utilization for bulk put and reduce number of roundtrips for reads.

CLAassistant · 2018-03-22T03:09:17Z

All committers have signed the CLA.

AndreMouche · 2018-03-22T03:21:21Z

proto/kvrpcpb.proto

 }

 message RawScanResponse {
    errorpb.Error region_error = 1;
    repeated KvPair kvs = 2;
 }

+message RawBatchScanRequest {
+    Context context = 1;
+    repeated bytes start_keys = 2;


It confused me, could you add some comments for this message?

It's for scanning multiple keys with different prefixes at once. start_key_limit is the limit of pairs for each key prefix, and total_limit is the limit of matched pairs for all prefixes. Normally it is the same as (number of prefixes * start_key_limit) but it could be smaller than that if it's necessary

Here is the reason why we wanted these. We are trying to use TiKV as a distributed storage backend and build bigtable data model on top of raw kv interface. And we mapped column family + row key + qualifier + timestamp as raw key. So to query the most recent version of several qualifiers of the same row key, BatchScan can be used in this scenario to scan records at once.

can you add comments to explain start_keys and each_limit here?

They are not understood intuitively.

Sure, let me add one more field all together at last. Since @zhangjinpeng1987 mentioned there could be duplicated results with this, it's better to have optional end keys as well.

zhangjinpeng87 · 2018-03-26T02:31:01Z

LGTM

siddontang · 2018-03-27T06:54:58Z

proto/kvrpcpb.proto

 }

 message RawScanResponse {
    errorpb.Error region_error = 1;
    repeated KvPair kvs = 2;
 }

+message RawBatchScanRequest {
+    Context context = 1;
+    repeated bytes start_keys = 2; // start keys for each scanning range


em, if you want to support end keys here, I prefer using repeated KeyRange ranges.

If you just want to avoid the case that scanning key1 + limit exceeding the next key2, you can sort the start keys in the server at first like [a, b, c,], then use [a, b) as the range for the first scan, [b, c) for the second, then go on.

End key is not for limiting records past next start key. It actually meant I'd like to stop at this end key even if number of scanned records does not go beyond limit. I prefer defining a KeyRange as well, didn't know it would be appropriate since other messages used two distinct fields. Let me update it.

siddontang · 2018-03-27T10:02:37Z

LGTM

zhangjinpeng87 · 2018-03-28T02:31:37Z

LGTM

My name is Sun Xiaoguang from Zhihu, where I help make a BigTable like system focusd on OLTP that is using TiKV. I've been contributing to TiKV and the library ecosystem for around a year now. I first touched the TiKV codebase since around March 2018. As part of my involvement with the project I've contributed several improvements to the TiKV Project: * [Added raw batch put/get/delete/scan](pingcap/kvproto#244) * [Changed `get_region` and `get_region_info` to use a common implementation](#3521) * [Print help message when starting tikv-ctl without argument](#3531) * [Clean up unused #[allow] directives](#3553) I also am the first and primary author of the [TiKV Rust Client](https://github.com/tikv/client-rust). You can see related PRs for that here: * [TiKV Rust Client RFC](tikv/rfcs#7) * [The initial version of Raw KV Implementation](tikv/client-rust#14) * [Change `raw::Client::get` to return `Option<Value>`](tikv/client-rust#24) * [Remove cf and batch_scan from `example/raw.rs`](tikv/client-rust#21) I also made the first PR to the mock TiKV project: [A basic implementation of mock TiKV API surface](tikv/mock-tikv#1) As part of my involvement, the TiKV team inquired if I was interested in becoming a maintainer for the project. This PR adds me to the list of maintainers. I have read and understand the expectations of maintainers described in the `GOVERNANCE.md` file.

My name is Sun Xiaoguang from Zhihu, where I help make a BigTable like system focusd on OLTP that is using TiKV. I've been contributing to TiKV and the library ecosystem for around a year now. I first touched the TiKV codebase since around March 2018. As part of my involvement with the project I've contributed several improvements to the TiKV Project: * [Added raw batch put/get/delete/scan](pingcap/kvproto#244) * [Changed `get_region` and `get_region_info` to use a common implementation](#3521) * [Print help message when starting tikv-ctl without argument](#3531) * [Clean up unused #[allow] directives](#3553) I also am the first and primary author of the [TiKV Rust Client](https://github.com/tikv/client-rust). You can see related PRs for that here: * [TiKV Rust Client RFC](tikv/rfcs#7) * [The initial version of Raw KV Implementation](tikv/client-rust#14) * [Change `raw::Client::get` to return `Option<Value>`](tikv/client-rust#24) * [Remove cf and batch_scan from `example/raw.rs`](tikv/client-rust#21) I also made the first PR to the mock TiKV project: [A basic implementation of mock TiKV API surface](tikv/mock-tikv#1) As part of my involvement, the TiKV team inquired if I was interested in becoming a maintainer for the project. This PR adds me to the list of maintainers. I have read and understand the expectations of maintainers described in the `GOVERNANCE.md` file. Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

@sunxiaoguang

* Title: Add @sunxiaoguang to MAINTAINERS.md My name is Sun Xiaoguang from Zhihu, where I help make a BigTable like system focusd on OLTP that is using TiKV. I've been contributing to TiKV and the library ecosystem for around a year now. I first touched the TiKV codebase since around March 2018. As part of my involvement with the project I've contributed several improvements to the TiKV Project: * [Added raw batch put/get/delete/scan](pingcap/kvproto#244) * [Changed `get_region` and `get_region_info` to use a common implementation](#3521) * [Print help message when starting tikv-ctl without argument](#3531) * [Clean up unused #[allow] directives](#3553) I also am the first and primary author of the [TiKV Rust Client](https://github.com/tikv/client-rust). You can see related PRs for that here: * [TiKV Rust Client RFC](tikv/rfcs#7) * [The initial version of Raw KV Implementation](tikv/client-rust#14) * [Change `raw::Client::get` to return `Option<Value>`](tikv/client-rust#24) * [Remove cf and batch_scan from `example/raw.rs`](tikv/client-rust#21) I also made the first PR to the mock TiKV project: [A basic implementation of mock TiKV API surface](tikv/mock-tikv#1) As part of my involvement, the TiKV team inquired if I was interested in becoming a maintainer for the project. This PR adds me to the list of maintainers. I have read and understand the expectations of maintainers described in the `GOVERNANCE.md` file. Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

@sunxiaoguang

* Title: Add @sunxiaoguang to MAINTAINERS.md My name is Sun Xiaoguang from Zhihu, where I help make a BigTable like system focusd on OLTP that is using TiKV. I've been contributing to TiKV and the library ecosystem for around a year now. I first touched the TiKV codebase since around March 2018. As part of my involvement with the project I've contributed several improvements to the TiKV Project: * [Added raw batch put/get/delete/scan](pingcap/kvproto#244) * [Changed `get_region` and `get_region_info` to use a common implementation](tikv#3521) * [Print help message when starting tikv-ctl without argument](tikv#3531) * [Clean up unused #[allow] directives](tikv#3553) I also am the first and primary author of the [TiKV Rust Client](https://github.com/tikv/client-rust). You can see related PRs for that here: * [TiKV Rust Client RFC](tikv/rfcs#7) * [The initial version of Raw KV Implementation](tikv/client-rust#14) * [Change `raw::Client::get` to return `Option<Value>`](tikv/client-rust#24) * [Remove cf and batch_scan from `example/raw.rs`](tikv/client-rust#21) I also made the first PR to the mock TiKV project: [A basic implementation of mock TiKV API surface](tikv/mock-tikv#1) As part of my involvement, the TiKV team inquired if I was interested in becoming a maintainer for the project. This PR adds me to the list of maintainers. I have read and understand the expectations of maintainers described in the `GOVERNANCE.md` file. Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

sunxiaoguang mentioned this pull request Mar 22, 2018

Add raw batch put/get/delete/scan tikv/tikv#2854

Merged

AndreMouche reviewed Mar 22, 2018

View reviewed changes

sunxiaoguang force-pushed the raw_batch branch 3 times, most recently from 05f2210 to ea82cbf Compare March 24, 2018 06:41

zhangjinpeng87 approved these changes Mar 26, 2018

View reviewed changes

zhangjinpeng87 changed the title ~~Add raw batch put/get/delete/scan~~ [DNM]Add raw batch put/get/delete/scan Mar 26, 2018

sunxiaoguang added 3 commits March 26, 2018 21:10

Add raw batch put/get/delete/scan

6a83713

Change start_key_limit to each_limit

5ee163b

Add end_keys for RawBatchScanRequest

78dff81

sunxiaoguang force-pushed the raw_batch branch from e1b239e to 78dff81 Compare March 26, 2018 13:24

siddontang reviewed Mar 27, 2018

View reviewed changes

Combine start_keys and end_keys into ranges

2818d35

zhangjinpeng87 changed the title ~~[DNM]Add raw batch put/get/delete/scan~~ Add raw batch put/get/delete/scan Mar 28, 2018

breezewish merged commit 1c5dddd into pingcap:master Apr 2, 2018

sunxiaoguang mentioned this pull request Mar 19, 2019

Add @sunxiaoguang to MAINTAINERS.md tikv/tikv#4397

Merged

sticnarf pushed a commit to sticnarf/kvproto that referenced this pull request Oct 27, 2019

Add raw batch put/get/delete/scan (pingcap#244)

17a153c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add raw batch put/get/delete/scan #244

Add raw batch put/get/delete/scan #244

sunxiaoguang commented Mar 22, 2018

CLAassistant commented Mar 22, 2018 •

edited

Loading

AndreMouche Mar 22, 2018

sunxiaoguang Mar 22, 2018

sunxiaoguang Mar 22, 2018 •

edited

Loading

siddontang Mar 26, 2018

sunxiaoguang Mar 26, 2018

zhangjinpeng87 commented Mar 26, 2018

siddontang Mar 27, 2018

sunxiaoguang Mar 27, 2018

siddontang commented Mar 27, 2018

zhangjinpeng87 commented Mar 28, 2018

Add raw batch put/get/delete/scan #244

Add raw batch put/get/delete/scan #244

Conversation

sunxiaoguang commented Mar 22, 2018

CLAassistant commented Mar 22, 2018 • edited Loading

AndreMouche Mar 22, 2018

Choose a reason for hiding this comment

sunxiaoguang Mar 22, 2018

Choose a reason for hiding this comment

sunxiaoguang Mar 22, 2018 • edited Loading

Choose a reason for hiding this comment

siddontang Mar 26, 2018

Choose a reason for hiding this comment

sunxiaoguang Mar 26, 2018

Choose a reason for hiding this comment

zhangjinpeng87 commented Mar 26, 2018

siddontang Mar 27, 2018

Choose a reason for hiding this comment

sunxiaoguang Mar 27, 2018

Choose a reason for hiding this comment

siddontang commented Mar 27, 2018

zhangjinpeng87 commented Mar 28, 2018

CLAassistant commented Mar 22, 2018 •

edited

Loading

sunxiaoguang Mar 22, 2018 •

edited

Loading