Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add raw batch put/get/delete/scan #244

Merged
merged 4 commits into from
Apr 2, 2018
Merged

Add raw batch put/get/delete/scan #244

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6a83713
Add raw batch put/get/delete/scan
sunxiaoguang Mar 24, 2018
5ee163b
Change start_key_limit to each_limit
sunxiaoguang Mar 25, 2018
78dff81
Add end_keys for RawBatchScanRequest
sunxiaoguang Mar 26, 2018
2818d35
Combine start_keys and end_keys into ranges
sunxiaoguang Mar 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4,927 changes: 3,316 additions & 1,611 deletions pkg/kvrpcpb/kvrpcpb.pb.go
View file
Open in desktop

Large diffs are not rendered by default.

217 changes: 176 additions & 41 deletions pkg/tikvpb/tikvpb.pb.go
View file
Open in desktop

Large diffs are not rendered by default.

44 changes: 44 additions & 0 deletions proto/kvrpcpb.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,26 @@ message RawPutResponse {
string error = 2;
}

message RawBatchPutRequest {
Context context = 1;
repeated KvPair pairs = 2;
}

message RawBatchPutResponse {
errorpb.Error region_error = 1;
string error = 2;
}

message RawBatchGetRequest {
Context context = 1;
repeated bytes keys = 2;
}

message RawBatchGetResponse {
errorpb.Error region_error = 1;
repeated KvPair pairs = 2;
}

message RawDeleteRequest {
Context context = 1;
bytes key = 2;
Expand All @@ -261,6 +281,16 @@ message RawDeleteResponse {
string error = 2;
}

message RawBatchDeleteRequest {
Context context = 1;
repeated bytes keys = 2;
}

message RawBatchDeleteResponse {
errorpb.Error region_error = 1;
string error = 2;
}

message DeleteRangeRequest {
Context context = 1;
bytes start_key = 2;
Expand All @@ -287,13 +317,27 @@ message RawScanRequest {
Context context = 1;
bytes start_key = 2;
uint32 limit = 3;
bool key_only = 4;
}

message RawScanResponse {
errorpb.Error region_error = 1;
repeated KvPair kvs = 2;
}

message RawBatchScanRequest {
Context context = 1;
repeated bytes start_keys = 2; // start keys for each scanning range
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

em, if you want to support end keys here, I prefer using repeated KeyRange ranges.

If you just want to avoid the case that scanning key1 + limit exceeding the next key2, you can sort the start keys in the server at first like [a, b, c,], then use [a, b) as the range for the first scan, [b, c) for the second, then go on.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

End key is not for limiting records past next start key. It actually meant I'd like to stop at this end key even if number of scanned records does not go beyond limit. I prefer defining a KeyRange as well, didn't know it would be appropriate since other messages used two distinct fields. Let me update it.

repeated bytes end_keys = 3; // optional end keys for each scaning range
uint32 each_limit = 4; // max number of returning kv pairs for each scanning range
bool key_only = 5;
}

message RawBatchScanResponse {
errorpb.Error region_error = 1;
repeated KvPair kvs = 2;
}

message WriteInfo {
uint64 start_ts = 1;
Op type = 2;
Expand Down
4 changes: 4 additions & 0 deletions proto/tikvpb.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,14 @@ service Tikv {

// RawKV commands.
rpc RawGet(kvrpcpb.RawGetRequest) returns (kvrpcpb.RawGetResponse) {}
rpc RawBatchGet(kvrpcpb.RawBatchGetRequest) returns (kvrpcpb.RawBatchGetResponse) {}
rpc RawPut(kvrpcpb.RawPutRequest) returns (kvrpcpb.RawPutResponse) {}
rpc RawBatchPut(kvrpcpb.RawBatchPutRequest) returns (kvrpcpb.RawBatchPutResponse) {}
rpc RawDelete(kvrpcpb.RawDeleteRequest) returns (kvrpcpb.RawDeleteResponse) {}
rpc RawBatchDelete(kvrpcpb.RawBatchDeleteRequest) returns (kvrpcpb.RawBatchDeleteResponse) {}
rpc RawScan(kvrpcpb.RawScanRequest) returns (kvrpcpb.RawScanResponse) {}
rpc RawDeleteRange(kvrpcpb.RawDeleteRangeRequest) returns (kvrpcpb.RawDeleteRangeResponse) {}
rpc RawBatchScan(kvrpcpb.RawBatchScanRequest) returns (kvrpcpb.RawBatchScanResponse) {}

// SQL push down commands.
rpc Coprocessor(coprocessor.Request) returns (coprocessor.Response) {}
Expand Down
16 changes: 8 additions & 8 deletions src/coprocessor.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1082,14 +1082,14 @@ static file_descriptor_proto_data: &'static [u8] = b"\
\x10.kvrpcpb.ContextR\x07context\x12\x0e\n\x02tp\x18\x02\x20\x01(\x03R\
\x02tp\x12\x12\n\x04data\x18\x03\x20\x01(\x0cR\x04data\x12-\n\x06ranges\
\x18\x04\x20\x03(\x0b2\x15.coprocessor.KeyRangeR\x06ranges\"\xbc\x02\n\
\x08Response\x12K\n\x04data\x18\x01\x20\x01(\x0cR\x04dataB7\xc8\xde\x1f\
\0\xda\xde\x1f/github.com/pingcap/tipb/sharedbytes.SharedBytes\x121\n\
\x0cregion_error\x18\x02\x20\x01(\x0b2\x0e.errorpb.ErrorR\x0bregionError\
\x12)\n\x06locked\x18\x03\x20\x01(\x0b2\x11.kvrpcpb.LockInfoR\x06locked\
\x12\x1f\n\x0bother_error\x18\x04\x20\x01(\tR\notherError\x12+\n\x05rang\
e\x18\x05\x20\x01(\x0b2\x15.coprocessor.KeyRangeR\x05range\x127\n\x0cexe\
c_details\x18\x06\x20\x01(\x0b2\x14.kvrpcpb.ExecDetailsR\x0bexecDetailsB\
&\n\x18com.pingcap.tikv.kvproto\xe0\xe2\x1e\x01\xc8\xe2\x1e\x01\xd0\xe2\
\x08Response\x12K\n\x04data\x18\x01\x20\x01(\x0cR\x04dataB7\xda\xde\x1f/\
github.com/pingcap/tipb/sharedbytes.SharedBytes\xc8\xde\x1f\0\x121\n\x0c\
region_error\x18\x02\x20\x01(\x0b2\x0e.errorpb.ErrorR\x0bregionError\x12\
)\n\x06locked\x18\x03\x20\x01(\x0b2\x11.kvrpcpb.LockInfoR\x06locked\x12\
\x1f\n\x0bother_error\x18\x04\x20\x01(\tR\notherError\x12+\n\x05range\
\x18\x05\x20\x01(\x0b2\x15.coprocessor.KeyRangeR\x05range\x127\n\x0cexec\
_details\x18\x06\x20\x01(\x0b2\x14.kvrpcpb.ExecDetailsR\x0bexecDetailsB&\
\n\x18com.pingcap.tikv.kvproto\xc8\xe2\x1e\x01\xd0\xe2\x1e\x01\xe0\xe2\
\x1e\x01J\xea\x0b\n\x06\x12\x04\0\0!\x01\n\x08\n\x01\x0c\x12\x03\0\0\x12\
\n\x08\n\x01\x02\x12\x03\x01\x08\x13\n\t\n\x02\x03\0\x12\x03\x03\x07\x16\
\n\t\n\x02\x03\x01\x12\x03\x04\x07\x16\n\t\n\x02\x03\x02\x12\x03\x05\x07\
Expand Down
2 changes: 1 addition & 1 deletion src/debugpb.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5510,7 +5510,7 @@ static file_descriptor_proto_data: &'static [u8] = b"\
onsistency\x12&.debugpb.RegionConsistencyCheckRequest\x1a'.debugpb.Regio\
nConsistencyCheckResponse\"\0\x12Y\n\x10ModifyTikvConfig\x12\x20.debugpb\
.ModifyTikvConfigRequest\x1a!.debugpb.ModifyTikvConfigResponse\"\0B&\n\
\x18com.pingcap.tikv.kvproto\xc8\xe2\x1e\x01\xd0\xe2\x1e\x01\xe0\xe2\x1e\
\x18com.pingcap.tikv.kvproto\xe0\xe2\x1e\x01\xc8\xe2\x1e\x01\xd0\xe2\x1e\
\x01J\x812\n\x07\x12\x05\0\0\xc7\x01\x01\n\x08\n\x01\x0c\x12\x03\0\0\x12\
\n\x08\n\x01\x02\x12\x03\x01\x08\x0f\n\t\n\x02\x03\0\x12\x03\x03\x07\x16\
\n\t\n\x02\x03\x01\x12\x03\x04\x07\x16\n\t\n\x02\x03\x02\x12\x03\x05\x07\
Expand Down
2 changes: 1 addition & 1 deletion src/errorpb.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2328,7 +2328,7 @@ static file_descriptor_proto_data: &'static [u8] = b"\
dR\x0cstaleCommand\x12>\n\x0fstore_not_match\x18\x08\x20\x01(\x0b2\x16.e\
rrorpb.StoreNotMatchR\rstoreNotMatch\x12K\n\x14raft_entry_too_large\x18\
\t\x20\x01(\x0b2\x1a.errorpb.RaftEntryTooLargeR\x11raftEntryTooLargeB&\n\
\x18com.pingcap.tikv.kvproto\xd0\xe2\x1e\x01\xe0\xe2\x1e\x01\xc8\xe2\x1e\
\x18com.pingcap.tikv.kvproto\xe0\xe2\x1e\x01\xc8\xe2\x1e\x01\xd0\xe2\x1e\
\x01J\x8f\x10\n\x06\x12\x04\0\0:\x01\n\x08\n\x01\x0c\x12\x03\0\0\x12\n\
\x08\n\x01\x02\x12\x03\x01\x08\x0f\n\t\n\x02\x03\0\x12\x03\x03\x07\x15\n\
\t\n\x02\x03\x01\x12\x03\x04\x07\x1d\n\x08\n\x01\x08\x12\x03\x06\0(\n\
Expand Down
2 changes: 1 addition & 1 deletion src/importpb.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1564,7 +1564,7 @@ static file_descriptor_proto_data: &'static [u8] = b"\
\x8b\x01\n\tImportSST\x12?\n\x06Upload\x12\x17.importpb.UploadRequest\
\x1a\x18.importpb.UploadResponse\"\0(\x01\x12=\n\x06Ingest\x12\x17.impor\
tpb.IngestRequest\x1a\x18.importpb.IngestResponse\"\0B&\n\x18com.pingcap\
.tikv.kvproto\xe0\xe2\x1e\x01\xd0\xe2\x1e\x01\xc8\xe2\x1e\x01J\x97\x15\n\
.tikv.kvproto\xd0\xe2\x1e\x01\xe0\xe2\x1e\x01\xc8\xe2\x1e\x01J\x97\x15\n\
\x06\x12\x04\0\0C\x01\n\x08\n\x01\x0c\x12\x03\0\0\x12\n\x08\n\x01\x02\
\x12\x03\x02\x08\x10\n\t\n\x02\x03\0\x12\x03\x04\x07\x15\n\t\n\x02\x03\
\x01\x12\x03\x05\x07\x16\n\t\n\x02\x03\x02\x12\x03\x06\x07\x16\n\t\n\x02\
Expand Down
Loading