storage: keyspace management #5265

AmoebaProtozoa · 2022-07-05T05:32:19Z

Signed-off-by: David 8039876+AmoebaProtozoa@users.noreply.github.com

What problem does this PR solve?

Added storage methods for saving and loading keyspace metadata
Issue Number: ref #5293

What is changed and how does it work?

storage: keyspace management

Check List

Tests

Unit test

Code changes

Release note

None

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

ti-chi-bot · 2022-07-05T05:32:22Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

AndreMouche
nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

server/storage/endpoint/key_path.go

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

server/storage/endpoint/keyspace.go

server/storage/keyspace_test.go

server/storage/endpoint/key_path.go

server/storage/endpoint/keyspace.go

server/storage/keyspace_test.go

server/storage/endpoint/keyspace.go

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

server/storage/endpoint/keyspace.go

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

server/storage/endpoint/key_path.go

AndreMouche

rest LGTM

server/storage/endpoint/key_path.go

AndreMouche · 2022-08-03T07:16:41Z

server/storage/endpoint/keyspace.go

+// SaveKeyspace saves the given keyspace to the storage.
+func (se *StorageEndpoint) SaveKeyspace(keyspace *keyspacepb.KeyspaceMeta) error {
+	key := KeyspaceMetaPath(keyspace.GetId())
+	return se.saveProto(key, keyspace)


Could we save json instead to make it read/debug-friendly?

Here some information defined in proto. maybe proto more suitable.

server/storage/endpoint/key_path.go

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

go.mod

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

codecov · 2022-08-07T06:26:18Z

Codecov Report

Merging #5265 (67dc293) into master (99e38eb) will increase coverage by 0.07%.
The diff coverage is 79.45%.

@@            Coverage Diff             @@
##           master    #5265      +/-   ##
==========================================
+ Coverage   75.58%   75.66%   +0.07%     
==========================================
  Files         312      313       +1     
  Lines       31061    31104      +43     
==========================================
+ Hits        23479    23536      +57     
+ Misses       5572     5556      -16     
- Partials     2010     2012       +2

Flag	Coverage Δ
unittests	`75.66% <79.45%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/grpc_service.go	`48.81% <ø> (-0.60%)`	⬇️
server/storage/storage.go	`71.15% <ø> (ø)`
server/tso/global_allocator.go	`61.25% <ø> (ø)`
server/storage/endpoint/keyspace.go	`73.33% <73.33%> (ø)`
server/schedule/filter/filters.go	`91.64% <79.41%> (-0.43%)`	⬇️
server/storage/endpoint/key_path.go	`100.00% <100.00%> (ø)`
server/schedulers/shuffle_hot_region.go	`55.55% <0.00%> (-10.11%)`	⬇️
server/storage/kv/etcd_kv.go	`79.71% <0.00%> (-5.80%)`	⬇️
server/storage/endpoint/meta.go	`63.63% <0.00%> (-2.28%)`	⬇️
server/region_syncer/server.go	`86.26% <0.00%> (-1.10%)`	⬇️
... and 16 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

rleungx

The rest LGTM.

server/storage/endpoint/key_path.go

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

nolouch · 2022-08-08T03:59:01Z

ci failed. maybe you need run make tidy.

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

nolouch

lgtm

nolouch · 2022-08-08T05:59:07Z

/merge

ti-chi-bot · 2022-08-08T05:59:08Z

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-08-08T05:59:10Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: 67dc293

ref #5265, ref #5291, ref #5293, ref #5297 Add APIv2 HTTP interface for keyspace management. Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

AmoebaProtozoa added 2 commits July 5, 2022 13:08

storage: keyspace storage

54a9a53

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

storage: encode spaceID string as base 10

235d463

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

ti-chi-bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. labels Jul 5, 2022

ti-chi-bot requested review from JmPotato and nolouch July 5, 2022 05:32

storage: add save/load keyspace ID

1de1307

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

AmoebaProtozoa mentioned this pull request Jul 11, 2022

implement keyspace management #5293

Closed

11 tasks

ti-chi-bot removed the do-not-merge/needs-linked-issue label Jul 11, 2022

sunxiaoguang reviewed Jul 13, 2022

View reviewed changes

server/storage/endpoint/key_path.go Outdated Show resolved Hide resolved

AmoebaProtozoa added 2 commits July 13, 2022 10:41

storage: add comments to keyspaceIDPath

a22119f

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

storage: remove keyspace

923454b

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

ystaticy reviewed Jul 13, 2022

View reviewed changes

server/storage/endpoint/keyspace.go Outdated Show resolved Hide resolved

ystaticy reviewed Jul 13, 2022

View reviewed changes

server/storage/endpoint/keyspace.go Outdated Show resolved Hide resolved

ystaticy reviewed Jul 13, 2022

View reviewed changes

server/storage/keyspace_test.go Outdated Show resolved Hide resolved

JmPotato reviewed Jul 13, 2022

View reviewed changes

server/storage/endpoint/key_path.go Show resolved Hide resolved

server/storage/endpoint/keyspace.go Outdated Show resolved Hide resolved

ystaticy reviewed Jul 13, 2022

View reviewed changes

server/storage/endpoint/keyspace.go Show resolved Hide resolved

ystaticy reviewed Jul 13, 2022

View reviewed changes

server/storage/keyspace_test.go Outdated Show resolved Hide resolved

ystaticy reviewed Jul 13, 2022

View reviewed changes

server/storage/endpoint/keyspace.go Show resolved Hide resolved

AmoebaProtozoa added 5 commits July 18, 2022 14:58

address comments

0c66986

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

address comments

8086113

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

add keyspaceID padding in etcd path

312a6d5

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

typo fix and kvproto update

8e6679f

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

Merge branch 'master' into keyspace_storage

8d43ab6

ystaticy reviewed Jul 19, 2022

View reviewed changes

server/storage/endpoint/keyspace.go Show resolved Hide resolved

AmoebaProtozoa added 3 commits July 21, 2022 18:23

Merge branch 'master' of github.com:tikv/pd into keyspace_storage

3b76bd0

update go mod

c1ab719

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

storage: fix lint

f097c9b

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

AmoebaProtozoa changed the title ~~[WIP] storage: keyspace management~~ storage: keyspace management Jul 27, 2022

AmoebaProtozoa marked this pull request as ready for review July 27, 2022 06:04

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 27, 2022

This was referenced Jul 27, 2022

apiv2: keyspace management #5313

Merged

client: keyspace client #5297

Merged

disksing reviewed Jul 28, 2022

View reviewed changes

server/storage/endpoint/key_path.go Outdated Show resolved Hide resolved

AndreMouche approved these changes Aug 3, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 3, 2022

rleungx reviewed Aug 3, 2022

View reviewed changes

server/storage/endpoint/key_path.go Show resolved Hide resolved

update keyspace

52e3fb5

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

nolouch reviewed Aug 5, 2022

View reviewed changes

go.mod Outdated Show resolved Hide resolved

AmoebaProtozoa added 2 commits August 7, 2022 14:11

update go mod

bca8542

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

Merge branch 'master' of github.com:tikv/pd into keyspace_storage

014f7d4

update kvproto reference

eb4f3c5

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

rleungx reviewed Aug 8, 2022

View reviewed changes

server/storage/endpoint/key_path.go Outdated Show resolved Hide resolved

update comments

ce432f2

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

AmoebaProtozoa added 2 commits August 8, 2022 13:03

make tidy

8612c9c

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

update pd-tso-bench go sum

67dc293

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

nolouch approved these changes Aug 8, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 8, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 8, 2022

ti-chi-bot merged commit e089172 into tikv:master Aug 8, 2022

AmoebaProtozoa deleted the keyspace_storage branch August 8, 2022 07:18

storage: keyspace management #5265

storage: keyspace management #5265

Uh oh!

Conversation

AmoebaProtozoa commented Jul 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

Uh oh!

ti-chi-bot commented Jul 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AndreMouche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AndreMouche Aug 3, 2022

Choose a reason for hiding this comment

Uh oh!

nolouch Aug 8, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Aug 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rleungx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nolouch commented Aug 8, 2022

Uh oh!

nolouch left a comment

Choose a reason for hiding this comment

Uh oh!

nolouch commented Aug 8, 2022

Uh oh!

ti-chi-bot commented Aug 8, 2022

Uh oh!

ti-chi-bot commented Aug 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

AmoebaProtozoa commented Jul 5, 2022 •

edited

Loading

ti-chi-bot commented Jul 5, 2022 •

edited

Loading

codecov bot commented Aug 7, 2022 •

edited

Loading