compacted restore: ignore some keys out of specific range #59112
Conversation
|
Skipping CI for Draft Pull Request. |
ti-chi-bot
bot
added
do-not-merge/work-in-progress
|
Hi @3pointer. Thanks for your PR. PRs from untrusted users cannot be marked as trusted with I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
ti-chi-bot
bot
added
size/L
ti-chi-bot
bot
removed
the
do-not-merge/work-in-progress
br/pkg/restore/utils/rewrite_rule.go
Outdated
| }) | ||
| } | ||
|
|
||
| return &RewriteRules{Data: dataRules, NewTableID: newTableID, TableIDRemapHint: remaps} | ||
| return &RewriteRules{Data: dataRules, NewTableID: newTableID, ShiftStartTs: shiftStartTs, StartTs: startTs, RestoredTs: restoredTs, TableIDRemapHint: remaps} |
There was a problem hiding this comment.
It seems the caller may manually fill the ShiftStartTs stuffs after getting the returned RewriteRules. So I think perhaps we don't need to add two new arguments to the function (anyway this requires us to modify all call sites...).
| @@ -245,11 +245,11 @@ func (wm *WithMigrations) Metas(metaNameIter MetaNameIter) MetaMigrationsIter { | |||
| }) | |||
| } | |||
|
|
|||
| func (wm *WithMigrations) Compactions(ctx context.Context, s storage.ExternalStorage) iter.TryNextor[*backuppb.LogFileSubcompaction] { | |||
| func (wm *WithMigrations) Compactions(ctx context.Context, s storage.ExternalStorage, shiftStartTS, restoreTS uint64) iter.TryNextor[*backuppb.LogFileSubcompaction] { | |||
There was a problem hiding this comment.
Is it possible to add shiftStartTS to the WithMigrations? I think in this context we may usually require this TS. Also it seems WithMigrations has a field of restored ts.
There was a problem hiding this comment.
already have, fixed.
br/pkg/task/stream.go
Outdated
| @@ -1914,7 +1914,7 @@ func parseFullBackupTablesStorage( | |||
| }, nil | |||
| } | |||
|
|
|||
| func initRewriteRules(schemasReplace *stream.SchemasReplace) map[int64]*restoreutils.RewriteRules { | |||
There was a problem hiding this comment.
Say... We may register the shiftedStartTs, startTs and restoredTs to each rewrite rules in the returned map.
There was a problem hiding this comment.
yes, we don't have such common rule for rewrite, it's bind with table.
ti-chi-bot
bot
added
approved
needs-1-more-lgtm
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #59112 +/- ##
================================================
+ Coverage 73.0289% 75.4203% +2.3914%
================================================
Files 1683 1705 +22
Lines 466114 480463 +14349
================================================
+ Hits 340398 362367 +21969
+ Misses 104792 95870 -8922
- Partials 20924 22226 +1302
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Leavrth, YuJuncen The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
ti-chi-bot
bot
added
lgtm
and removed
needs-1-more-lgtm
|
/retest |
|
@BornChanger: Cannot trigger testing until a trusted user reviews the PR and leaves an In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
What problem does this PR solve?
Issue Number: close #58238
Problem Summary:
Add time range filter for compacted sst files to reduce the download traffic during restore.
What changed and how does it work?
use shiftStartTs for default cf to filter sst data.
use startTs for write cf to filter sst data.
use restoredTs for both default and write cf.
Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.