log-backup: added manual flush RPCs #18027

YuJuncen · 2024-12-18T09:23:25Z

What is changed and how it works?

Issue Number: Close #18026

What's Changed:

Added a new RPC endpoint `flush_now` for the service `LogBackup`.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Release note

None

YuJuncen · 2024-12-20T06:35:36Z

/test pull-unit-test

Signed-off-by: hillium <yujuncen@pingcap.com> Signed-off-by: hillium <yu745514916@live.com>

Signed-off-by: hillium <yu745514916@live.com>

BornChanger · 2024-12-20T06:50:34Z

components/backup-stream/src/endpoint.rs

@@ -857,20 +865,36 @@ where
        }
    }

-    pub fn on_force_flush(&self, task: String) {
+    pub fn on_force_flush(&self, task: TaskSelectorRef<'_>, sender: Sender<FlushResult>) {


do we need to prevent too much force flush here if the callers abuse it?

For now we don't need to care this. Because only 1 task is allowed.

BornChanger · 2024-12-20T06:56:56Z

components/backup-stream/src/endpoint.rs

                }),
                min_ts: mts,
            })
            .await
        })
    }

-    fn on_exec_flush(&mut self, task: String, resolved: ResolvedRegions) {
+    fn on_exec_flush(&mut self, task: String, resolved: ResolvedRegions, cb: Sender<FlushResult>) {


where can we know there is force flush occurs? from tikv log or metrics?

For now we can get the number of ForceFlush tasks by the metric tikv_log_backup_interal_actor_acting_duration_sec_count.

Leavrth · 2024-12-27T04:00:52Z

components/backup-stream/src/endpoint.rs

-                }
-            })));
+        let fut = self.do_flush(task.clone(), resolved);
+        let syncer = self.checkpoint_mgr.make_syncer();


maybe we can make syncer in the on_force_flush to avoid the normal flush execution path.

ti-chi-bot · 2024-12-30T06:11:18Z

@BornChanger: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: hillium <yu745514916@live.com>

components/backup-stream/src/endpoint.rs

Signed-off-by: hillium <yu745514916@live.com>

Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>

YuJuncen · 2025-01-07T09:55:19Z

/test pull-unit-test

ti-chi-bot · 2025-01-07T10:19:05Z

[LGTM Timeline notifier]

Timeline:

2024-12-27 04:01:03.309131674 +0000 UTC m=+63798.665136195: ☑️ agreed by Leavrth.
2025-01-07 10:19:03.8691087 +0000 UTC m=+262487.157940406: ☑️ agreed by 3pointer.

ti-chi-bot · 2025-01-07T23:53:49Z

@YuJuncen: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: hillium <yu745514916@live.com>

ti-chi-bot · 2025-01-08T10:04:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3pointer, BornChanger, iosmanthus, Leavrth

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [iosmanthus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: yes Indicates the PR's author has signed the dco. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 18, 2024

YuJuncen force-pushed the add-manual-flush branch from a4a5526 to 296b139 Compare December 19, 2024 09:55

YuJuncen and others added 7 commits December 20, 2024 14:39

partial commit

2ec793b

Signed-off-by: hillium <yujuncen@pingcap.com> Signed-off-by: hillium <yu745514916@live.com>

implement FlushNow

9543025

Signed-off-by: hillium <yu745514916@live.com>

added manual flush

b5eb58c

Signed-off-by: hillium <yu745514916@live.com>

wait flushing to client then exit

3d399ad

Signed-off-by: hillium <yu745514916@live.com>

added test case, restored kvproto

91542fd

Signed-off-by: hillium <yu745514916@live.com>

rm tmp files

ec6d54c

Signed-off-by: hillium <yu745514916@live.com>

added more assertions

60c5410

Signed-off-by: hillium <yu745514916@live.com>

YuJuncen force-pushed the add-manual-flush branch from 296b139 to 60c5410 Compare December 20, 2024 06:39

BornChanger reviewed Dec 20, 2024

View reviewed changes

Leavrth approved these changes Dec 27, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Dec 27, 2024

BornChanger approved these changes Dec 30, 2024

View reviewed changes

don't sync with manager when not required

8219ff5

Signed-off-by: hillium <yu745514916@live.com>

3pointer reviewed Jan 6, 2025

View reviewed changes

components/backup-stream/src/endpoint.rs Outdated Show resolved Hide resolved

address comments

6bb531c

Signed-off-by: hillium <yu745514916@live.com>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 6, 2025

make waiter simpler

ec8cbf1

Signed-off-by: hillium <yu745514916@live.com>

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 6, 2025

make clippy happy

a682c21

Signed-off-by: hillium <yu745514916@live.com>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 6, 2025

Merge branch 'master' into add-manual-flush

7f45006

Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>

3pointer approved these changes Jan 7, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 7, 2025

ti-chi-bot bot added 2 commits January 7, 2025 10:33

Merge branch 'master' into add-manual-flush

ee2308a

Merge branch 'master' into add-manual-flush

27e75d5

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 8, 2025

remove syncing

469ab2d

Signed-off-by: hillium <yu745514916@live.com>

YuJuncen force-pushed the add-manual-flush branch from 145a002 to 469ab2d Compare January 8, 2025 08:48

YuJuncen mentioned this pull request Jan 8, 2025

br/operator: add force flush for log backup pingcap/tidb#58799

Open

iosmanthus approved these changes Jan 8, 2025

View reviewed changes

ti-chi-bot bot added the approved label Jan 8, 2025

ti-chi-bot bot merged commit 222019f into tikv:master Jan 8, 2025
8 checks passed

ti-chi-bot bot added this to the Pool milestone Jan 8, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

log-backup: added manual flush RPCs #18027

log-backup: added manual flush RPCs #18027

YuJuncen commented Dec 18, 2024

YuJuncen commented Dec 20, 2024

BornChanger Dec 20, 2024

YuJuncen Dec 20, 2024

BornChanger Dec 20, 2024

YuJuncen Dec 25, 2024 •

edited

Loading

Leavrth Dec 27, 2024

ti-chi-bot bot commented Dec 30, 2024

YuJuncen commented Jan 7, 2025

ti-chi-bot bot commented Jan 7, 2025

ti-chi-bot bot commented Jan 7, 2025

ti-chi-bot bot commented Jan 8, 2025

log-backup: added manual flush RPCs #18027

log-backup: added manual flush RPCs #18027

Conversation

YuJuncen commented Dec 18, 2024

What is changed and how it works?

Related changes

Check List

Release note

YuJuncen commented Dec 20, 2024

BornChanger Dec 20, 2024

Choose a reason for hiding this comment

YuJuncen Dec 20, 2024

Choose a reason for hiding this comment

BornChanger Dec 20, 2024

Choose a reason for hiding this comment

YuJuncen Dec 25, 2024 • edited Loading

Choose a reason for hiding this comment

Leavrth Dec 27, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented Dec 30, 2024

YuJuncen commented Jan 7, 2025

ti-chi-bot bot commented Jan 7, 2025

[LGTM Timeline notifier]

ti-chi-bot bot commented Jan 7, 2025

ti-chi-bot bot commented Jan 8, 2025

YuJuncen Dec 25, 2024 •

edited

Loading