Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BR download SST consider disk usage and report error if almost full #17224

Closed
nolouch opened this issue Jul 1, 2024 · 7 comments · Fixed by #17238
Closed

BR download SST consider disk usage and report error if almost full #17224

nolouch opened this issue Jul 1, 2024 · 7 comments · Fixed by #17238
Labels
affects-7.1 affects-7.5 affects-8.1 component/backup-restore Component: backup, import, external_storage type/enhancement The issue or PR belongs to an enhancement.

Comments

@nolouch
Copy link
Contributor

nolouch commented Jul 1, 2024 •
edited
Loading

Problem

In #10537, tikv introduces the disk usage detection mechanism. the overview is :

During the raft proposal command, we will detect the disk usage, If the disk is full, some write commands cannot be processed and report the DiskFull error to the upper level. It avoids the disk full situation occurring in some scenarios.

But in BR restore it may not work well. BR restore includes two steps, DownloadSST then IngestSST, if the disk is full, the IngestSST may fail with the upper check. but DownloadSST still can work because it does not need to do a raft proposal. like here is tiflash's disk is full during BR restore

Proposal

In order to prevent the disk full during the BR restore, we may let DownloadSST also consider the disk usage like #10537.
@nolouch nolouch added the type/enhancement The issue or PR belongs to an enhancement. label Jul 1, 2024
@Connor1996
Copy link
Member

/cc @LykxSassinator

This was referenced Jul 3, 2024
Merged
Merged
@ti-chi-bot ti-chi-bot bot closed this as completed in 485c434 Jul 8, 2024
@BornChanger
Copy link
Contributor

/label severity:major

@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jul 25, 2024

@BornChanger: The label(s) severity:major cannot be applied. These labels are supported: challenge-program, compatibility-breaker, high-performance, hptc, wontfix, do-not-merge/cherry-pick-not-approved, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.5, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.5, needs-cherry-pick-release-8.1, needs-cherry-pick-release-8.2, affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.2, may-affects-5.4, may-affects-6.1, may-affects-6.5, may-affects-7.1, may-affects-7.5, may-affects-8.1.

In response to this:

/label severity:major

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@BornChanger
Copy link
Contributor

/label affects-7.1

@BornChanger
Copy link
Contributor

BornChanger commented Jul 25, 2024 •
edited
Loading

/label affects-7.5

@BornChanger
Copy link
Contributor

/label affects-8.1

@ti-chi-bot ti-chi-bot mentioned this issue Jul 31, 2024
Merged
9 tasks
@BornChanger
Copy link
Contributor

/component backup-restore

@ti-chi-bot ti-chi-bot bot added the component/backup-restore Component: backup, import, external_storage label Aug 5, 2024
ti-chi-bot bot pushed a commit that referenced this issue Aug 5, 2024
@ti-chi-bot @RidRisR
br: pre-check TiKV disk space before download (#17238) (#17341)
002ebde 
close #17224

Add a disk usage check when execute `download` and `apply` RPC from br.
When the disk is not `Normal`, the request would be rejected.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
@JaySon-Huang JaySon-Huang mentioned this issue Aug 5, 2024
Closed
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Sep 23, 2024
@RidRisR @ti-chi-bot
This is an automated cherry-pick of tikv#17238
23d48a4 
close tikv#17224

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Sep 23, 2024
Merged
9 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 27, 2024
@ti-chi-bot @RidRisR @YuJuncen
br: pre-check TiKV disk space before download (#17238) (#17569)
816595f 
close #17224

Add a disk usage check when execute `download` and `apply` RPC from br.
When the disk is not `Normal`, the request would be rejected.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com>
Co-authored-by: hillium <yujuncen@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-7.1 affects-7.5 affects-8.1 component/backup-restore Component: backup, import, external_storage type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

br: pre-check TiKV disk space before download RidRisR/tikv
3 participants
@nolouch @Connor1996 @BornChanger