Skip to content

Don't allow replication to overconsume space on PDisk #17418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Apr 22, 2025
Merged

Don't allow replication to overconsume space on PDisk #17418

merged 4 commits into from
Apr 22, 2025

Conversation

serbel324
Copy link
Collaborator

@serbel324 serbel324 commented Apr 18, 2025 •
edited
Loading

Changelog entry

Previously replication didn't handle StatusFlags from PDisk properly and didn't stop when available space was running low. Because of this, space was consumed until VDisk and corresponding storage group became read-only. With this change, replication will postpone on YELLOW_STOP color flag and resume after a certain delay if enough space is available.

#13608

Changelog category

  • Bugfix

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 18, 2025 •
edited
Loading

🟢 2025-04-21 11:58:21 UTC The validation of the Pull Request description is successful.

@serbel324 serbel324 force-pushed the bs/vdisk/replication-color-flag branch 2 times, most recently from a70152e to c35266b Compare April 18, 2025 10:32
@serbel324 serbel324 force-pushed the bs/vdisk/replication-color-flag branch from c35266b to e897b70 Compare April 18, 2025 10:56
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 18, 2025 •
edited
Loading

⚪ 2025-04-18 10:58:23 UTC Pre-commit check linux-x86_64-release-asan for ca9dc23 has started.
⚪ 2025-04-18 10:58:51 UTC Artifacts will be uploaded here
⚪ 2025-04-18 11:02:17 UTC ya make is running...
🟡 2025-04-18 12:18:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12896 12693 0 143 34 26

⚪ 2025-04-18 12:19:28 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-18 12:33:52 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
252 (only retried tests) 183 0 37 7 25

⚪ 2025-04-18 12:34:01 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-04-18 12:46:25 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
94 (only retried tests) 34 0 34 2 24

🟢 2025-04-18 12:46:32 UTC Build successful.
🟡 2025-04-18 12:47:06 UTC ydbd size 3.9 GiB changed* by +185.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 3892bcc merge: ca9dc23 diff diff %
ydbd size 4 147 912 224 Bytes 4 148 101 752 Bytes +185.1 KiB +0.005%
ydbd stripped size 1 432 002 008 Bytes 1 432 061 592 Bytes +58.2 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 18, 2025 •
edited
Loading

⚪ 2025-04-18 10:59:50 UTC Pre-commit check linux-x86_64-relwithdebinfo for ca9dc23 has started.
⚪ 2025-04-18 11:00:01 UTC Artifacts will be uploaded here
⚪ 2025-04-18 11:02:47 UTC ya make is running...
🟡 2025-04-18 12:06:37 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20548 19171 0 3 1264 110

⚪ 2025-04-18 12:08:25 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-04-18 12:20:31 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
172 (only retried tests) 77 0 0 0 95

🟢 2025-04-18 12:20:38 UTC Build successful.
🟢 2025-04-18 12:20:57 UTC ydbd size 2.2 GiB changed* by +96.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 3892bcc merge: ca9dc23 diff diff %
ydbd size 2 363 755 904 Bytes 2 363 854 656 Bytes +96.4 KiB +0.004%
ydbd stripped size 493 912 800 Bytes 493 931 104 Bytes +17.9 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 18, 2025 •
edited
Loading

⚪ 2025-04-18 13:10:11 UTC Pre-commit check linux-x86_64-release-asan for 1379958 has started.
⚪ 2025-04-18 13:10:27 UTC Artifacts will be uploaded here
⚪ 2025-04-18 13:13:24 UTC ya make is running...
🟡 2025-04-18 14:35:50 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12930 12744 0 130 31 25

⚪ 2025-04-18 14:36:59 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-18 14:51:01 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
246 (only retried tests) 178 0 40 4 24

⚪ 2025-04-18 14:51:10 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-04-18 15:03:28 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
95 (only retried tests) 33 0 36 2 24

🟢 2025-04-18 15:03:36 UTC Build successful.
🟡 2025-04-18 15:04:05 UTC ydbd size 3.9 GiB changed* by +185.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a0b1351 merge: 1379958 diff diff %
ydbd size 4 147 916 432 Bytes 4 148 106 064 Bytes +185.2 KiB +0.005%
ydbd stripped size 1 432 002 008 Bytes 1 432 061 656 Bytes +58.2 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 18, 2025 •
edited
Loading

⚪ 2025-04-18 13:10:31 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1379958 has started.
⚪ 2025-04-18 13:10:47 UTC Artifacts will be uploaded here
⚪ 2025-04-18 13:13:42 UTC ya make is running...
🟡 2025-04-18 14:16:59 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20580 19181 0 6 1279 114

⚪ 2025-04-18 14:18:37 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-04-18 14:32:02 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
193 (only retried tests) 97 0 0 0 96

🟢 2025-04-18 14:32:09 UTC Build successful.
🟢 2025-04-18 14:32:32 UTC ydbd size 2.2 GiB changed* by +96.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: a0b1351 merge: 1379958 diff diff %
ydbd size 2 363 760 352 Bytes 2 363 859 184 Bytes +96.5 KiB +0.004%
ydbd stripped size 493 913 568 Bytes 493 931 872 Bytes +17.9 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions github-actions bot added bugfix and removed bugfix labels Apr 21, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 22, 2025 •
edited
Loading

⚪ 2025-04-22 06:54:16 UTC Pre-commit check linux-x86_64-release-asan for 80848d2 has started.
⚪ 2025-04-22 06:54:43 UTC Artifacts will be uploaded here
⚪ 2025-04-22 06:58:16 UTC ya make is running...
🟡 2025-04-22 08:18:56 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13067 12918 0 93 34 22

⚪ 2025-04-22 08:20:06 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-22 08:33:28 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
196 (only retried tests) 120 0 38 16 22

⚪ 2025-04-22 08:33:38 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-04-22 08:47:21 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
98 (only retried tests) 38 0 36 3 21

🟢 2025-04-22 08:47:28 UTC Build successful.
🟡 2025-04-22 08:47:57 UTC ydbd size 3.8 GiB changed* by +173.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 52cd0e1 merge: 80848d2 diff diff %
ydbd size 4 130 489 976 Bytes 4 130 667 480 Bytes +173.3 KiB +0.004%
ydbd stripped size 1 430 160 088 Bytes 1 430 210 328 Bytes +49.1 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 22, 2025 •
edited
Loading

⚪ 2025-04-22 06:56:55 UTC Pre-commit check linux-x86_64-relwithdebinfo for 80848d2 has started.
⚪ 2025-04-22 06:57:45 UTC Artifacts will be uploaded here
⚪ 2025-04-22 07:01:16 UTC ya make is running...
🟡 2025-04-22 08:16:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20716 19338 0 8 1266 104

⚪ 2025-04-22 08:18:40 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-04-22 08:42:43 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
167 (only retried tests) 71 0 0 0 96

🟢 2025-04-22 08:42:50 UTC Build successful.
🟢 2025-04-22 08:43:14 UTC ydbd size 2.2 GiB changed* by +97.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 52cd0e1 merge: 80848d2 diff diff %
ydbd size 2 351 523 096 Bytes 2 351 623 384 Bytes +97.9 KiB +0.004%
ydbd stripped size 493 707 360 Bytes 493 729 312 Bytes +21.4 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@serbel324 serbel324 linked an issue Apr 22, 2025 that may be closed by this pull request
Handle color status flags during VDisk replication and stop replication when close to OOS #13608
Closed
@serbel324 serbel324 merged commit 0af4225 into ydb-platform:main Apr 22, 2025
14 checks passed
serbel324 added a commit to serbel324/ydb that referenced this pull request May 14, 2025
@serbel324
Don't allow replication to overconsume space on PDisk (ydb-platform#1…
79a3c25 
…7418)
@serbel324 serbel324 mentioned this pull request May 15, 2025
Merged
serbel324 added a commit that referenced this pull request May 15, 2025
@serbel324
Don't allow replication to overconsume space on PDisk (#17418) (#18296)
a3e836e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexvru alexvru alexvru approved these changes

Assignees
No one assigned
Labels
bugfix
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Handle color status flags during VDisk replication and stop replication when close to OOS
2 participants
@serbel324 @alexvru