Skip to content

Conversation

@abyss7
Copy link
Collaborator

@abyss7 abyss7 commented Dec 6, 2024

Changelog entry

Use new YQL mechanic for the Arrow buffer resize - to shrink in 100% of times when consuming blocks for DqOutput. Otherwise we get over-allocated buffers with very small accounted logical size - it leads to OOMs

Changelog category

  • Performance improvement

@abyss7 abyss7 requested review from a team as code owners December 6, 2024 14:56
@abyss7 abyss7 requested a review from gridnevvvit December 6, 2024 14:59
@github-actions
Copy link

github-actions bot commented Dec 6, 2024

2024-12-06 15:00:16 UTC Pre-commit check linux-x86_64-relwithdebinfo for 968f956 has started.
2024-12-06 15:00:51 UTC Artifacts will be uploaded here
2024-12-06 15:04:22 UTC ya make is running...
🟢 2024-12-06 16:13:51 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
32567 29735 0 0 2720 112

🟢 2024-12-06 16:16:37 UTC Build successful.
🟢 2024-12-06 16:16:53 UTC ydbd size 2.5 GiB changed* by +424 Bytes, which is < 100.0 KiB vs main: OK

ydbd size dash main: 17066f1 merge: 968f956 diff diff %
ydbd size 2 699 201 576 Bytes 2 699 202 000 Bytes +424 Bytes +0.000%
ydbd stripped size 483 500 720 Bytes 483 500 784 Bytes +64 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Dec 6, 2024

2024-12-06 15:00:25 UTC Pre-commit check linux-x86_64-release-asan for 968f956 has started.
2024-12-06 15:00:37 UTC Artifacts will be uploaded here
2024-12-06 15:03:46 UTC ya make is running...
🟡 2024-12-06 16:22:03 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15186 15099 0 33 9 45

🟢 2024-12-06 16:23:20 UTC Build successful.
🟢 2024-12-06 16:23:49 UTC ydbd size 4.9 GiB changed* by +856 Bytes, which is < 100.0 KiB vs main: OK

ydbd size dash main: 17066f1 merge: 968f956 diff diff %
ydbd size 5 305 570 352 Bytes 5 305 571 208 Bytes +856 Bytes +0.000%
ydbd stripped size 1 365 574 736 Bytes 1 365 575 056 Bytes +320 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@abyss7 abyss7 merged commit b3ab002 into ydb-platform:main Dec 9, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants