speed up merging for sparsed columns #8365

ivanmorozov333 · 2024-08-28T06:11:33Z

Performance improvement

github-actions · 2024-08-28T06:12:45Z

⚪ 2024-08-28 06:12:44 UTC Pre-commit check for c6b170b has started.
⚪ 2024-08-28 06:15:18 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-28 06:20:25 UTC Build successful.

github-actions · 2024-08-28T06:12:53Z

⚪ 2024-08-28 06:12:52 UTC Pre-commit check for c6b170b has started.
⚪ 2024-08-28 06:15:26 UTC Check linux-x86_64-relwithdebinfo is running...
⚫ 2024-08-28 06:26:26 UTC Check cancelled

github-actions · 2024-08-28T06:13:12Z

⚪ 2024-08-28 06:13:12 UTC Pre-commit check for c6b170b has started.
⚪ 2024-08-28 06:15:48 UTC Check linux-x86_64-release-asan is running...
⚫ 2024-08-28 06:26:02 UTC Check cancelled

github-actions · 2024-08-28T06:29:38Z

⚪ 2024-08-28 06:29:38 UTC Pre-commit check for c943068 has started.
⚪ 2024-08-28 06:32:27 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-28 06:37:28 UTC Build successful.

github-actions · 2024-08-28T06:29:45Z

⚪ 2024-08-28 06:29:45 UTC Pre-commit check for c943068 has started.
⚪ 2024-08-28 06:32:26 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-28 07:24:09 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14323	12950	0	5	1350	18

⚪ 2024-08-28 07:25:24 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-08-28 08:01:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
26 (only retried tests)	13	0	1	1	11

⚪ 2024-08-28 08:02:04 UTC Failed tests rerun (try 3) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-28 08:10:49 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
17 (only retried tests)	6	0	0	0	11

🟢 2024-08-28 08:10:56 UTC Build successful.
🟡 2024-08-28 08:11:35 UTC ydbd size 8.2 GiB changed* by +626.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `10a3f7c`	merge: `c943068`	diff	diff %
ydbd size	8 800 099 440 Bytes	8 800 741 112 Bytes	+626.6 KiB	+0.007%
ydbd stripped size	479 862 376 Bytes	479 885 480 Bytes	+22.6 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-08-28T06:30:05Z

⚪ 2024-08-28 06:30:05 UTC Pre-commit check for c943068 has started.
⚪ 2024-08-28 06:32:46 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-28 07:55:37 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
9893	9838	0	13	25	17

🟢 2024-08-28 07:56:29 UTC Build successful.
🟡 2024-08-28 07:56:57 UTC ydbd size 5.5 GiB changed* by +364.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `10a3f7c`	merge: `c943068`	diff	diff %
ydbd size	5 928 922 528 Bytes	5 929 295 808 Bytes	+364.5 KiB	+0.006%
ydbd stripped size	1 489 124 400 Bytes	1 489 204 464 Bytes	+78.2 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-08-28T10:39:31Z

⚪ 2024-08-28 10:39:30 UTC Pre-commit check for 78ed8f8 has started.
⚪ 2024-08-28 10:42:04 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-28 12:09:19 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
9897	9832	0	21	26	18

🟢 2024-08-28 12:10:12 UTC Build successful.
🟡 2024-08-28 12:10:40 UTC ydbd size 5.5 GiB changed* by +316.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `22f58dc`	merge: `78ed8f8`	diff	diff %
ydbd size	5 929 468 480 Bytes	5 929 792 840 Bytes	+316.8 KiB	+0.005%
ydbd stripped size	1 489 195 408 Bytes	1 489 267 152 Bytes	+70.1 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-08-28T10:40:40Z

⚪ 2024-08-28 10:40:39 UTC Pre-commit check for 78ed8f8 has started.
⚪ 2024-08-28 10:44:03 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-28 10:49:15 UTC Build successful.

github-actions · 2024-08-28T10:42:17Z

⚪ 2024-08-28 10:42:17 UTC Pre-commit check for 78ed8f8 has started.
⚪ 2024-08-28 10:44:59 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-28 11:44:30 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14328	12956	0	2	1349	21

⚪ 2024-08-28 11:45:56 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-28 11:53:39 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
26 (only retried tests)	13	0	0	0	13

🟢 2024-08-28 11:53:47 UTC Build successful.
🟡 2024-08-28 11:54:26 UTC ydbd size 8.2 GiB changed* by +597.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `419b953`	merge: `78ed8f8`	diff	diff %
ydbd size	8 800 969 632 Bytes	8 801 581 160 Bytes	+597.2 KiB	+0.007%
ydbd stripped size	479 883 272 Bytes	479 906 632 Bytes	+22.8 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

aavdonkin · 2024-08-28T13:49:05Z

ydb/core/tx/columnshard/engines/changes/compaction/abstract/merger.h

+                }
+                ++chunkIdx;
+            }
+            RemapPortionIndexToResultIndex = std::move(result);


Почему сразу не строить RemapPortionIndexToResultIndex, а через result?

если кто-то захочет посередине выйти - мы мусор не сохраним, даже если exception вылетит... но это, скорее, психологический момент. предпочитаю "атомарно" менять сложные конструкции

aavdonkin · 2024-08-28T14:03:06Z

ydb/core/tx/columnshard/engines/changes/compaction/sparsed/logic.cpp

+            Cursors.emplace_back(p, Context);
+            if (mergingContext.HasRemapInfo(idx)) {
+                CursorPositions.emplace_back(TCursorPosition(&Cursors.back(), mergingContext.GetRemapPortionIndexToResultIndex(idx)));
+                if (CursorPositions.back().IsFinished()) {


А нельзя до добавления проверять на IsFinished?

да можно - лень было переменную объявлять. это условие редкое

Conflicts: ydb/core/tx/columnshard/engines/changes/compaction/merger.cpp

github-actions bot added the performance label Aug 28, 2024

speed up merging for sparsed columns

9cb8435

ivanmorozov333 force-pushed the speed_up_sparsed_merge branch from ac88928 to 9cb8435 Compare August 28, 2024 06:25

correction

cbcafee

aavdonkin reviewed Aug 28, 2024

View reviewed changes

aavdonkin marked this pull request as ready for review August 28, 2024 16:00

aavdonkin approved these changes Aug 28, 2024

View reviewed changes

ivanmorozov333 merged commit f5de1a7 into ydb-platform:main Aug 28, 2024
10 of 12 checks passed

ivanmorozov333 deleted the speed_up_sparsed_merge branch August 28, 2024 16:03

rekby pushed a commit to rekby-forks/ydb that referenced this pull request Aug 28, 2024

speed up merging for sparsed columns (ydb-platform#8365)

0dc0c03

niksaveliev mentioned this pull request Aug 29, 2024

Fix pqv1 max partitions #8485

Merged

GrigoriyPA mentioned this pull request Aug 30, 2024

Fixed compute actor finalizing in case of node disconnect #8511

Closed

siarheivesialou mentioned this pull request Aug 30, 2024

LOGBROKER 8891 fix not using optional fields in sqs json api properly #8523

Merged

stanislav-shchetinin pushed a commit to stanislav-shchetinin/ydb that referenced this pull request Aug 30, 2024

speed up merging for sparsed columns (ydb-platform#8365)

90de929

siarheivesialou mentioned this pull request Sep 4, 2024

fix memory leak in ymq proxy #8706

Merged

This was referenced Sep 4, 2024

YQ-3151 added issues for timeout and cancelled #8718

Merged

YQ kqprun added cancel after for script requests #8768

Merged

Fixed unit test ManyColumnShards #8777

Merged

niksaveliev mentioned this pull request Sep 9, 2024

Fix CLI max partitions default #8944

Merged

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 9, 2024

speed up merging for sparsed columns (ydb-platform#8365)

3ed7f54

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 10, 2024

speed up merging for sparsed columns (ydb-platform#8365)

c019b08

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 11, 2024

speed up merging for sparsed columns (ydb-platform#8365)

9015b58

niksaveliev mentioned this pull request Sep 12, 2024

Fix reading without consumer from federation #9136

Merged

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 13, 2024

speed up merging for sparsed columns (ydb-platform#8365)

7235068

Conflicts: ydb/core/tx/columnshard/engines/changes/compaction/merger.cpp

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 14, 2024

speed up merging for sparsed columns (ydb-platform#8365)

497e981

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 15, 2024

speed up merging for sparsed columns (ydb-platform#8365)

6254bb4

zverevgeny mentioned this pull request Sep 16, 2024

Integrate latest changes in CS into stable branch #9271

Closed

StekPerepolnen mentioned this pull request Sep 17, 2024

doc hc api images source #9376

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

speed up merging for sparsed columns #8365

speed up merging for sparsed columns #8365

ivanmorozov333 commented Aug 28, 2024

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

aavdonkin Aug 28, 2024

ivanmorozov333 Aug 28, 2024

aavdonkin Aug 28, 2024

ivanmorozov333 Aug 28, 2024

speed up merging for sparsed columns #8365

speed up merging for sparsed columns #8365

Conversation

ivanmorozov333 commented Aug 28, 2024

github-actions bot commented Aug 28, 2024 • edited Loading

github-actions bot commented Aug 28, 2024 • edited Loading

github-actions bot commented Aug 28, 2024 • edited Loading

github-actions bot commented Aug 28, 2024 • edited Loading

github-actions bot commented Aug 28, 2024 • edited Loading

github-actions bot commented Aug 28, 2024 • edited Loading

github-actions bot commented Aug 28, 2024 • edited Loading

github-actions bot commented Aug 28, 2024 • edited Loading

github-actions bot commented Aug 28, 2024 • edited Loading

aavdonkin Aug 28, 2024

Choose a reason for hiding this comment

ivanmorozov333 Aug 28, 2024

Choose a reason for hiding this comment

aavdonkin Aug 28, 2024

Choose a reason for hiding this comment

ivanmorozov333 Aug 28, 2024

Choose a reason for hiding this comment

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024 •

edited

Loading