Skip to content

dont create temporary sparsed array on merging #8174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Aug 24, 2024
Merged

dont create temporary sparsed array on merging #8174

merged 13 commits into from
Aug 24, 2024

Conversation

ivanmorozov333
Copy link
Collaborator

  • Performance improvement

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 16:20:13 UTC Pre-commit check for defb4b2 has started.
⚪ 2024-08-22 16:24:05 UTC Check linux-x86_64-relwithdebinfo is running...
⚫ 2024-08-22 16:38:56 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 16:20:40 UTC Pre-commit check for defb4b2 has started.
⚪ 2024-08-22 16:23:41 UTC Check linux-x86_64-release-clang14 is running...
⚫ 2024-08-22 16:38:55 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 16:21:04 UTC Pre-commit check for defb4b2 has started.
⚪ 2024-08-22 16:23:52 UTC Check linux-x86_64-release-asan is running...
⚫ 2024-08-22 16:39:01 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 16:41:24 UTC Pre-commit check for 60f0aa9 has started.
⚪ 2024-08-22 16:45:06 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-22 16:50:29 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 16:43:03 UTC Pre-commit check for 60f0aa9 has started.
⚪ 2024-08-22 16:45:57 UTC Check linux-x86_64-release-clang14 is running...
🔴 2024-08-22 16:50:17 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 16:43:06 UTC Pre-commit check for 60f0aa9 has started.
⚪ 2024-08-22 16:45:53 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-08-22 16:48:51 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 16:53:50 UTC Pre-commit check for 0bc2e36 has started.
⚪ 2024-08-22 16:56:37 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-22 17:03:01 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 16:54:01 UTC Pre-commit check for 0bc2e36 has started.
⚪ 2024-08-22 16:56:49 UTC Check linux-x86_64-release-asan is running...
⚫ 2024-08-22 18:00:10 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 16:55:55 UTC Pre-commit check for 0bc2e36 has started.
⚪ 2024-08-22 16:58:43 UTC Check linux-x86_64-relwithdebinfo is running...
⚫ 2024-08-22 18:00:10 UTC Check cancelled

@ivanmorozov333 ivanmorozov333 force-pushed the merging_unification_for_sparse branch from 25bf9ed to 358e503 Compare August 22, 2024 17:59
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 18:03:57 UTC Pre-commit check for beb4626 has started.
⚪ 2024-08-22 18:06:54 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-22 19:36:57 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11236 11157 0 33 27 19

🟢 2024-08-22 19:37:58 UTC Build successful.
🟡 2024-08-22 19:38:30 UTC ydbd size 5.5 GiB changed* by +109.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: de42a46 merge: beb4626 diff diff %
ydbd size 5 890 879 200 Bytes 5 890 991 592 Bytes +109.8 KiB +0.002%
ydbd stripped size 1 479 181 744 Bytes 1 479 188 624 Bytes +6.7 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 18:05:19 UTC Pre-commit check for beb4626 has started.
⚪ 2024-08-22 18:08:20 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-08-22 19:34:42 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40324 34978 0 8 5320 18

🟢 2024-08-22 19:38:10 UTC Build successful.
🟢 2024-08-22 19:38:46 UTC ydbd size 8.2 GiB changed* by -99.9 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: de42a46 merge: beb4626 diff diff %
ydbd size 8 753 334 272 Bytes 8 753 231 992 Bytes -99.9 KiB -0.001%
ydbd stripped size 476 022 536 Bytes 475 902 664 Bytes -117.1 KiB -0.025%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 18:10:27 UTC Pre-commit check for beb4626 has started.
⚪ 2024-08-22 18:13:19 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-22 18:18:26 UTC Build successful.

@ivanmorozov333 ivanmorozov333 added the rebase-and-check Rebase PR with the current base branch and check label Aug 23, 2024
@github-actions github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Aug 23, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 05:16:11 UTC Pre-commit check for 75d4fd8 has started.
⚪ 2024-08-23 05:19:02 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-23 06:48:09 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11238 11156 0 30 29 23

🟢 2024-08-23 06:49:09 UTC Build successful.
🟡 2024-08-23 06:49:38 UTC ydbd size 5.5 GiB changed* by +109.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 4a97ccb merge: 75d4fd8 diff diff %
ydbd size 5 890 879 200 Bytes 5 890 991 592 Bytes +109.8 KiB +0.002%
ydbd stripped size 1 479 181 744 Bytes 1 479 188 624 Bytes +6.7 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 05:16:19 UTC Pre-commit check for 75d4fd8 has started.
⚪ 2024-08-23 05:19:07 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-23 05:23:59 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 05:21:41 UTC Pre-commit check for 75d4fd8 has started.
⚪ 2024-08-23 05:24:30 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-23 06:30:51 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40325 34967 0 20 5320 18

⚪ 2024-08-23 06:34:20 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-23 06:41:38 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
37 (only retried tests) 28 0 0 0 9

🟢 2024-08-23 06:41:45 UTC Build successful.
🟢 2024-08-23 06:42:19 UTC ydbd size 8.2 GiB changed* by -99.9 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 4a97ccb merge: 75d4fd8 diff diff %
ydbd size 8 753 334 272 Bytes 8 753 231 992 Bytes -99.9 KiB -0.001%
ydbd stripped size 476 022 536 Bytes 475 902 664 Bytes -117.1 KiB -0.025%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 07:40:33 UTC Pre-commit check for 943d638 has started.
⚪ 2024-08-23 07:43:21 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-23 07:48:50 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 07:40:46 UTC Pre-commit check for 943d638 has started.
⚪ 2024-08-23 07:43:39 UTC Check linux-x86_64-release-asan is running...
⚫ 2024-08-23 08:30:15 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 07:42:52 UTC Pre-commit check for 943d638 has started.
⚪ 2024-08-23 07:45:59 UTC Check linux-x86_64-relwithdebinfo is running...
⚫ 2024-08-23 08:30:18 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 08:33:31 UTC Pre-commit check for 2553dbb has started.
⚪ 2024-08-23 08:37:48 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-23 10:05:28 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11234 11160 0 27 27 20

🟢 2024-08-23 10:06:26 UTC Build successful.
🟡 2024-08-23 10:06:59 UTC ydbd size 5.5 GiB changed* by +192.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 62a41e4 merge: 2553dbb diff diff %
ydbd size 5 890 857 720 Bytes 5 891 055 032 Bytes +192.7 KiB +0.003%
ydbd stripped size 1 479 182 192 Bytes 1 479 207 504 Bytes +24.7 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 08:33:38 UTC Pre-commit check for 2553dbb has started.
⚪ 2024-08-23 08:37:38 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-23 08:43:12 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 08:33:54 UTC Pre-commit check for 2553dbb has started.
⚪ 2024-08-23 08:36:49 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-23 09:43:41 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40319 34977 0 12 5319 11

⚪ 2024-08-23 09:47:13 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-23 09:54:22 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
22 (only retried tests) 14 0 0 0 8

🟢 2024-08-23 09:54:30 UTC Build successful.
🟢 2024-08-23 09:55:05 UTC ydbd size 8.2 GiB changed* by +25.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 62a41e4 merge: 2553dbb diff diff %
ydbd size 8 753 305 120 Bytes 8 753 331 552 Bytes +25.8 KiB +0.000%
ydbd stripped size 476 022 728 Bytes 475 908 488 Bytes -111.6 KiB -0.024%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 15:24:20 UTC Pre-commit check for 60771b3 has started.
⚫ 2024-08-23 15:26:18 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 15:28:35 UTC Pre-commit check for 3b8975c has started.
⚪ 2024-08-23 15:31:26 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-23 15:37:00 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 15:29:47 UTC Pre-commit check for 3b8975c has started.
⚪ 2024-08-23 15:32:40 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-23 17:05:49 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11241 11170 0 20 29 22

🟢 2024-08-23 17:06:51 UTC Build successful.
🟡 2024-08-23 17:07:25 UTC ydbd size 5.5 GiB changed* by +158.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: f2f6db7 merge: 3b8975c diff diff %
ydbd size 5 896 700 600 Bytes 5 896 863 128 Bytes +158.7 KiB +0.003%
ydbd stripped size 1 481 889 616 Bytes 1 481 906 960 Bytes +16.9 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

⚪ 2024-08-23 15:29:57 UTC Pre-commit check for 3b8975c has started.
⚪ 2024-08-23 15:32:46 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-23 16:34:02 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40374 34958 0 42 5358 16

⚪ 2024-08-23 16:37:37 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-23 16:44:33 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
60 (only retried tests) 51 0 0 0 9

🟢 2024-08-23 16:44:40 UTC Build successful.
🟢 2024-08-23 16:45:17 UTC ydbd size 8.2 GiB changed* by -27.1 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: f2f6db7 merge: 3b8975c diff diff %
ydbd size 8 760 955 032 Bytes 8 760 927 304 Bytes -27.1 KiB -0.000%
ydbd stripped size 478 047 176 Bytes 477 932 936 Bytes -111.6 KiB -0.024%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@ivanmorozov333 ivanmorozov333 merged commit b5dba78 into ydb-platform:main Aug 24, 2024
10 of 12 checks passed
@ivanmorozov333 ivanmorozov333 deleted the merging_unification_for_sparse branch August 24, 2024 05:15
This was referenced Aug 27, 2024
Merged
Merged
rekby pushed a commit to rekby-forks/ydb that referenced this pull request Aug 28, 2024
@ivanmorozov333 @rekby
dont create temporary sparsed array on merging (ydb-platform#8174)
5e7a2a8
@niksaveliev niksaveliev mentioned this pull request Aug 29, 2024
Merged
This was referenced Aug 30, 2024
Merged
Merged
stanislav-shchetinin pushed a commit to stanislav-shchetinin/ydb that referenced this pull request Aug 30, 2024
@ivanmorozov333 @stanislav-shchetinin
dont create temporary sparsed array on merging (ydb-platform#8174)
f4e3323
@siarheivesialou siarheivesialou mentioned this pull request Sep 4, 2024
Merged
@niksaveliev niksaveliev mentioned this pull request Sep 9, 2024
Merged
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 9, 2024
@ivanmorozov333 @zverevgeny
dont create temporary sparsed array on merging (ydb-platform#8174)
57833ee
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 10, 2024
@ivanmorozov333 @zverevgeny
dont create temporary sparsed array on merging (ydb-platform#8174)
10c8377
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 11, 2024
@ivanmorozov333 @zverevgeny
dont create temporary sparsed array on merging (ydb-platform#8174)
318e589
@niksaveliev niksaveliev mentioned this pull request Sep 12, 2024
Merged
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 13, 2024
@ivanmorozov333 @zverevgeny
dont create temporary sparsed array on merging (ydb-platform#8174)
bddf866 
Conflicts:
	ydb/core/tx/columnshard/engines/changes/compaction/abstract/merger.cpp
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 14, 2024
@ivanmorozov333 @zverevgeny
dont create temporary sparsed array on merging (ydb-platform#8174)
46fef84 
Conflicts:
	ydb/core/tx/columnshard/engines/changes/compaction/abstract/merger.cpp
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 15, 2024
@ivanmorozov333 @zverevgeny
dont create temporary sparsed array on merging (ydb-platform#8174)
b22741b
@zverevgeny zverevgeny mentioned this pull request Sep 16, 2024
Closed
This was referenced Sep 17, 2024
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@swalrus1 swalrus1 swalrus1 approved these changes

@aavdonkin aavdonkin aavdonkin approved these changes

Assignees
No one assigned
Labels
performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ivanmorozov333 @swalrus1 @aavdonkin