Skip to content

implement sparse data accessor #7055

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Jul 28, 2024
Merged

implement sparse data accessor #7055

merged 8 commits into from
Jul 28, 2024

Conversation

ivanmorozov333
Copy link
Collaborator

  • New feature
  • Improvement
  • Performance improvement

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 24, 2024 •
edited
Loading

⚪ 2024-07-24 16:53:50 UTC Pre-commit check for 39797a1 has started.
⚪ 2024-07-24 16:56:49 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-24 16:57:01 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 24, 2024 •
edited
Loading

⚪ 2024-07-24 16:54:00 UTC Pre-commit check for 39797a1 has started.
⚪ 2024-07-24 16:56:50 UTC Check linux-x86_64-release-clang14 is running...
🔴 2024-07-24 16:57:03 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 24, 2024 •
edited
Loading

⚪ 2024-07-24 16:56:49 UTC Pre-commit check for 39797a1 has started.
⚪ 2024-07-24 16:59:45 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-07-24 16:59:57 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

⚪ 2024-07-25 06:08:53 UTC Pre-commit check for bf1621b has started.

@github-actions GitHub Actions
Copy link

⚪ 2024-07-25 06:09:25 UTC Pre-commit check for bf1621b has started.

@github-actions GitHub Actions
Copy link

⚪ 2024-07-25 06:09:25 UTC Pre-commit check for bf1621b has started.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 06:30:35 UTC Pre-commit check for c46b3db has started.
⚪ 2024-07-25 06:33:39 UTC Check linux-x86_64-release-clang14 is running...
🔴 2024-07-25 06:45:17 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 06:30:44 UTC Pre-commit check for c46b3db has started.
⚪ 2024-07-25 06:33:40 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-25 06:43:34 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

⚪ 2024-07-25 06:33:10 UTC Pre-commit check for c46b3db has started.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 08:09:47 UTC Pre-commit check for 99f98e2 has started.
⚪ 2024-07-25 08:12:43 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-25 08:16:21 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 08:10:52 UTC Pre-commit check for 99f98e2 has started.
⚪ 2024-07-25 08:13:43 UTC Check linux-x86_64-release-clang14 is running...
⚫ 2024-07-25 08:19:52 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 08:11:25 UTC Pre-commit check for 99f98e2 has started.
⚪ 2024-07-25 08:14:21 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-07-25 08:18:33 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 08:23:35 UTC Pre-commit check for 37ff54f has started.
⚪ 2024-07-25 08:26:30 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-07-25 08:41:09 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 08:23:49 UTC Pre-commit check for 37ff54f has started.
⚪ 2024-07-25 08:26:48 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-25 08:37:35 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 08:24:03 UTC Pre-commit check for 37ff54f has started.
⚪ 2024-07-25 08:27:01 UTC Check linux-x86_64-release-clang14 is running...
🔴 2024-07-25 08:43:41 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 09:04:45 UTC Pre-commit check for 11ed162 has started.
⚪ 2024-07-25 09:07:26 UTC Check linux-x86_64-release-clang14 is running...
🔴 2024-07-25 09:10:36 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 09:05:36 UTC Pre-commit check for 11ed162 has started.
⚪ 2024-07-25 09:08:34 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-07-25 09:15:10 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 09:05:43 UTC Pre-commit check for 11ed162 has started.
⚪ 2024-07-25 09:08:39 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-25 09:17:23 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 09:19:44 UTC Pre-commit check for a6d51ea has started.
⚪ 2024-07-25 09:23:04 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-25 09:31:35 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 09:20:05 UTC Pre-commit check for a6d51ea has started.
⚪ 2024-07-25 09:23:05 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-25 11:40:56 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10958 10346 0 100 329 183

🟢 2024-07-25 11:42:02 UTC Build successful.
🟡 2024-07-25 11:43:10 UTC ydbd size 5.2 GiB changed* by +1.6 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: f36b8b5 merge: a6d51ea diff diff %
ydbd size 5 561 105 728 Bytes 5 562 766 816 Bytes +1.6 MiB +0.030%
ydbd stripped size 1 197 649 392 Bytes 1 197 799 632 Bytes +146.7 KiB +0.013%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 09:21:52 UTC Pre-commit check for a6d51ea has started.
⚪ 2024-07-25 09:24:49 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-07-25 11:15:41 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39640 34341 0 5 5274 20

🔴 2024-07-25 11:22:19 UTC ydbd size 8.1 GiB changed* by +2.8 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 833f929 merge: a6d51ea diff diff %
ydbd size 8 661 919 088 Bytes 8 664 885 176 Bytes +2.8 MiB +0.034%
ydbd stripped size 471 320 640 Bytes 471 398 272 Bytes +75.8 KiB +0.016%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 18:27:16 UTC Pre-commit check for 543a2d8 has started.
⚪ 2024-07-25 18:31:02 UTC Check linux-x86_64-release-clang14 is running...
⚫ 2024-07-25 18:37:25 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 18:28:54 UTC Pre-commit check for 543a2d8 has started.
⚪ 2024-07-25 18:32:16 UTC Check linux-x86_64-release-asan is running...
⚫ 2024-07-25 18:37:25 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 18:30:03 UTC Pre-commit check for 543a2d8 has started.
⚪ 2024-07-25 18:33:27 UTC Check linux-x86_64-relwithdebinfo is running...
⚫ 2024-07-25 18:37:25 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 18:38:49 UTC Pre-commit check for 0a0e1df has started.
⚪ 2024-07-25 18:42:01 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-07-25 19:05:53 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 18:38:52 UTC Pre-commit check for 0a0e1df has started.
⚪ 2024-07-25 18:42:07 UTC Check linux-x86_64-release-clang14 is running...
🔴 2024-07-25 19:17:56 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

⚪ 2024-07-25 18:41:13 UTC Pre-commit check for 0a0e1df has started.
⚪ 2024-07-25 18:44:21 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-25 18:50:36 UTC Build failed. see the logs.

@ivanmorozov333 ivanmorozov333 changed the title implement sparse implementation for merger implement sparse data accessor for merger Jul 26, 2024
@ivanmorozov333 ivanmorozov333 changed the title implement sparse data accessor for merger implement sparse data accessor Jul 26, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 27, 2024 •
edited
Loading

⚪ 2024-07-27 04:47:50 UTC Pre-commit check for 7600156 has started.
⚪ 2024-07-27 04:51:16 UTC Check linux-x86_64-release-asan is running...
⚫ 2024-07-27 04:52:16 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 27, 2024 •
edited
Loading

⚪ 2024-07-27 04:48:38 UTC Pre-commit check for 7600156 has started.
⚪ 2024-07-27 04:51:52 UTC Check linux-x86_64-release-clang14 is running...
⚫ 2024-07-27 04:52:15 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 27, 2024 •
edited
Loading

⚪ 2024-07-27 04:53:38 UTC Pre-commit check for 9d7df8a has started.
⚪ 2024-07-27 04:56:36 UTC Check linux-x86_64-release-clang14 is running...
⚫ 2024-07-27 05:05:46 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 27, 2024 •
edited
Loading

⚪ 2024-07-27 04:53:40 UTC Pre-commit check for 9d7df8a has started.
⚪ 2024-07-27 04:58:45 UTC Check linux-x86_64-relwithdebinfo is running...
⚫ 2024-07-27 05:05:47 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 27, 2024 •
edited
Loading

⚪ 2024-07-27 04:56:03 UTC Pre-commit check for 9d7df8a has started.
⚪ 2024-07-27 05:00:29 UTC Check linux-x86_64-release-asan is running...
⚫ 2024-07-27 05:05:47 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 27, 2024 •
edited
Loading

⚪ 2024-07-27 05:07:20 UTC Pre-commit check for fea6775 has started.
⚪ 2024-07-27 05:10:24 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-27 05:40:13 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 27, 2024 •
edited
Loading

⚪ 2024-07-27 05:09:58 UTC Pre-commit check for fea6775 has started.
⚪ 2024-07-27 05:13:11 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-27 07:34:30 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11099 10429 0 162 328 180

🟢 2024-07-27 07:35:36 UTC Build successful.
🔴 2024-07-27 07:36:36 UTC ydbd size 5.2 GiB changed* by +4.6 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 0032663 merge: fea6775 diff diff %
ydbd size 5 565 592 704 Bytes 5 570 381 576 Bytes +4.6 MiB +0.086%
ydbd stripped size 1 198 426 832 Bytes 1 199 193 424 Bytes +748.6 KiB +0.064%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 27, 2024 •
edited
Loading

⚪ 2024-07-27 05:10:45 UTC Pre-commit check for fea6775 has started.
⚪ 2024-07-27 05:14:08 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-07-27 07:29:24 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39786 34416 0 68 5276 26

⚪ 2024-07-27 07:33:22 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-07-27 07:41:35 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
94 (only retried tests) 13 0 66 1 14

🔴 2024-07-27 07:45:24 UTC ydbd size 8.1 GiB changed* by +8.2 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 0032663 merge: fea6775 diff diff %
ydbd size 8 668 948 704 Bytes 8 677 500 824 Bytes +8.2 MiB +0.099%
ydbd stripped size 471 559 232 Bytes 471 855 424 Bytes +289.2 KiB +0.063%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
⚫ 2024-07-27 07:45:25 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 27, 2024 •
edited
Loading

⚪ 2024-07-27 07:49:19 UTC Pre-commit check for c7013dd has started.
⚪ 2024-07-27 07:52:34 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-27 08:05:40 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 27, 2024 •
edited
Loading

⚪ 2024-07-27 07:49:23 UTC Pre-commit check for c7013dd has started.
⚪ 2024-07-27 07:52:35 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-07-27 09:49:40 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39779 34494 0 1 5274 10

⚪ 2024-07-27 09:53:31 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-07-27 10:01:30 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11 (only retried tests) 2 0 0 1 8

🟢 2024-07-27 10:04:50 UTC Build successful.
🔴 2024-07-27 10:05:33 UTC ydbd size 8.1 GiB changed* by +8.2 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 0032663 merge: c7013dd diff diff %
ydbd size 8 668 948 704 Bytes 8 677 515 480 Bytes +8.2 MiB +0.099%
ydbd stripped size 471 559 232 Bytes 471 856 384 Bytes +290.2 KiB +0.063%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 27, 2024 •
edited
Loading

⚪ 2024-07-27 07:49:32 UTC Pre-commit check for c7013dd has started.
⚪ 2024-07-27 07:52:57 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-27 10:16:32 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11097 10505 0 86 330 176

🟢 2024-07-27 10:17:37 UTC Build successful.
🔴 2024-07-27 10:18:30 UTC ydbd size 5.2 GiB changed* by +4.6 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 0032663 merge: c7013dd diff diff %
ydbd size 5 565 592 704 Bytes 5 570 401 128 Bytes +4.6 MiB +0.086%
ydbd stripped size 1 198 426 832 Bytes 1 199 196 944 Bytes +752.1 KiB +0.064%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@ivanmorozov333 ivanmorozov333 merged commit ae7146c into ydb-platform:main Jul 28, 2024
10 of 12 checks passed
@ivanmorozov333 ivanmorozov333 deleted the sparsed_arrays_usage branch July 28, 2024 05:32
@zverevgeny zverevgeny linked an issue Jul 30, 2024 that may be closed by this pull request
Effective writes for sparse data #6863
Closed
@GrigoriyPA GrigoriyPA mentioned this pull request Jul 30, 2024
Merged
MrLolthe1st pushed a commit to MrLolthe1st/ydb that referenced this pull request Aug 12, 2024
@ivanmorozov333 @MrLolthe1st
implement sparse data accessor (ydb-platform#7055)
8839545
@ildar-khisambeev ildar-khisambeev mentioned this pull request Sep 3, 2024
Merged
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 9, 2024
@ivanmorozov333 @zverevgeny
implement sparse data accessor (ydb-platform#7055)
fbc1c45 
 Conflicts:
	ydb/core/tx/columnshard/engines/changes/compaction/merger.cpp
	ydb/core/tx/columnshard/engines/scheme/versions/abstract_scheme.cpp
	ydb/core/tx/columnshard/splitter/chunks.h
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 10, 2024
@ivanmorozov333 @zverevgeny
implement sparse data accessor (ydb-platform#7055)
77704e4 
 Conflicts:
	ydb/core/tx/columnshard/engines/changes/compaction/merger.cpp
	ydb/core/tx/columnshard/engines/scheme/versions/abstract_scheme.cpp
	ydb/core/tx/columnshard/splitter/chunks.h
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 11, 2024
@ivanmorozov333 @zverevgeny
implement sparse data accessor (ydb-platform#7055)
e7555cb 
 Conflicts:
	ydb/core/tx/columnshard/engines/changes/compaction/merger.cpp
	ydb/core/tx/columnshard/engines/scheme/versions/abstract_scheme.cpp
	ydb/core/tx/columnshard/splitter/chunks.h
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 12, 2024
@ivanmorozov333 @zverevgeny
implement sparse data accessor (ydb-platform#7055)
3e05e6b 
Conflicts:
	ydb/core/tx/columnshard/engines/changes/compaction/merger.cpp
	ydb/core/tx/columnshard/engines/scheme/versions/abstract_scheme.cpp
zverevgeny added a commit to zverevgeny/ydb that referenced this pull request Sep 13, 2024
@zverevgeny
fix implement sparse data accessor (ydb-platform#7055) merge error
068b0da
zverevgeny added a commit to zverevgeny/ydb that referenced this pull request Sep 13, 2024
@zverevgeny
fix implement sparse data accessor (ydb-platform#7055) merge error
99df9f9
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 13, 2024
@ivanmorozov333 @zverevgeny
implement sparse data accessor (ydb-platform#7055)
81f6924 
 Conflicts:
	ydb/core/tx/columnshard/engines/changes/compaction/merger.cpp
	ydb/core/tx/columnshard/engines/scheme/versions/abstract_scheme.cpp
+ fix unused
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 15, 2024
@ivanmorozov333 @zverevgeny
implement sparse data accessor (ydb-platform#7055)
3282d3c 
Conflicts:
	ydb/core/tx/columnshard/engines/changes/compaction/merger.cpp
	ydb/core/tx/columnshard/engines/scheme/versions/abstract_scheme.cpp
@zverevgeny zverevgeny mentioned this pull request Sep 16, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zverevgeny zverevgeny zverevgeny approved these changes

Assignees
No one assigned
Labels
new-feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Effective writes for sparse data
2 participants
@ivanmorozov333 @zverevgeny