Skip to content

HTAP: Query Processor #8917

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 18 commits into from
Sep 11, 2024
Merged

HTAP: Query Processor #8917

merged 18 commits into from
Sep 11, 2024

Conversation

nikvas0
Copy link
Collaborator

@nikvas0 nikvas0 commented Sep 9, 2024

Changelog entry

Always use arbiter if there are columnshards

Changelog category

  • Not for changelog (changelog entry is not required)

Additional information

...

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 9, 2024 •
edited
Loading

⚪ 2024-09-09 07:10:46 UTC Pre-commit check linux-x86_64-release-clang14 for 7701b0e has started.
⚪ 2024-09-09 07:13:58 UTC ya make is running...
🟢 2024-09-09 08:02:52 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 9, 2024 •
edited
Loading

⚪ 2024-09-09 07:10:49 UTC Pre-commit check linux-x86_64-release-asan for 7701b0e has started.
⚪ 2024-09-09 07:14:07 UTC ya make is running...
🔴 2024-09-09 09:27:27 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11548 11069 0 194 164 121

🟢 2024-09-09 09:28:30 UTC Build successful.
🟢 2024-09-09 09:29:07 UTC ydbd size 5.6 GiB changed* by +53.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: b51e0c4 merge: 7701b0e diff diff %
ydbd size 6 031 994 520 Bytes 6 032 049 208 Bytes +53.4 KiB +0.001%
ydbd stripped size 1 510 130 608 Bytes 1 510 142 512 Bytes +11.6 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 9, 2024 •
edited
Loading

⚪ 2024-09-09 07:13:47 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7701b0e has started.
⚪ 2024-09-09 07:17:06 UTC ya make is running...
🟡 2024-09-09 09:06:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40822 34820 0 312 5578 112

⚪ 2024-09-09 09:10:45 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-09-09 09:25:46 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
596 (only retried tests) 150 0 279 50 117

⚪ 2024-09-09 09:25:58 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-09-09 09:40:18 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
446 (only retried tests) 42 0 278 24 102

🟢 2024-09-09 09:40:27 UTC Build successful.
🟢 2024-09-09 09:41:29 UTC ydbd size 8.4 GiB changed* by +10.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: b51e0c4 merge: 7701b0e diff diff %
ydbd size 9 017 211 592 Bytes 9 017 221 976 Bytes +10.1 KiB +0.000%
ydbd stripped size 487 420 872 Bytes 487 423 368 Bytes +2.4 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@nikvas0 nikvas0 linked an issue Sep 9, 2024 that may be closed by this pull request
HTAP #8918
Closed
4 tasks
@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 9, 2024 •
edited
Loading

⚪ 2024-09-09 12:26:40 UTC Pre-commit check linux-x86_64-release-clang14 for 902a830 has started.
⚫ 2024-09-09 12:28:09 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 9, 2024 •
edited
Loading

⚪ 2024-09-09 12:37:10 UTC Pre-commit check linux-x86_64-release-clang14 for a613b6f has started.
⚪ 2024-09-09 12:40:26 UTC ya make is running...
🔴 2024-09-09 13:22:14 UTC Build failed, see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 9, 2024 •
edited
Loading

⚪ 2024-09-09 12:41:22 UTC Pre-commit check linux-x86_64-relwithdebinfo for a613b6f has started.
⚪ 2024-09-09 12:45:52 UTC ya make is running...
🔴 2024-09-09 13:29:17 UTC Build failed, see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 9, 2024 •
edited
Loading

⚪ 2024-09-09 12:45:45 UTC Pre-commit check linux-x86_64-release-asan for a613b6f has started.
⚪ 2024-09-09 12:50:00 UTC ya make is running...
🔴 2024-09-09 13:32:01 UTC Build failed, see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 06:52:46 UTC Pre-commit check linux-x86_64-relwithdebinfo for 592c849 has started.
⚪ 2024-09-10 06:55:58 UTC ya make is running...
🟡 2024-09-10 08:38:22 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40835 34829 0 308 5586 112

⚪ 2024-09-10 08:41:55 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-09-10 08:56:37 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
593 (only retried tests) 140 0 277 50 126

⚪ 2024-09-10 08:56:48 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-09-10 09:11:55 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
452 (only retried tests) 53 0 262 34 103

🟢 2024-09-10 09:12:03 UTC Build successful.
🟡 2024-09-10 09:15:15 UTC ydbd size 8.4 GiB changed* by +367.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: dd142f6 merge: 592c849 diff diff %
ydbd size 9 024 182 440 Bytes 9 024 559 128 Bytes +367.9 KiB +0.004%
ydbd stripped size 487 703 400 Bytes 487 720 392 Bytes +16.6 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 06:52:48 UTC Pre-commit check linux-x86_64-release-asan for 592c849 has started.
⚪ 2024-09-10 06:56:05 UTC ya make is running...
🔴 2024-09-10 09:10:52 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11548 11064 0 202 150 132

🟢 2024-09-10 09:12:00 UTC Build successful.
🟡 2024-09-10 09:12:41 UTC ydbd size 5.6 GiB changed* by +244.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: dd142f6 merge: 592c849 diff diff %
ydbd size 6 036 912 800 Bytes 6 037 163 072 Bytes +244.4 KiB +0.004%
ydbd stripped size 1 511 084 976 Bytes 1 511 136 400 Bytes +50.2 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 06:53:40 UTC Pre-commit check linux-x86_64-release-clang14 for 592c849 has started.
⚪ 2024-09-10 06:57:00 UTC ya make is running...
🟢 2024-09-10 07:49:57 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 09:55:41 UTC Pre-commit check linux-x86_64-release-clang14 for f50c5cf has started.
⚪ 2024-09-10 09:56:22 UTC Artifacts will be uploaded here
⚪ 2024-09-10 10:00:02 UTC ya make is running...
🟢 2024-09-10 10:20:45 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 09:56:49 UTC Pre-commit check linux-x86_64-release-asan for f50c5cf has started.
⚪ 2024-09-10 09:56:59 UTC Artifacts will be uploaded here
⚪ 2024-09-10 10:00:17 UTC ya make is running...
🔴 2024-09-10 11:44:34 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11532 11466 0 14 25 27

🟢 2024-09-10 11:45:33 UTC Build successful.
🟡 2024-09-10 11:46:02 UTC ydbd size 5.6 GiB changed* by +244.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 7001202 merge: f50c5cf diff diff %
ydbd size 6 037 335 152 Bytes 6 037 584 992 Bytes +244.0 KiB +0.004%
ydbd stripped size 1 511 156 464 Bytes 1 511 207 760 Bytes +50.1 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 09:57:01 UTC Pre-commit check linux-x86_64-relwithdebinfo for f50c5cf has started.
⚪ 2024-09-10 09:57:12 UTC Artifacts will be uploaded here
⚪ 2024-09-10 10:00:20 UTC ya make is running...
🟡 2024-09-10 11:18:05 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40816 35352 0 7 5431 26

⚪ 2024-09-10 11:21:43 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-10 11:32:01 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
43 (only retried tests) 22 0 0 0 21

🟢 2024-09-10 11:32:08 UTC Build successful.
🟡 2024-09-10 11:32:48 UTC ydbd size 8.4 GiB changed* by +369.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 17a9a61 merge: f50c5cf diff diff %
ydbd size 9 024 517 808 Bytes 9 024 896 464 Bytes +369.8 KiB +0.004%
ydbd stripped size 487 708 328 Bytes 487 725 640 Bytes +16.9 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 12:19:14 UTC Pre-commit check linux-x86_64-release-clang14 for 2cae56a has started.
⚪ 2024-09-10 12:19:41 UTC Artifacts will be uploaded here
⚪ 2024-09-10 12:22:55 UTC ya make is running...
🟢 2024-09-10 12:29:03 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 13:38:47 UTC Pre-commit check linux-x86_64-release-asan for be7fb48 has started.
⚪ 2024-09-10 13:38:59 UTC Artifacts will be uploaded here
⚪ 2024-09-10 13:42:04 UTC ya make is running...
⚫ 2024-09-10 13:54:38 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 13:38:53 UTC Pre-commit check linux-x86_64-relwithdebinfo for be7fb48 has started.
⚪ 2024-09-10 13:39:03 UTC Artifacts will be uploaded here
⚪ 2024-09-10 13:42:03 UTC ya make is running...
⚫ 2024-09-10 13:54:37 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 13:39:17 UTC Pre-commit check linux-x86_64-release-clang14 for be7fb48 has started.
⚪ 2024-09-10 13:39:27 UTC Artifacts will be uploaded here
⚪ 2024-09-10 13:42:35 UTC ya make is running...
⚫ 2024-09-10 13:54:36 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 13:56:14 UTC Pre-commit check linux-x86_64-relwithdebinfo for cf5ac64 has started.
⚪ 2024-09-10 13:56:46 UTC Artifacts will be uploaded here
⚪ 2024-09-10 14:00:18 UTC ya make is running...
⚫ 2024-09-10 15:18:54 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 13:57:59 UTC Pre-commit check linux-x86_64-release-clang14 for cf5ac64 has started.
⚪ 2024-09-10 13:58:09 UTC Artifacts will be uploaded here
⚪ 2024-09-10 14:01:11 UTC ya make is running...
🟢 2024-09-10 14:35:59 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 13:58:45 UTC Pre-commit check linux-x86_64-release-asan for cf5ac64 has started.
⚪ 2024-09-10 13:58:55 UTC Artifacts will be uploaded here
⚪ 2024-09-10 14:01:58 UTC ya make is running...
⚫ 2024-09-10 15:18:53 UTC Check cancelled

@nikvas0 nikvas0 requested a review from gridnevvvit September 10, 2024 15:17
@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 15:19:37 UTC Pre-commit check linux-x86_64-release-clang14 for a8246c3 has started.
⚪ 2024-09-10 15:19:41 UTC Artifacts will be uploaded here
⚪ 2024-09-10 15:22:42 UTC ya make is running...
🟢 2024-09-10 15:36:54 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 15:22:34 UTC Pre-commit check linux-x86_64-release-asan for a8246c3 has started.
⚪ 2024-09-10 15:22:57 UTC Artifacts will be uploaded here
⚪ 2024-09-10 15:25:56 UTC ya make is running...
🔴 2024-09-10 17:00:20 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11779 11485 0 6 258 30

🟢 2024-09-10 17:01:40 UTC Build successful.
🟡 2024-09-10 17:04:24 UTC ydbd size 5.6 GiB changed* by +288.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 26c9911 merge: a8246c3 diff diff %
ydbd size 6 038 719 744 Bytes 6 039 015 128 Bytes +288.5 KiB +0.005%
ydbd stripped size 1 511 295 472 Bytes 1 511 355 280 Bytes +58.4 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 15:24:21 UTC Pre-commit check linux-x86_64-relwithdebinfo for a8246c3 has started.
⚪ 2024-09-10 15:24:48 UTC Artifacts will be uploaded here
⚪ 2024-09-10 15:27:52 UTC ya make is running...
🟡 2024-09-10 17:05:38 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
41073 35376 0 6 5666 25

⚪ 2024-09-10 17:10:13 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-10 17:20:17 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
274 (only retried tests) 183 0 0 7 84

🟢 2024-09-10 17:20:27 UTC Build successful.
🟡 2024-09-10 17:21:05 UTC ydbd size 8.4 GiB changed* by +399.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: ce3201c merge: a8246c3 diff diff %
ydbd size 9 026 362 848 Bytes 9 026 771 640 Bytes +399.2 KiB +0.005%
ydbd stripped size 487 745 192 Bytes 487 768 776 Bytes +23.0 KiB +0.005%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 20:47:51 UTC Pre-commit check linux-x86_64-release-clang14 for b2ff69b has started.
⚪ 2024-09-10 20:48:00 UTC Artifacts will be uploaded here
⚪ 2024-09-10 20:51:06 UTC ya make is running...
🟢 2024-09-10 21:40:56 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 20:49:24 UTC Pre-commit check linux-x86_64-release-asan for b2ff69b has started.
⚪ 2024-09-10 20:49:34 UTC Artifacts will be uploaded here
⚪ 2024-09-10 20:52:32 UTC ya make is running...
🔴 2024-09-10 22:55:38 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11785 11635 0 21 31 98

🟢 2024-09-10 22:56:38 UTC Build successful.
🟡 2024-09-10 22:59:22 UTC ydbd size 5.6 GiB changed* by +288.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 189aa4b merge: b2ff69b diff diff %
ydbd size 6 038 777 360 Bytes 6 039 073 144 Bytes +288.9 KiB +0.005%
ydbd stripped size 1 511 302 256 Bytes 1 511 362 192 Bytes +58.5 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 20:49:51 UTC Pre-commit check linux-x86_64-relwithdebinfo for b2ff69b has started.
⚪ 2024-09-10 20:50:01 UTC Artifacts will be uploaded here
⚪ 2024-09-10 20:53:15 UTC ya make is running...
🟡 2024-09-10 22:39:26 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
41076 35526 0 9 5447 94

⚪ 2024-09-10 22:42:58 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-09-10 22:53:34 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
119 (only retried tests) 31 0 2 0 86

⚪ 2024-09-10 22:53:43 UTC ya make is running... (failed tests rerun, try 3)
🟢 2024-09-10 23:01:07 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
93 (only retried tests) 10 0 0 0 83

🟢 2024-09-10 23:01:14 UTC Build successful.
🟡 2024-09-10 23:01:56 UTC ydbd size 8.4 GiB changed* by +395.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 189aa4b merge: b2ff69b diff diff %
ydbd size 9 026 486 672 Bytes 9 026 891 392 Bytes +395.2 KiB +0.004%
ydbd stripped size 487 751 912 Bytes 487 771 400 Bytes +19.0 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

gridnevvvit
gridnevvvit reviewed Sep 11, 2024
View reviewed changes
ydb/core/kqp/executer_actor/kqp_data_executer.cpp
@@ -2437,6 +2483,12 @@ class TKqpDataExecuter : public TKqpExecuterBase<TKqpDataExecuter, EExecType::Da
std::sort(sendingShards.begin(), sendingShards.end());
std::sort(receivingShards.begin(), receivingShards.end());

NProtoBuf::RepeatedField<ui64> sendingColumnShards(sendingColumnShardsSet.begin(), sendingColumnShardsSet.end());
Copy link
Member

@gridnevvvit gridnevvvit Sep 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

а в чем принципиальная необходимость собирать их отдельно от даташардов?

@gridnevvvit gridnevvvit self-requested a review September 11, 2024 08:38
@nikvas0 nikvas0 merged commit 71c4756 into ydb-platform:main Sep 11, 2024
10 of 12 checks passed
@shnikd shnikd mentioned this pull request Sep 11, 2024
Closed
nikvas0 added a commit to nikvas0/ydb that referenced this pull request Sep 11, 2024
@nikvas0
HTAP: Query Processor (ydb-platform#8917)
d92cbf0
This was referenced Sep 12, 2024
Merged
Merged
Merged
@zverevgeny zverevgeny mentioned this pull request Sep 16, 2024
Closed
nikvas0 added a commit to nikvas0/ydb that referenced this pull request Sep 17, 2024
@nikvas0
HTAP: Query Processor (ydb-platform#8917)
0093054
@StekPerepolnen StekPerepolnen mentioned this pull request Sep 17, 2024
Merged
This was referenced Sep 18, 2024
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gridnevvvit gridnevvvit gridnevvvit approved these changes

Assignees
No one assigned
Labels
not-for-changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

HTAP
2 participants
@nikvas0 @gridnevvvit