Added Query Hints for the Optimizer #7629

pavelvelikhov · 2024-08-09T16:40:29Z

Changelog entry

Query Hints for Cardinality and Join algorithms are added to the Cost Based Optimizer

Changelog category

Improvement

Additional information

Here is an example of how the new hints can be used in queries:

Cardinality hints: a list of hints, each hint specifies the aliases or table names that are joined and specifies how the cardinality needs to be changed. Options for the 'op' field are: +,-,*,/ and #, where # means the value will be replaced by a constant
Join hints: each hint specifies the aliases or table names that are joined and specifies the algorithm that needs to be picked for this specific join.

PRAGMA ydb.OptCardinalityHints = '[{"labels":["R"], "op":"#", "value":1}, {"labels":["R","S"], "op":"#", "value":10e6}]';
PRAGMA ydb.OptJoinAlgoHints = '[{"labels":["R","S"], "algo":"GraceJoin"}]';

SELECT *
FROM `/Root/R` as R
    INNER JOIN
        `/Root/S` as S
    ON R.id = S.id

github-actions · 2024-08-09T16:43:33Z

⚪ 2024-08-09 16:56:31 UTC Pre-commit check for 1f41d0c has started.
⚪ 2024-08-09 17:00:13 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-09 17:05:25 UTC Build failed. see the logs.

github-actions · 2024-08-09T16:43:37Z

⚪ 2024-08-09 16:58:23 UTC Pre-commit check for 1f41d0c has started.
⚪ 2024-08-09 17:01:19 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-08-09 17:07:42 UTC Build failed. see the logs.

github-actions · 2024-08-09T16:43:59Z

⚪ 2024-08-09 16:57:59 UTC Pre-commit check for 1f41d0c has started.
⚪ 2024-08-09 17:00:56 UTC Check linux-x86_64-release-clang14 is running...
🔴 2024-08-09 17:04:09 UTC Build failed. see the logs.

github-actions · 2024-08-11T18:58:58Z

⚪ 2024-08-12 05:59:07 UTC Pre-commit check for cb96689 has started.
⚪ 2024-08-12 06:02:08 UTC Check linux-x86_64-release-clang14 is running...
🔴 2024-08-12 06:06:01 UTC Build failed. see the logs.

github-actions · 2024-08-11T18:59:08Z

⚪ 2024-08-12 05:58:52 UTC Pre-commit check for cb96689 has started.
⚪ 2024-08-12 06:01:46 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-12 06:06:40 UTC Build failed. see the logs.

github-actions · 2024-08-11T18:59:38Z

⚪ 2024-08-12 05:59:10 UTC Pre-commit check for cb96689 has started.
⚪ 2024-08-12 06:02:13 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-08-12 06:08:53 UTC Build failed. see the logs.

github-actions · 2024-08-12T06:12:38Z

⚪ 2024-08-12 07:11:58 UTC Pre-commit check for a4d6773 has started.
⚪ 2024-08-12 07:14:51 UTC Check linux-x86_64-release-clang14 is running...
🔴 2024-08-12 07:17:33 UTC Build failed. see the logs.

github-actions · 2024-08-12T06:13:14Z

⚪ 2024-08-12 07:11:23 UTC Pre-commit check for a4d6773 has started.
⚪ 2024-08-12 07:14:26 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-08-12 07:21:31 UTC Build failed. see the logs.

github-actions · 2024-08-12T06:13:26Z

⚪ 2024-08-12 07:11:27 UTC Pre-commit check for a4d6773 has started.
⚪ 2024-08-12 07:14:30 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-12 07:20:28 UTC Build failed. see the logs.

github-actions · 2024-08-12T07:40:27Z

⚪ 2024-08-12 07:40:26 UTC Pre-commit check for db3966e has started.
⚪ 2024-08-12 07:43:29 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-08-12 10:31:27 UTC Too many tests failed, NOT going to retry
🔴 2024-08-12 10:34:36 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
76703	62575	0	515	13597	16

🟢 2024-08-12 10:44:18 UTC Build successful.
🟡 2024-08-12 10:44:56 UTC ydbd size 8.1 GiB changed* by +1.4 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `a2e8760`	merge: `db3966e`	diff	diff %
ydbd size	8 694 050 184 Bytes	8 695 510 088 Bytes	+1.4 MiB	+0.017%
ydbd stripped size	472 889 384 Bytes	472 943 336 Bytes	+52.7 KiB	+0.011%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-08-12T07:40:28Z

⚪ 2024-08-12 07:40:28 UTC Pre-commit check for db3966e has started.
⚪ 2024-08-12 07:43:27 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-12 08:03:40 UTC Build successful.

github-actions · 2024-08-12T07:40:53Z

⚪ 2024-08-12 07:40:53 UTC Pre-commit check for db3966e has started.
⚪ 2024-08-12 07:43:59 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-12 10:21:17 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15130	15063	0	21	32	14

🟢 2024-08-12 10:22:57 UTC Build successful.
🟡 2024-08-12 10:23:26 UTC ydbd size 5.4 GiB changed* by +952.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `a2e8760`	merge: `db3966e`	diff	diff %
ydbd size	5 846 754 776 Bytes	5 847 729 784 Bytes	+952.2 KiB	+0.017%
ydbd stripped size	1 468 579 088 Bytes	1 468 768 176 Bytes	+184.7 KiB	+0.013%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-08-12T20:31:41Z

⚪ 2024-08-12 20:31:41 UTC Pre-commit check for 56450b4 has started.
⚪ 2024-08-12 20:34:34 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-12 20:55:22 UTC Build successful.

github-actions · 2024-08-12T20:32:06Z

⚪ 2024-08-12 20:32:06 UTC Pre-commit check for 56450b4 has started.
⚪ 2024-08-12 20:35:03 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-12 22:37:06 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
76744	63092	0	44	13596	12

⚪ 2024-08-12 22:47:50 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-08-12 22:55:26 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
60 (only retried tests)	8	0	42	1	9

🟡 2024-08-12 22:56:06 UTC ydbd size 8.1 GiB changed* by +1.4 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `c66a55e`	merge: `56450b4`	diff	diff %
ydbd size	8 700 374 976 Bytes	8 701 839 000 Bytes	+1.4 MiB	+0.017%
ydbd stripped size	473 081 160 Bytes	473 139 208 Bytes	+56.7 KiB	+0.012%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-08-12T20:32:16Z

⚪ 2024-08-12 20:32:15 UTC Pre-commit check for 56450b4 has started.
⚪ 2024-08-12 20:35:25 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-12 22:56:28 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15140	15081	0	15	31	13

🟢 2024-08-12 22:58:07 UTC Build successful.
🟡 2024-08-12 22:58:35 UTC ydbd size 5.5 GiB changed* by +948.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `c66a55e`	merge: `56450b4`	diff	diff %
ydbd size	5 851 206 264 Bytes	5 852 177 192 Bytes	+948.2 KiB	+0.017%
ydbd stripped size	1 469 322 960 Bytes	1 469 507 952 Bytes	+180.7 KiB	+0.013%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-08-13T12:28:13Z

⚪ 2024-08-13 12:28:12 UTC Pre-commit check for d310794 has started.
⚪ 2024-08-13 12:31:20 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-13 14:21:47 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
76749	63126	0	4	13606	13

⚪ 2024-08-13 14:31:07 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-13 14:39:37 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
28 (only retried tests)	19	0	0	0	9

🟢 2024-08-13 14:39:45 UTC Build successful.
🟡 2024-08-13 14:40:17 UTC ydbd size 8.1 GiB changed* by +1.4 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `7c90c71`	merge: `d310794`	diff	diff %
ydbd size	8 700 589 208 Bytes	8 702 069 024 Bytes	+1.4 MiB	+0.017%
ydbd stripped size	473 092 552 Bytes	473 151 240 Bytes	+57.3 KiB	+0.012%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

pavelvelikhov · 2024-08-13T12:41:52Z

ydb/library/yql/providers/dq/planner/execution_planner.cpp

@zverevgeny Можешь посмотреть, нормальное изменение?

Changed email

github-actions · 2024-08-13T15:12:53Z

⚪ 2024-08-13 15:12:53 UTC Pre-commit check for ab3d0f0 has started.
⚪ 2024-08-13 15:16:52 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-13 15:23:32 UTC Build successful.

github-actions · 2024-08-13T15:16:19Z

⚪ 2024-08-13 15:16:19 UTC Pre-commit check for ab3d0f0 has started.
⚪ 2024-08-13 15:19:57 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-13 17:39:46 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15146	15076	0	12	44	14

🟢 2024-08-13 17:42:11 UTC Build successful.
🟡 2024-08-13 17:42:42 UTC ydbd size 5.5 GiB changed* by +959.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `4bce249`	merge: `ab3d0f0`	diff	diff %
ydbd size	5 851 518 760 Bytes	5 852 500 992 Bytes	+959.2 KiB	+0.017%
ydbd stripped size	1 469 384 272 Bytes	1 469 567 600 Bytes	+179.0 KiB	+0.012%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-08-13T15:17:42Z

⚪ 2024-08-13 15:17:42 UTC Pre-commit check for ab3d0f0 has started.
⚪ 2024-08-13 15:21:24 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-13 17:20:39 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
76752	63135	0	3	13597	17

⚪ 2024-08-13 17:30:23 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-13 17:41:06 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
25 (only retried tests)	14	0	0	1	10

🟢 2024-08-13 17:41:14 UTC Build successful.
🟡 2024-08-13 17:41:52 UTC ydbd size 8.1 GiB changed* by +1.4 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `4bce249`	merge: `ab3d0f0`	diff	diff %
ydbd size	8 700 672 744 Bytes	8 702 148 336 Bytes	+1.4 MiB	+0.017%
ydbd stripped size	473 100 232 Bytes	473 154 888 Bytes	+53.4 KiB	+0.012%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

yumkam · 2024-08-13T17:20:17Z

ydb/library/yql/providers/dq/planner/execution_planner.cpp

@@ -40,6 +40,17 @@ using namespace NKikimr::NMiniKQL;

 using namespace Yql::DqsProto;

+namespace {
+    TString RemoveAliases(TString attributeName) {
+        for (size_t i = attributeName.size() - 1; i>0; i--) {


nit: .rfind('.')?

А-а, find_last_of тут подошел бы, но кажется он у меня не сработал почему-то

size_t -> int

pavelvelikhov requested review from a team as code owners August 9, 2024 16:40

github-actions bot added the improvement label Aug 9, 2024

pavelvelikhov changed the title ~~Card hints~~ Added Qery Hints for the Optimizer Aug 9, 2024

github-actions bot added improvement and removed improvement labels Aug 9, 2024

pavelvelikhov changed the title ~~Added Qery Hints for the Optimizer~~ Added Query Hints for the Optimizer Aug 12, 2024

github-actions bot added improvement and removed improvement labels Aug 12, 2024

pavelvelikhov added 5 commits August 13, 2024 06:20

Early commit to save work

a072ebc

Finished basic implementation

e12ecf8

Fixed some issues and unit tests

f095c29

Minor code cleanup

c596fb7

Fixed build errors

aadabf7

pavelvelikhov force-pushed the CardHints branch from e82fffc to aadabf7 Compare August 13, 2024 06:53

pavelvelikhov commented Aug 13, 2024

View reviewed changes

pashandor789 previously approved these changes Aug 13, 2024

View reviewed changes

github-actions bot added improvement and removed improvement labels Aug 13, 2024

resetius previously approved these changes Aug 13, 2024

View reviewed changes

Fixed memory leak and minor cleanup

4da8869

Changed email

pavelvelikhov dismissed stale reviews from resetius and pashandor789 via 4da8869 August 13, 2024 15:10

pashandor789 self-requested a review August 13, 2024 15:13

pashandor789 approved these changes Aug 13, 2024

View reviewed changes

pavelvelikhov requested a review from resetius August 13, 2024 15:37

yumkam reviewed Aug 13, 2024

View reviewed changes

pavelvelikhov enabled auto-merge (squash) August 13, 2024 17:47

resetius approved these changes Aug 13, 2024

View reviewed changes

pavelvelikhov merged commit 8053e6d into ydb-platform:main Aug 13, 2024
10 of 12 checks passed

naspirato pushed a commit to naspirato/ydb that referenced this pull request Aug 14, 2024

Added Query Hints for the Optimizer (ydb-platform#7629)

c785fe1

pavelvelikhov added a commit to pavelvelikhov/ydb that referenced this pull request Sep 5, 2024

Added Query Hints for the Optimizer (ydb-platform#7629)

0761181

Added Query Hints for the Optimizer #7629

Added Query Hints for the Optimizer #7629

Uh oh!

Conversation

pavelvelikhov commented Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog entry

Changelog category

Additional information

Uh oh!

github-actions bot commented Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pavelvelikhov Aug 13, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yumkam Aug 13, 2024

Choose a reason for hiding this comment

Uh oh!

pavelvelikhov Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

resetius Aug 13, 2024

Choose a reason for hiding this comment

Uh oh!

pavelvelikhov commented Aug 9, 2024 •

edited

Loading

github-actions bot commented Aug 9, 2024 •

edited

Loading

github-actions bot commented Aug 9, 2024 •

edited

Loading

github-actions bot commented Aug 9, 2024 •

edited

Loading

github-actions bot commented Aug 11, 2024 •

edited

Loading

github-actions bot commented Aug 11, 2024 •

edited

Loading

github-actions bot commented Aug 11, 2024 •

edited

Loading

github-actions bot commented Aug 12, 2024 •

edited

Loading

github-actions bot commented Aug 12, 2024 •

edited

Loading

github-actions bot commented Aug 12, 2024 •

edited

Loading

github-actions bot commented Aug 12, 2024 •

edited

Loading

github-actions bot commented Aug 12, 2024 •

edited

Loading

github-actions bot commented Aug 12, 2024 •

edited

Loading

github-actions bot commented Aug 12, 2024 •

edited

Loading

github-actions bot commented Aug 12, 2024 •

edited

Loading

github-actions bot commented Aug 12, 2024 •

edited

Loading

github-actions bot commented Aug 13, 2024 •

edited

Loading

github-actions bot commented Aug 13, 2024 •

edited

Loading

github-actions bot commented Aug 13, 2024 •

edited

Loading

github-actions bot commented Aug 13, 2024 •

edited

Loading

pavelvelikhov Aug 13, 2024 •

edited

Loading