Fix multiple streamlookup #13426

yumkam · 2025-01-16T09:12:42Z

Changelog entry

Handle multiple streamlookup joins in query

Changelog category

Bugfix
Not for changelog (changelog entry is not required)

Additional information

Fixes

SELECT ... FROM a
LEFT JOIN /* streamlookup() */ b ON ...
LEFT JOIN /* streamlookup() */ c ON ...

There were several issues:

DqReadWrap -> TDqLookupSourceWrap conversion failed to handle ExtractMembers (current fix: just drop)
DqJoin -> DqCnStreamLookup conversion failed to handle ExtractMembers (current fix: just drop)
DqReadWrap -> TDqLookupSourceWrap conversion only handled 2-way EquiJoin
DqJoin -> DqCnStreamLookup conversion failed to handle (Void) as LeftLabel

(should be replaced in logical optimize)

github-actions · 2025-01-16T09:14:00Z

⚪ 2025-01-16 09:13:59 UTC Pre-commit check linux-x86_64-release-asan for 3f935b0 has started.
⚪ 2025-01-16 09:14:11 UTC Artifacts will be uploaded here
⚪ 2025-01-16 09:17:10 UTC ya make is running...
🟡 2025-01-16 10:16:49 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11907	11841	0	25	13	28

⚪ 2025-01-16 10:17:49 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-16 10:30:06 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
111 (only retried tests)	83	0	2	2	24

⚪ 2025-01-16 10:30:15 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-16 10:41:34 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
50 (only retried tests)	24	0	1	2	23

🟢 2025-01-16 10:41:41 UTC Build successful.
🟡 2025-01-16 10:42:08 UTC ydbd size 3.6 GiB changed* by +141.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `3c12dfd`	merge: `3f935b0`	diff	diff %
ydbd size	3 902 731 168 Bytes	3 902 876 240 Bytes	+141.7 KiB	+0.004%
ydbd stripped size	1 362 485 872 Bytes	1 362 526 224 Bytes	+39.4 KiB	+0.003%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-16T09:14:01Z

⚪ 2025-01-16 09:14:01 UTC Pre-commit check linux-x86_64-relwithdebinfo for 3f935b0 has started.
⚪ 2025-01-16 09:14:12 UTC Artifacts will be uploaded here
⚪ 2025-01-16 09:17:04 UTC ya make is running...
🟡 2025-01-16 10:07:46 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
25848	23336	0	9	2380	123

⚪ 2025-01-16 10:10:24 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-16 10:23:05 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
193 (only retried tests)	75	0	1	0	117

⚪ 2025-01-16 10:23:15 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-01-16 10:36:09 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
170 (only retried tests)	53	0	0	0	117

🟢 2025-01-16 10:36:24 UTC Build successful.
🟢 2025-01-16 10:36:45 UTC ydbd size 2.1 GiB changed* by +53.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `3c12dfd`	merge: `3f935b0`	diff	diff %
ydbd size	2 252 413 208 Bytes	2 252 468 312 Bytes	+53.8 KiB	+0.002%
ydbd stripped size	473 386 000 Bytes	473 393 232 Bytes	+7.1 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-17T10:10:11Z

⚪ 2025-01-17 10:10:11 UTC Pre-commit check linux-x86_64-release-asan for a7b411f has started.
⚪ 2025-01-17 10:10:23 UTC Artifacts will be uploaded here
⚪ 2025-01-17 10:13:16 UTC ya make is running...
🟡 2025-01-17 11:08:04 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11622	11570	0	19	8	25

⚪ 2025-01-17 11:09:07 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-17 11:19:17 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
86 (only retried tests)	63	0	0	0	23

🟢 2025-01-17 11:19:24 UTC Build successful.
🟡 2025-01-17 11:19:49 UTC ydbd size 3.6 GiB changed* by +145.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `db1f75f`	merge: `a7b411f`	diff	diff %
ydbd size	3 903 151 288 Bytes	3 903 300 480 Bytes	+145.7 KiB	+0.004%
ydbd stripped size	1 362 659 120 Bytes	1 362 703 568 Bytes	+43.4 KiB	+0.003%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-17T10:10:15Z

⚪ 2025-01-17 10:10:14 UTC Pre-commit check linux-x86_64-relwithdebinfo for a7b411f has started.
⚪ 2025-01-17 10:10:26 UTC Artifacts will be uploaded here
⚪ 2025-01-17 10:13:33 UTC ya make is running...
🟡 2025-01-17 11:05:00 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
25866	23359	0	4	2380	123

⚪ 2025-01-17 11:07:05 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-17 11:19:42 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
184 (only retried tests)	65	0	0	0	119

🟢 2025-01-17 11:19:49 UTC Build successful.
🟢 2025-01-17 11:20:11 UTC ydbd size 2.1 GiB changed* by +53.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `db1f75f`	merge: `a7b411f`	diff	diff %
ydbd size	2 252 494 928 Bytes	2 252 549 984 Bytes	+53.8 KiB	+0.002%
ydbd stripped size	473 412 144 Bytes	473 419 312 Bytes	+7.0 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

rvu1024 · 2025-01-17T13:07:49Z

ydb/library/yql/dq/type_ann/dq_type_ann.cpp

    const auto leftRowType = GetSeqItemType(leftInputType);
-    const auto rightRowType = GetSeqItemType(cnStreamLookup.RightInput().Raw()->GetTypeAnn());
+    if (!leftRowType) {
+        ctx.AddError(TIssue(ctx.GetPosition(rightInput.Pos()), "DqCnStreamLookup: Failed to annotate left row type"));


улучшить сообщение, включить тип

Унифицировал код с GetDqJoinResultType(input), там тип входит в текст исключения

ydb/library/yql/dq/type_ann/dq_type_ann.cpp

already added by GetDqJoinResultType

github-actions · 2025-01-17T15:39:17Z

⚪ 2025-01-17 15:39:17 UTC Pre-commit check linux-x86_64-relwithdebinfo for 295bc81 has started.
⚪ 2025-01-17 15:39:29 UTC Artifacts will be uploaded here
⚪ 2025-01-17 15:42:34 UTC ya make is running...
⚫ 2025-01-17 15:42:53 UTC Check cancelled

github-actions · 2025-01-17T15:41:38Z

⚪ 2025-01-17 15:41:38 UTC Pre-commit check linux-x86_64-release-asan for 295bc81 has started.
⚪ 2025-01-17 15:42:07 UTC Artifacts will be uploaded here
⚫ 2025-01-17 15:42:51 UTC Check cancelled

github-actions · 2025-01-17T15:46:55Z

⚪ 2025-01-17 15:46:54 UTC Pre-commit check linux-x86_64-release-asan for 2021b88 has started.
⚪ 2025-01-17 15:46:58 UTC Artifacts will be uploaded here
⚫ 2025-01-17 15:48:51 UTC Check cancelled

github-actions · 2025-01-17T15:46:57Z

⚪ 2025-01-17 15:46:57 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2021b88 has started.
⚪ 2025-01-17 15:47:08 UTC Artifacts will be uploaded here
⚫ 2025-01-17 15:48:51 UTC Check cancelled

github-actions · 2025-01-17T15:50:42Z

⚪ 2025-01-17 15:50:41 UTC Pre-commit check linux-x86_64-release-asan for ee102c1 has started.
⚪ 2025-01-17 15:50:53 UTC Artifacts will be uploaded here
⚪ 2025-01-17 15:53:56 UTC ya make is running...
🔴 2025-01-17 16:05:25 UTC Build failed, see the logs. Also see fail summary

github-actions · 2025-01-17T15:55:27Z

⚪ 2025-01-17 15:55:27 UTC Pre-commit check linux-x86_64-relwithdebinfo for ee102c1 has started.
⚪ 2025-01-17 15:55:44 UTC Artifacts will be uploaded here
⚪ 2025-01-17 15:58:47 UTC ya make is running...
🔴 2025-01-17 16:07:05 UTC Build failed, see the logs. Also see fail summary

github-actions · 2025-01-17T16:59:57Z

⚪ 2025-01-17 16:59:56 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5ab99de has started.
⚪ 2025-01-17 17:00:08 UTC Artifacts will be uploaded here
⚪ 2025-01-17 17:03:00 UTC ya make is running...
⚫ 2025-01-17 17:59:26 UTC Check cancelled

github-actions · 2025-01-17T17:00:33Z

⚪ 2025-01-17 17:00:32 UTC Pre-commit check linux-x86_64-release-asan for 5ab99de has started.
⚪ 2025-01-17 17:00:45 UTC Artifacts will be uploaded here
⚪ 2025-01-17 17:03:49 UTC ya make is running...
⚫ 2025-01-17 17:59:23 UTC Check cancelled

github-actions · 2025-01-17T18:03:32Z

⚪ 2025-01-17 18:03:31 UTC Pre-commit check linux-x86_64-relwithdebinfo for aa86cfe has started.
⚪ 2025-01-17 18:03:44 UTC Artifacts will be uploaded here
⚪ 2025-01-17 18:06:35 UTC ya make is running...
🟡 2025-01-17 18:56:09 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
25873	23360	0	1	2389	123

⚪ 2025-01-17 18:58:39 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-17 19:10:17 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
190 (only retried tests)	74	0	0	0	116

🟢 2025-01-17 19:10:25 UTC Build successful.
🟢 2025-01-17 19:10:43 UTC ydbd size 2.1 GiB changed* by +48.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `83627e0`	merge: `aa86cfe`	diff	diff %
ydbd size	2 252 375 912 Bytes	2 252 426 000 Bytes	+48.9 KiB	+0.002%
ydbd stripped size	473 426 864 Bytes	473 438 192 Bytes	+11.1 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-17T18:03:35Z

⚪ 2025-01-17 18:03:35 UTC Pre-commit check linux-x86_64-release-asan for aa86cfe has started.
⚪ 2025-01-17 18:03:47 UTC Artifacts will be uploaded here
⚪ 2025-01-17 18:06:55 UTC ya make is running...
🟡 2025-01-17 19:06:34 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11653	11596	0	20	12	25

⚪ 2025-01-17 19:07:41 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-17 19:19:30 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
88 (only retried tests)	58	0	0	7	23

🟢 2025-01-17 19:19:38 UTC Build successful.
🟡 2025-01-17 19:20:04 UTC ydbd size 3.6 GiB changed* by +113.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `83627e0`	merge: `aa86cfe`	diff	diff %
ydbd size	3 902 985 256 Bytes	3 903 101 856 Bytes	+113.9 KiB	+0.003%
ydbd stripped size	1 362 617 520 Bytes	1 362 655 120 Bytes	+36.7 KiB	+0.003%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

(cherry picked from commit 38d0188)

yumkam added 6 commits January 16, 2025 11:47

streamlookup: verify RightInput is TDqLookupSourceWrap

05381ff

(should be replaced in logical optimize)

streamlookup: drop ExtractMembers on right side

d932ed0

streamlookup: support n-way EquiJoin (n > 2)

5e6b728

stream lookup: handle (Void) label

243f6f2

verify check bounds

3a20d1c

cosmetics

4295e7c

github-actions bot added the bugfix label Jan 16, 2025

yumkam added 3 commits January 17, 2025 11:56

add tests

1d46e6f

tweak tests

aa51210

fixup! add tests

6315ad7

yumkam changed the title ~~[WIP] Fix multiple streamlookup~~ Fix multiple streamlookup Jan 17, 2025

github-actions bot added bugfix and removed bugfix labels Jan 17, 2025

yumkam marked this pull request as ready for review January 17, 2025 10:09

yumkam requested review from a team as code owners January 17, 2025 10:09

github-actions bot added bugfix and removed bugfix labels Jan 17, 2025

yumkam requested a review from rvu1024 January 17, 2025 12:09

rvu1024 reviewed Jan 17, 2025

View reviewed changes

yumkam added 2 commits January 17, 2025 18:28

drop duplicated Error

59c19d3

already added by GetDqJoinResultType

DqCnStreamLookup annotation: more type verification

e8f4369

fixup! DqCnStreamLookup annotation: more type verification

b0ce0f8

fixup! DqCnStreamLookup annotation: more type verification

b38ecfb

yumkam added the rebase-and-check Rebase PR with the current base branch and check label Jan 17, 2025

github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Jan 17, 2025

DqCnStreamLookup annotation: all type verification

4280fc4

yumkam requested a review from rvu1024 January 20, 2025 11:54

rvu1024 approved these changes Jan 23, 2025

View reviewed changes

yumkam merged commit 38d0188 into ydb-platform:main Jan 23, 2025
12 checks passed

yumkam added a commit to yumkam/ydb that referenced this pull request Jan 23, 2025

Fix multiple streamlookup (ydb-platform#13426)

c76d282

(cherry picked from commit 38d0188)

yumkam deleted the fix-multiple-streamlookup branch January 23, 2025 15:05

yumkam added a commit that referenced this pull request Jan 24, 2025

Fix multiple streamlookup (backport #13426) (#13765)

1c132e0

This was referenced Jan 27, 2025

YQ-4052 fixed url escaping for s3 insert #13891

Merged

YQ-4043 fixed hanging in kqp write actor #14017

Merged

azevaykin pushed a commit to azevaykin/ydb that referenced this pull request Feb 3, 2025

Fix multiple streamlookup (ydb-platform#13426)

cde63be

GrigoriyPA mentioned this pull request Feb 17, 2025

merge to yq stable YQ remove anonymous namespace from CA #14656

Closed

GrigoriyPA mentioned this pull request Mar 6, 2025

merge to yq stable YQ added YDB_AUTH_TICKET_HEADER for fq proxy ydb requests #15432

Closed

Fix multiple streamlookup #13426

Fix multiple streamlookup #13426

Uh oh!

Conversation

yumkam commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog entry

Changelog category

Additional information

Uh oh!

github-actions bot commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rvu1024 Jan 17, 2025

Choose a reason for hiding this comment

Uh oh!

yumkam Jan 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yumkam commented Jan 16, 2025 •

edited

Loading

github-actions bot commented Jan 16, 2025 •

edited

Loading

github-actions bot commented Jan 16, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading