YQ-3713 Shared reading: SqlIn predicate #11132

kardymonds · 2024-10-30T19:36:17Z

Changelog entry

...

Changelog category

Not for changelog (changelog entry is not required)

Additional information

...

github-actions · 2024-10-30T19:39:12Z

⚪ 2024-10-30 19:39:12 UTC Pre-commit check linux-x86_64-relwithdebinfo for e568449 has started.
⚪ 2024-10-30 19:39:45 UTC Artifacts will be uploaded here
⚪ 2024-10-30 19:43:51 UTC ya make is running...
🔴 2024-10-30 19:47:11 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-10-30T19:40:38Z

⚪ 2024-10-30 19:40:38 UTC Pre-commit check linux-x86_64-release-asan for e568449 has started.
⚪ 2024-10-30 19:40:49 UTC Artifacts will be uploaded here
⚪ 2024-10-30 19:44:07 UTC ya make is running...
🔴 2024-10-30 19:47:04 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-10-30T21:07:38Z

⚪ 2024-10-30 21:07:37 UTC Pre-commit check linux-x86_64-release-asan for 0040ea0 has started.
⚪ 2024-10-30 21:07:49 UTC Artifacts will be uploaded here
⚪ 2024-10-30 21:10:49 UTC ya make is running...
🟡 2024-10-30 22:19:20 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
9246	9157	0	35	22	32

🟢 2024-10-30 22:20:07 UTC Build successful.
🟡 2024-10-30 22:20:40 UTC ydbd size 5.7 GiB changed* by +283.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `4e69825`	merge: `0040ea0`	diff	diff %
ydbd size	6 148 007 408 Bytes	6 148 297 464 Bytes	+283.3 KiB	+0.005%
ydbd stripped size	1 534 217 296 Bytes	1 534 306 544 Bytes	+87.2 KiB	+0.006%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-30T21:07:39Z

⚪ 2024-10-30 21:07:39 UTC Pre-commit check linux-x86_64-relwithdebinfo for 0040ea0 has started.
⚪ 2024-10-30 21:07:50 UTC Artifacts will be uploaded here
⚪ 2024-10-30 21:10:49 UTC ya make is running...
🟡 2024-10-30 22:03:18 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
40050	34405	0	4	5546	95

⚪ 2024-10-30 22:06:39 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-30 22:18:31 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
99 (only retried tests)	3	0	2	1	93

⚪ 2024-10-30 22:18:39 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-10-30 22:29:58 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
95 (only retried tests)	1	0	2	0	92

🟢 2024-10-30 22:30:05 UTC Build successful.
🟡 2024-10-30 22:30:27 UTC ydbd size 2.8 GiB changed* by +185.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `4e69825`	merge: `0040ea0`	diff	diff %
ydbd size	3 038 351 824 Bytes	3 038 541 656 Bytes	+185.4 KiB	+0.006%
ydbd stripped size	481 279 224 Bytes	481 296 632 Bytes	+17.0 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-31T05:54:40Z

⚪ 2024-10-31 05:54:39 UTC Pre-commit check linux-x86_64-relwithdebinfo for d3c0ab6 has started.
⚪ 2024-10-31 05:55:11 UTC Artifacts will be uploaded here
⚫ 2024-10-31 06:12:01 UTC Check cancelled

github-actions · 2024-10-31T05:55:16Z

⚪ 2024-10-31 05:55:15 UTC Pre-commit check linux-x86_64-release-asan for d3c0ab6 has started.
⚪ 2024-10-31 05:55:26 UTC Artifacts will be uploaded here
⚫ 2024-10-31 06:12:01 UTC Check cancelled

github-actions · 2024-10-31T06:14:33Z

⚪ 2024-10-31 06:14:33 UTC Pre-commit check linux-x86_64-relwithdebinfo for b911c36 has started.
⚪ 2024-10-31 06:14:37 UTC Artifacts will be uploaded here
⚪ 2024-10-31 06:17:43 UTC ya make is running...
🟡 2024-10-31 07:09:51 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
40049	34406	0	1	5546	96

⚪ 2024-10-31 07:13:11 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-10-31 07:24:22 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
97 (only retried tests)	4	0	0	0	93

🟢 2024-10-31 07:24:29 UTC Build successful.
🟡 2024-10-31 07:24:48 UTC ydbd size 2.8 GiB changed* by +175.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `4e69825`	merge: `b911c36`	diff	diff %
ydbd size	3 038 351 824 Bytes	3 038 531 144 Bytes	+175.1 KiB	+0.006%
ydbd stripped size	481 279 224 Bytes	481 295 224 Bytes	+15.6 KiB	+0.003%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-31T06:16:33Z

⚪ 2024-10-31 06:16:33 UTC Pre-commit check linux-x86_64-release-asan for b911c36 has started.
⚪ 2024-10-31 06:16:44 UTC Artifacts will be uploaded here
⚪ 2024-10-31 06:19:44 UTC ya make is running...
🟡 2024-10-31 07:27:10 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
9195	9101	0	49	13	32

🟢 2024-10-31 07:27:58 UTC Build successful.
🟡 2024-10-31 07:28:31 UTC ydbd size 5.7 GiB changed* by +262.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `4e69825`	merge: `b911c36`	diff	diff %
ydbd size	6 148 007 408 Bytes	6 148 276 288 Bytes	+262.6 KiB	+0.004%
ydbd stripped size	1 534 217 296 Bytes	1 534 295 728 Bytes	+76.6 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

uzhastik · 2024-10-31T08:50:42Z

ydb/library/yql/providers/pq/provider/yql_pq_dq_integration.cpp

-                    }
+                auto serializedProto = topicSource.FilterPredicate().Ref().Content();
+                if (!predicateProto.ParseFromString(serializedProto)) {
+                    YQL_CLOG(ERROR, ProviderPq) << "ParseFromString failed";


нужно ошибку вернуть

Заменил на YQL_ENSURE()

uzhastik · 2024-10-31T08:51:15Z

ydb/library/yql/providers/pq/provider/yql_pq_logical_opt.cpp

+        TStringBuilder err;
+        NYql::NConnector::NApi::TPredicate predicateProto;
+        if (!NYql::SerializeFilterPredicate(predicate, &predicateProto, err)) {
+            ctx.AddWarning(TIssue(ctx.GetPosition(node.Pos()), "Failed to serialize filter predicate for source: " + err));


почему это не ошибка?

Тут не все предикаты поддерживаются. Выше MakePushdownPredicate что-то проверяет, но возможны всё равно различия. В идеале да, нужно фейлить запрос

uzhastik · 2024-10-31T08:51:37Z

ydb/library/yql/providers/pq/provider/yql_pq_logical_opt.cpp

+
+        TString serializedProto;
+        if (!predicateProto.SerializeToString(&serializedProto)) {
+            YQL_CLOG(ERROR, ProviderPq) << "SerializeToString failed";


надо ошибку вернуть

Заменил на YQL_ENSURE()

github-actions · 2024-11-01T09:01:57Z

⚪ 2024-11-01 09:01:57 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4b739da has started.
⚪ 2024-11-01 09:02:06 UTC Artifacts will be uploaded here
⚪ 2024-11-01 09:05:18 UTC ya make is running...
🟡 2024-11-01 10:02:57 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
40077	34428	0	3	5548	98

⚪ 2024-11-01 10:06:18 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-11-01 10:17:53 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
97 (only retried tests)	5	0	0	0	92

🟢 2024-11-01 10:18:00 UTC Build successful.
🟡 2024-11-01 10:18:22 UTC ydbd size 2.8 GiB changed* by +176.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `497f6e4`	merge: `4b739da`	diff	diff %
ydbd size	3 040 227 376 Bytes	3 040 407 936 Bytes	+176.3 KiB	+0.006%
ydbd stripped size	481 413 976 Bytes	481 433 624 Bytes	+19.2 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-11-01T09:02:41Z

⚪ 2024-11-01 09:02:40 UTC Pre-commit check linux-x86_64-release-asan for 4b739da has started.
⚪ 2024-11-01 09:02:51 UTC Artifacts will be uploaded here
⚪ 2024-11-01 09:05:51 UTC ya make is running...
🟡 2024-11-01 10:14:08 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
9243	9158	0	38	17	30

🟢 2024-11-01 10:14:55 UTC Build successful.
🟡 2024-11-01 10:15:30 UTC ydbd size 5.7 GiB changed* by +261.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `497f6e4`	merge: `4b739da`	diff	diff %
ydbd size	6 151 724 848 Bytes	6 151 992 352 Bytes	+261.2 KiB	+0.004%
ydbd stripped size	1 534 747 728 Bytes	1 534 826 096 Bytes	+76.5 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

…icate

github-actions · 2024-11-02T05:57:04Z

⚪ 2024-11-02 05:57:04 UTC Pre-commit check linux-x86_64-relwithdebinfo for 64810ff has started.
⚪ 2024-11-02 05:57:14 UTC Artifacts will be uploaded here
⚪ 2024-11-02 06:20:01 UTC ya make is running...
🟡 2024-11-02 07:15:33 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
40078	34429	0	2	5548	99

⚪ 2024-11-02 07:18:54 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-11-02 07:30:54 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
100 (only retried tests)	7	0	0	0	93

🟢 2024-11-02 07:31:01 UTC Build successful.
🟡 2024-11-02 07:31:22 UTC ydbd size 2.8 GiB changed* by +176.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `ef036a2`	merge: `64810ff`	diff	diff %
ydbd size	3 039 243 304 Bytes	3 039 423 856 Bytes	+176.3 KiB	+0.006%
ydbd stripped size	481 450 872 Bytes	481 470 520 Bytes	+19.2 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-11-02T05:57:23Z

⚪ 2024-11-02 05:57:22 UTC Pre-commit check linux-x86_64-release-asan for 64810ff has started.
⚪ 2024-11-02 05:57:32 UTC Artifacts will be uploaded here
⚪ 2024-11-02 06:20:11 UTC ya make is running...
🟡 2024-11-02 07:28:55 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
9252	9163	0	37	16	36

🟢 2024-11-02 07:29:43 UTC Build successful.
🟡 2024-11-02 07:30:16 UTC ydbd size 5.7 GiB changed* by +257.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `ef036a2`	merge: `64810ff`	diff	diff %
ydbd size	6 149 929 792 Bytes	6 150 193 280 Bytes	+257.3 KiB	+0.004%
ydbd stripped size	1 534 982 160 Bytes	1 535 056 496 Bytes	+72.6 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-11-05T09:09:43Z

⚪ 2024-11-05 09:09:43 UTC Pre-commit check linux-x86_64-release-asan for 5f819e2 has started.
⚪ 2024-11-05 09:09:53 UTC Artifacts will be uploaded here
⚪ 2024-11-05 09:12:54 UTC ya make is running...
🟡 2024-11-05 10:22:12 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
9252	9164	0	38	17	33

🟢 2024-11-05 10:23:02 UTC Build successful.
🟡 2024-11-05 10:23:32 UTC ydbd size 5.7 GiB changed* by +265.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `7672737`	merge: `5f819e2`	diff	diff %
ydbd size	6 150 333 736 Bytes	6 150 605 416 Bytes	+265.3 KiB	+0.004%
ydbd stripped size	1 535 125 648 Bytes	1 535 208 176 Bytes	+80.6 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-11-05T09:09:55Z

⚪ 2024-11-05 09:09:55 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5f819e2 has started.
⚪ 2024-11-05 09:10:05 UTC Artifacts will be uploaded here
⚪ 2024-11-05 09:13:07 UTC ya make is running...
🟡 2024-11-05 10:07:17 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
40092	34385	0	9	5587	111

⚪ 2024-11-05 10:10:37 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-11-05 10:22:26 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
146 (only retried tests)	49	0	0	0	97

🟢 2024-11-05 10:22:34 UTC Build successful.
🟡 2024-11-05 10:22:56 UTC ydbd size 2.8 GiB changed* by +172.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `7672737`	merge: `5f819e2`	diff	diff %
ydbd size	3 039 362 384 Bytes	3 039 538 840 Bytes	+172.3 KiB	+0.006%
ydbd stripped size	481 619 384 Bytes	481 634 936 Bytes	+15.2 KiB	+0.003%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

kardymonds added 3 commits October 24, 2024 11:51

add is not distinct predicate

c0a372d

add sql in predicate

b4e1a8b

check types

9983c22

github-actions bot added the not-for-changelog label Oct 30, 2024

fix test

80f6bbd

fix style and one test

db40a1b

kardymonds marked this pull request as ready for review October 31, 2024 05:59

kardymonds requested review from a team as code owners October 31, 2024 05:59

style fix

bf737b6

GrigoriyPA previously approved these changes Oct 31, 2024

View reviewed changes

uzhastik reviewed Oct 31, 2024

View reviewed changes

add yql_ensure

5f0833b

kardymonds dismissed GrigoriyPA’s stale review via 5f0833b November 1, 2024 08:59

vitstn previously approved these changes Nov 1, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/main' into YQ-3713-sql-in-pred…

0ca09f1

…icate

kardymonds dismissed vitstn’s stale review via 0ca09f1 November 2, 2024 05:53

kardymonds added the rebase-and-check Rebase PR with the current base branch and check label Nov 5, 2024

github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Nov 5, 2024

GrigoriyPA approved these changes Nov 6, 2024

View reviewed changes

kardymonds merged commit 7cf0383 into ydb-platform:main Nov 6, 2024
14 checks passed

GrigoriyPA mentioned this pull request Nov 6, 2024

YQ RD support strings parsing in Json columns #11331

Merged

kardymonds added a commit to kardymonds/ydb that referenced this pull request Nov 6, 2024

YQ-3713 Shared reading: SqlIn predicate (ydb-platform#11132)

d9bdf2b

niksaveliev mentioned this pull request Nov 8, 2024

Fix read session correct close ut #11407

Merged

This was referenced Nov 10, 2024

YQ RD enable LLVM in purecalc filters #11442

Merged

YQ WM fixed race with actor context #11576

Merged

niksaveliev mentioned this pull request Nov 14, 2024

Commit for autopartitioned topics #11629

Merged

YQ-3713 Shared reading: SqlIn predicate #11132

YQ-3713 Shared reading: SqlIn predicate #11132

Uh oh!

Conversation

kardymonds commented Oct 30, 2024

Changelog entry

Changelog category

Additional information

Uh oh!

github-actions bot commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uzhastik Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

kardymonds Nov 1, 2024

Choose a reason for hiding this comment

Uh oh!

uzhastik Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

kardymonds Nov 1, 2024

Choose a reason for hiding this comment

Uh oh!

uzhastik Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

kardymonds Nov 1, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Oct 30, 2024 •

edited

Loading

github-actions bot commented Oct 30, 2024 •

edited

Loading

github-actions bot commented Oct 30, 2024 •

edited

Loading

github-actions bot commented Oct 30, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Nov 1, 2024 •

edited

Loading

github-actions bot commented Nov 1, 2024 •

edited

Loading

github-actions bot commented Nov 2, 2024 •

edited

Loading

github-actions bot commented Nov 2, 2024 •

edited

Loading

github-actions bot commented Nov 5, 2024 •

edited

Loading

github-actions bot commented Nov 5, 2024 •

edited

Loading