support trailing generic query responses #1441

gridnevvvit · 2024-01-30T20:43:39Z

Changelog entry

improve performance for single row responses in generic query

Changelog category

Improvement

Additional information

...

github-actions · 2024-01-30T20:46:59Z

⚪ 2024-01-30 20:46:59 UTC Pre-commit check for ded3818 has started.
⚪ 2024-01-30 20:47:01 UTC Build linux-x86_64-release-asan is running...
🟢 2024-01-30 21:23:06 UTC Build successful.
⚪ 2024-01-30 21:23:18 UTC Tests are running...
🔴 2024-01-30 23:03:45 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16022	15728	0	52	53	189

github-actions · 2024-01-30T20:47:00Z

⚪ 2024-01-30 20:47:00 UTC Pre-commit check for ded3818 has started.
⚪ 2024-01-30 20:47:03 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-01-30 21:22:45 UTC Build successful.
⚪ 2024-01-30 21:23:01 UTC Tests are running...
🔴 2024-01-30 22:52:49 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
60221	50730	0	44	9263	184

github-actions · 2024-01-31T18:01:47Z

⚪ 2024-01-31 18:01:47 UTC Pre-commit check for 95fc2b4 has started.
⚪ 2024-01-31 18:01:48 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-01-31 18:38:38 UTC Build successful.
⚪ 2024-01-31 18:38:49 UTC Tests are running...
🔴 2024-01-31 20:28:45 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
60412	49815	0	605	9397	595

github-actions · 2024-01-31T18:02:33Z

⚪ 2024-01-31 18:02:32 UTC Pre-commit check for 95fc2b4 has started.
⚪ 2024-01-31 18:02:34 UTC Build linux-x86_64-release-asan is running...
🟢 2024-01-31 18:38:38 UTC Build successful.
⚪ 2024-01-31 18:38:49 UTC Tests are running...
🔴 2024-01-31 20:35:22 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16086	14711	0	621	193	561

github-actions · 2024-01-31T22:30:46Z

⚪ 2024-01-31 22:30:45 UTC Pre-commit check for 0e90a02 has started.
⚪ 2024-01-31 22:30:48 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-01-31 23:09:38 UTC Build successful.
⚪ 2024-01-31 23:09:51 UTC Tests are running...
🔴 2024-01-31 23:15:22 UTC Test run completed, no test results found for commit aeab9d4. Please check build logs.
⚫ 2024-01-31 23:15:26 UTC Check cancelled

github-actions · 2024-01-31T22:30:53Z

⚪ 2024-01-31 22:30:53 UTC Pre-commit check for 0e90a02 has started.
⚪ 2024-01-31 22:30:55 UTC Build linux-x86_64-release-asan is running...
🟢 2024-01-31 23:07:37 UTC Build successful.
⚪ 2024-01-31 23:07:54 UTC Tests are running...
🔴 2024-01-31 23:15:34 UTC Test run completed, no test results found for commit aeab9d4. Please check build logs.
⚫ 2024-01-31 23:15:37 UTC Check cancelled

github-actions · 2024-01-31T23:16:56Z

⚪ 2024-01-31 23:16:56 UTC Pre-commit check for 9d1fa49 has started.
⚪ 2024-01-31 23:16:58 UTC Build linux-x86_64-release-asan is running...
🟢 2024-01-31 23:19:11 UTC Build successful.
⚪ 2024-01-31 23:19:24 UTC Tests are running...
🔴 2024-02-01 00:55:18 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16040	15930	0	16	50	44

github-actions · 2024-01-31T23:19:11Z

⚪ 2024-01-31 23:19:11 UTC Pre-commit check for 9d1fa49 has started.
⚪ 2024-01-31 23:19:13 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-01-31 23:21:37 UTC Build successful.
⚪ 2024-01-31 23:21:51 UTC Tests are running...
🟢 2024-02-01 00:49:23 UTC Tests successful.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
60356	51043	0	0	9270	43

ydb/core/grpc_services/query/rpc_execute_query.cpp

ydb/core/kqp/common/events/query.h

spuchin · 2024-02-01T22:27:02Z

ydb/core/kqp/query_data/kqp_query_data.cpp

    return ydbResult;
 }

+Ydb::ResultSet* TKqpExecuterTxResult::GetTrailingYdb(google::protobuf::Arena* arena, TMaybe<ui64>) {


В чем смысл второго аргумента?

spuchin · 2024-02-01T22:33:06Z

ydb/core/kqp/session_actor/kqp_session_actor.cpp


        // Result for scan query is sent directly to target actor.
        Y_ABORT_UNLESS(response->GetArena());
+        if (QueryState->PreparedQuery && QueryState->IsStreamResult()) {


Эта секция очень похожа на следующую, где !QueryState->IsStreamResult().
Там есть какие-то принципиальные отличия?

ydb/core/kqp/executer_actor/kqp_executer_impl.h

spuchin · 2024-02-01T22:39:38Z

ydb/core/kqp/executer_actor/kqp_executer_impl.h

+                auto ackEv = MakeHolder<NYql::NDq::TEvDqCompute::TEvChannelDataAck>();
+                ackEv->Record.SetSeqNo(computeData.Proto.GetSeqNo());
+                ackEv->Record.SetChannelId(channel.Id);
+                ackEv->Record.SetFreeSpace(50_MB);


Мы же проверили что канал уже FINISHED, какая ему разница на FreeSpace?

да, просто нужно как-то заполнить финальный ак. не вижу тут проблемы какой-то.

Ну константы всегда вопросы вызывают, почему просто 0 не поставить? )

короче, шаг влево - тесты начинают зависать. похоже что есть какие-то проверки на получателе, могу покопать отдельно.

spuchin · 2024-02-01T22:43:38Z

ydb/core/kqp/executer_actor/kqp_executer_impl.h

+        this->Send(channelComputeActorId, ackEv.Release(), /* TODO: undelivery */ 0, /* cookie */ channel.Id);
+    }
+
+    void HandleStreamAck(TEvKqpExecuter::TEvStreamDataAck::TPtr& ev) {


Вот тут кстати не до конца понятно.

Если используется механизм trailing result, то аков нам сюда не прилетит.

Если не используется, то почему бы rpc актору не отправлять Ack'и напрямую в CA?

вообще идея норм, я поправлю, но пока я тут оставлю этот код на всякий.

короче, шаг влево - тесты начинают зависать. похоже что есть какие-то проверки на получателе, могу покопать отдельно.

ydb/core/grpc_services/query/rpc_execute_query.cpp

github-actions · 2024-02-02T16:16:26Z

⚪ 2024-02-02 16:16:26 UTC Pre-commit check for 2baeee6 has started.
⚪ 2024-02-02 16:16:28 UTC Build linux-x86_64-relwithdebinfo is running...
⚫ 2024-02-02 16:21:02 UTC Check cancelled

github-actions · 2024-02-02T16:18:43Z

⚪ 2024-02-02 16:18:43 UTC Pre-commit check for 2baeee6 has started.
⚪ 2024-02-02 16:18:45 UTC Build linux-x86_64-release-asan is running...
⚫ 2024-02-02 16:21:03 UTC Check cancelled

github-actions · 2024-02-02T16:24:39Z

⚪ 2024-02-02 16:24:38 UTC Pre-commit check for 7208485 has started.
⚪ 2024-02-02 16:24:41 UTC Build linux-x86_64-relwithdebinfo is running...
⚫ 2024-02-02 16:51:31 UTC Check cancelled

github-actions · 2024-02-02T16:26:59Z

⚪ 2024-02-02 16:26:59 UTC Pre-commit check for 7208485 has started.
⚪ 2024-02-02 16:27:01 UTC Build linux-x86_64-release-asan is running...
⚫ 2024-02-02 16:51:32 UTC Check cancelled

github-actions · 2024-02-02T16:52:36Z

⚪ 2024-02-03 10:39:14 UTC Pre-commit check for 12af35a has started.
⚪ 2024-02-03 10:39:17 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-02-03 10:40:31 UTC Build successful.
⚪ 2024-02-03 10:40:49 UTC Tests are running...
🔴 2024-02-03 12:01:45 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
58860	49567	0	3	9263	27

github-actions · 2024-02-02T16:55:07Z

⚪ 2024-02-03 14:44:11 UTC Pre-commit check for 50a622b has started.
⚪ 2024-02-03 14:44:14 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-02-03 14:45:26 UTC Build successful.
⚪ 2024-02-03 14:45:42 UTC Tests are running...
🔴 2024-02-03 14:53:13 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
58760	49458	0	6	9268	28

github-actions · 2024-02-02T16:55:12Z

⚪ 2024-02-02 16:55:11 UTC Pre-commit check for 50a622b has started.
⚪ 2024-02-02 16:55:14 UTC Build linux-x86_64-release-asan is running...
🟢 2024-02-02 17:10:14 UTC Build successful.
⚪ 2024-02-02 17:10:27 UTC Tests are running...
🔴 2024-02-02 18:46:22 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14533	14399	0	41	61	32

github-actions · 2024-02-03T10:39:25Z

⚪ 2024-02-03 10:39:25 UTC Pre-commit check for 12af35a has started.
⚪ 2024-02-03 10:39:27 UTC Build linux-x86_64-release-asan is running...
🟢 2024-02-03 10:40:37 UTC Build successful.
⚪ 2024-02-03 10:40:50 UTC Tests are running...
🔴 2024-02-03 12:16:48 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14530	14395	0	31	53	51

github-actions bot added the improvement label Jan 30, 2024

gridnevvvit force-pushed the trailing-generic-query-responses branch from bb1736b to 148e7bc Compare January 31, 2024 16:55

gridnevvvit force-pushed the trailing-generic-query-responses branch from 148e7bc to aeab9d4 Compare January 31, 2024 22:27

gridnevvvit force-pushed the trailing-generic-query-responses branch from aeab9d4 to ab89e4b Compare January 31, 2024 23:14

spuchin reviewed Feb 1, 2024

View reviewed changes

gridnevvvit force-pushed the trailing-generic-query-responses branch from ab89e4b to fb174eb Compare February 2, 2024 16:13

gridnevvvit force-pushed the trailing-generic-query-responses branch from fb174eb to e13b4f1 Compare February 2, 2024 16:20

gridnevvvit force-pushed the trailing-generic-query-responses branch from e13b4f1 to d12fb13 Compare February 2, 2024 16:51

support trailing generic query responses

66d80ab

gridnevvvit force-pushed the trailing-generic-query-responses branch from d12fb13 to 66d80ab Compare February 2, 2024 16:53

spuchin approved these changes Feb 2, 2024

View reviewed changes

gridnevvvit merged commit 5594dac into ydb-platform:main Feb 3, 2024

gridnevvvit deleted the trailing-generic-query-responses branch February 3, 2024 16:34

FloatingCrowbar mentioned this pull request Feb 4, 2024

Сhange keys format for service partitions #1569

Merged

vitstn mentioned this pull request Feb 7, 2024

YQL-17725 drop b and t suffixes #1660

Merged

GrigoriyPA mentioned this pull request Feb 7, 2024

KqpRun add results into clear execution #1661

Merged

This was referenced Feb 7, 2024

YQL-17725 tune doc #1667

Merged

Switch pg_parser BC tests on again #1709

Merged

unmute test #1711

Merged

ildar-khisambeev mentioned this pull request Feb 8, 2024

LOGBROKER-8783 bugfix #1713

Merged

vitstn mentioned this pull request Feb 8, 2024

Mark mutable_id for YT writes #1744

Merged

niksaveliev mentioned this pull request Feb 9, 2024

Fix pq writer and few renames #1757

Merged

GrigoriyPA mentioned this pull request Feb 9, 2024

Moved optimizers from yql/providers/dq to yql/dq #1758

Merged

pavelvelikhov mentioned this pull request Feb 9, 2024

Cannonized two plans #1773

Merged

niksaveliev mentioned this pull request Feb 12, 2024

Workload read without consumer and metrics fixes #1792

Merged

starlinskiy mentioned this pull request Feb 12, 2024

runtime dispatching #1800

Closed

vitstn mentioned this pull request Feb 12, 2024

quick fix for column name of select version() #1801

Merged

GrigoriyPA mentioned this pull request Feb 12, 2024

YQ 2786 fix test fq restarts #1810

Merged

vitstn mentioned this pull request Feb 12, 2024

Support of database sys cache #1817

Merged

starlinskiy mentioned this pull request Feb 12, 2024

runtime dispatching (fixed) #1847

Merged

pavelvelikhov mentioned this pull request Feb 13, 2024

Fixed a problem with simplified plan JSONs #1878

Merged

This was referenced Feb 13, 2024

Case insensivity for pg_catalog tables/cluster, columns. Support of pg_class:relam, and some columns in pg_database & pg_namespace #1893

Merged

fixed pg_get_userbyid function #1936

Merged

support of AnyNonArray type, fixes for array_agg #1958

Merged

pavelvelikhov mentioned this pull request Feb 15, 2024

Fixed a problem with plans in PG ut #1974

Merged

niksaveliev mentioned this pull request Feb 16, 2024

Fix kafka with enabled proxy #2017

Merged

vitstn mentioned this pull request Feb 16, 2024

mock of savepoints #2026

Merged

aakulaga-ydb mentioned this pull request Feb 24, 2024

Fix TzTimestamp wrong encoding (YQL-17920) #2222

Merged

GrigoriyPA mentioned this pull request Feb 26, 2024

YQ-2898 fix query failed with unknown uncommitted upload issue #2240

Merged

support trailing generic query responses #1441

support trailing generic query responses #1441

Uh oh!

Conversation

gridnevvvit commented Jan 30, 2024

Changelog entry

Changelog category

Additional information

Uh oh!

github-actions bot commented Jan 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Jan 30, 2024 •

edited

Loading

github-actions bot commented Jan 30, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 3, 2024 •

edited

Loading