Skip to content

YDB-2568 Enable match_recognize in ydb #6807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Aug 6, 2024

Conversation

kardymonds
Copy link
Collaborator

Changelog entry

...

Changelog category

  • Not for changelog (changelog entry is not required)

Additional information

...

@kardymonds kardymonds added the rebase-and-check Rebase PR with the current base branch and check label Jul 18, 2024
@github-actions github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Jul 18, 2024
Copy link

github-actions bot commented Jul 18, 2024

2024-07-18 06:48:16 UTC Pre-commit check for a820608 has started.
2024-07-18 06:51:13 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-18 07:21:20 UTC Build successful.

Copy link

github-actions bot commented Jul 18, 2024

2024-07-18 06:48:22 UTC Pre-commit check for a820608 has started.
2024-07-18 06:51:22 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-18 09:20:25 UTC Build successful.
🔴 2024-07-18 09:23:41 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11024 10437 0 86 328 173

🟢 2024-07-18 09:24:35 UTC ydbd size 5.2 GiB changed* by +4.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 1e12440 merge: a820608 diff diff %
ydbd size 5 546 740 432 Bytes 5 546 745 128 Bytes +4.6 KiB +0.000%
ydbd stripped size 1 195 287 920 Bytes 1 195 287 984 Bytes +64 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 18, 2024

2024-07-18 06:48:24 UTC Pre-commit check for a820608 has started.
2024-07-18 06:51:18 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-18 09:01:31 UTC Build successful.
🔴 2024-07-18 09:06:17 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39544 34303 0 6 5221 14

🟢 2024-07-18 09:07:04 UTC ydbd size 8.0 GiB changed* by +6.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 1e12440 merge: a820608 diff diff %
ydbd size 8 642 121 848 Bytes 8 642 128 056 Bytes +6.1 KiB +0.000%
ydbd stripped size 470 514 976 Bytes 470 515 104 Bytes +128 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 19, 2024

2024-07-19 05:37:52 UTC Pre-commit check for eeb4245 has started.
2024-07-19 05:40:48 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-19 08:05:28 UTC Build successful.
🔴 2024-07-19 08:07:36 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11044 10463 0 89 318 174

🟢 2024-07-19 08:08:22 UTC ydbd size 5.2 GiB changed* by +19.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 19153a4 merge: eeb4245 diff diff %
ydbd size 5 546 451 224 Bytes 5 546 471 608 Bytes +19.9 KiB +0.000%
ydbd stripped size 1 195 225 712 Bytes 1 195 229 744 Bytes +3.9 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 19, 2024

2024-07-19 05:38:16 UTC Pre-commit check for eeb4245 has started.
2024-07-19 05:41:24 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-19 06:18:18 UTC Build successful.

Copy link

github-actions bot commented Jul 19, 2024

2024-07-19 05:38:37 UTC Pre-commit check for eeb4245 has started.
2024-07-19 05:41:42 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-19 08:01:25 UTC Build successful.
🔴 2024-07-19 08:06:07 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39570 34326 0 7 5222 15

🟢 2024-07-19 08:06:58 UTC ydbd size 8.0 GiB changed* by +14.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 19153a4 merge: eeb4245 diff diff %
ydbd size 8 641 993 624 Bytes 8 642 008 296 Bytes +14.3 KiB +0.000%
ydbd stripped size 470 492 384 Bytes 470 493 344 Bytes +960 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 19, 2024

2024-07-19 14:01:58 UTC Pre-commit check for 4d6fdcf has started.
2024-07-19 14:04:32 UTC Check cancelled

Copy link

github-actions bot commented Jul 19, 2024

2024-07-19 14:01:58 UTC Pre-commit check for 4d6fdcf has started.
2024-07-19 14:04:32 UTC Check cancelled

Copy link

github-actions bot commented Jul 19, 2024

2024-07-19 14:02:01 UTC Pre-commit check for 4d6fdcf has started.
2024-07-19 14:04:33 UTC Check cancelled

@kardymonds kardymonds marked this pull request as ready for review July 19, 2024 14:02
@kardymonds kardymonds requested a review from a team as a code owner July 19, 2024 14:02
Copy link

github-actions bot commented Jul 19, 2024

2024-07-19 14:05:58 UTC Pre-commit check for 230face has started.
2024-07-19 14:09:01 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-19 16:43:53 UTC Build successful.
🔴 2024-07-19 16:47:18 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11061 10462 0 96 331 172

🔴 2024-07-19 16:48:18 UTC ydbd size 5.2 GiB changed* by +7.5 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 19153a4 merge: 230face diff diff %
ydbd size 5 546 451 224 Bytes 5 554 344 944 Bytes +7.5 MiB +0.142%
ydbd stripped size 1 195 225 712 Bytes 1 196 739 696 Bytes +1.4 MiB +0.127%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 19, 2024

2024-07-19 14:08:46 UTC Pre-commit check for 230face has started.
2024-07-19 14:11:43 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-19 14:44:52 UTC Build successful.

Copy link

github-actions bot commented Jul 19, 2024

2024-07-19 14:08:56 UTC Pre-commit check for 230face has started.
2024-07-19 14:11:56 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-19 16:25:08 UTC Build successful.
🔴 2024-07-19 16:29:33 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39594 34354 0 4 5225 11

🔴 2024-07-19 16:30:16 UTC ydbd size 8.1 GiB changed* by +11.8 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 19153a4 merge: 230face diff diff %
ydbd size 8 641 993 624 Bytes 8 654 418 600 Bytes +11.8 MiB +0.144%
ydbd stripped size 470 492 384 Bytes 471 054 528 Bytes +549.0 KiB +0.119%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 30, 2024

2024-07-30 14:27:23 UTC Pre-commit check for 6831cc4 has started.
2024-07-30 14:30:27 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-30 15:02:22 UTC Build successful.

Copy link

github-actions bot commented Jul 30, 2024

2024-07-30 14:27:50 UTC Pre-commit check for 6831cc4 has started.
2024-07-30 14:30:58 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-30 17:07:41 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13345 12750 0 90 333 172

🟢 2024-07-30 17:09:00 UTC Build successful.
🟢 2024-07-30 17:09:32 UTC ydbd size 5.2 GiB changed* by +14.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: ee419db merge: 6831cc4 diff diff %
ydbd size 5 571 118 576 Bytes 5 571 133 136 Bytes +14.2 KiB +0.000%
ydbd stripped size 1 199 220 976 Bytes 1 199 226 032 Bytes +4.9 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 30, 2024

2024-07-30 14:28:14 UTC Pre-commit check for 6831cc4 has started.
2024-07-30 14:31:24 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-07-30 17:03:03 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
74775 61180 0 17 13561 17

2024-07-30 17:11:23 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-07-30 17:19:37 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27 (only retried tests) 3 0 16 0 8

🟢 2024-07-30 17:20:10 UTC ydbd size 8.1 GiB changed* by +6.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: ee419db merge: 6831cc4 diff diff %
ydbd size 8 678 462 728 Bytes 8 678 469 056 Bytes +6.2 KiB +0.000%
ydbd stripped size 471 881 056 Bytes 471 880 800 Bytes -256 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 31, 2024

2024-07-31 06:38:12 UTC Pre-commit check for 5bb8a9e has started.
2024-07-31 06:43:16 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-31 07:18:12 UTC Build successful.

Copy link

github-actions bot commented Jul 31, 2024

2024-07-31 06:38:52 UTC Pre-commit check for 5bb8a9e has started.
2024-07-31 06:43:17 UTC Check linux-x86_64-release-asan is running...
2024-07-31 08:23:08 UTC Check cancelled

Copy link

github-actions bot commented Jul 31, 2024

2024-07-31 06:38:58 UTC Pre-commit check for 5bb8a9e has started.
2024-07-31 06:43:16 UTC Check linux-x86_64-relwithdebinfo is running...
2024-07-31 08:23:08 UTC Check cancelled

@@ -123,7 +123,8 @@ TIntrusivePtr<IKqpHost> CreateKqpHost(TIntrusivePtr<IKqpGateway> gateway,
const TMaybe<TString>& applicationName = Nothing(), const NKikimr::NMiniKQL::IFunctionRegistry* funcRegistry = nullptr,
bool keepConfigChanges = false, bool isInternalCall = false, TKqpTempTablesState::TConstPtr tempTablesState = nullptr,
NActors::TActorSystem* actorSystem = nullptr /*take from TLS by default*/,
NYql::TExprContext* ctx = nullptr);
NYql::TExprContext* ctx = nullptr,
const NKikimrConfig::TQueryServiceConfig& queryServiceConfig = NKikimrConfig::TQueryServiceConfig());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Давай дефолт для конфига уберем. Иначе можно забыть его дотащить из конфига

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Сделал неопциональным

Copy link

github-actions bot commented Jul 31, 2024

2024-07-31 20:45:25 UTC Pre-commit check for e8ac0d8 has started.
2024-07-31 20:48:45 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-31 21:27:04 UTC Build successful.

Copy link

github-actions bot commented Jul 31, 2024

2024-07-31 20:45:29 UTC Pre-commit check for e8ac0d8 has started.
2024-07-31 20:48:45 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-07-31 20:59:50 UTC Build failed. see the logs.

@@ -1033,7 +1033,7 @@ class TKqpHost : public IKqpHost {
std::optional<TKqpFederatedQuerySetup> federatedQuerySetup, const TIntrusiveConstPtr<NACLib::TUserToken>& userToken,
const NKikimr::NMiniKQL::IFunctionRegistry* funcRegistry, bool keepConfigChanges, bool isInternalCall,
TKqpTempTablesState::TConstPtr tempTablesState = nullptr, NActors::TActorSystem* actorSystem = nullptr,
NYql::TExprContext* ctx = nullptr)
NYql::TExprContext* ctx = nullptr, const NKikimrConfig::TQueryServiceConfig& queryServiceConfig = NKikimrConfig::TQueryServiceConfig())
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Давай уберем дефолт для query service config

@@ -145,8 +145,8 @@ TExprNode::TPtr ExpandMatchRecognize(const TExprNode::TPtr& node, TExprContext&
TExprNode::TPtr sortOrder;
ExtractSortKeyAndOrder(pos, sortTraits, sortKey, sortOrder, ctx);
TExprNode::TPtr result;
YQL_ENSURE(sortOrder->ChildrenSize() == 1, "Expect ORDER BY timestamp for MATCH_RECOGNIZE");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Зачем так? Потенциально могут быть сценарии использования без сортировки. И сортировка может быть не обязательно по столбцу timestamp. Как минимум текст ошибки нужно переписать

Copy link
Collaborator

@uzhastik uzhastik Aug 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

давайте текст ошибки перепишем и даже уберем YQL_ENSURE в пользу if. сортировку сделали обязательной для дуализма аналитического и стримингового MR - код должен без проблем переноситься с аналитики на поток

Copy link
Collaborator

@zverevgeny zverevgeny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

До этого изменения сортировка не была обязательной. Это валидный сценарий. Он работал, и на него написаны тесты. Непонятно, почему решили убрать поддержку MATCH_RECOGNIZE без ORDER BY.

@uzhastik
Copy link
Collaborator

uzhastik commented Aug 1, 2024

До этого изменения сортировка не была обязательной. Это валидный сценарий. Он работал, и на него написаны тесты. Непонятно, почему решили убрать поддержку MATCH_RECOGNIZE без ORDER BY.

сортировка была обязательной для стриминг режима. теперь она обязательна для всех режимов, что позволяет клиенту легко перекладывать M_R запрос с аналитики на поток. можно поддержать M_R без требования сортировки, но сразу для обоих режимов. но т.к. я ценности в такой обработке пока не вижу, то проще было отрезать ненужный функционал. чем писать его поддержку еще и для стриминга

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 07:12:52 UTC Pre-commit check for 52c2d5a has started.
2024-08-05 07:16:26 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-05 09:38:59 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13337 13280 0 9 31 17

🟢 2024-08-05 09:40:30 UTC Build successful.
🟢 2024-08-05 09:40:59 UTC ydbd size 5.4 GiB changed* by +6.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: f64d670 merge: 52c2d5a diff diff %
ydbd size 5 825 703 504 Bytes 5 825 709 840 Bytes +6.2 KiB +0.000%
ydbd stripped size 1 463 157 232 Bytes 1 463 154 096 Bytes -3.1 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 07:12:59 UTC Pre-commit check for 52c2d5a has started.
2024-08-05 07:16:07 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-05 07:47:31 UTC Build successful.

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 07:13:12 UTC Pre-commit check for 52c2d5a has started.
2024-08-05 07:16:16 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-05 10:06:28 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
74865 61251 0 10 13593 11

2024-08-05 10:16:34 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-08-05 10:24:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
25 (only retried tests) 9 0 7 1 8

2024-08-05 10:31:02 UTC Failed tests rerun (try 3) linux-x86_64-relwithdebinfo is running...
🔴 2024-08-05 10:38:18 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16 (only retried tests) 1 0 7 0 8

🟢 2024-08-05 10:45:13 UTC Build successful.
🟢 2024-08-05 10:45:52 UTC ydbd size 8.1 GiB changed* by +6.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: f64d670 merge: 52c2d5a diff diff %
ydbd size 8 665 991 720 Bytes 8 665 998 144 Bytes +6.3 KiB +0.000%
ydbd stripped size 471 452 840 Bytes 471 452 584 Bytes -256 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

uzhastik
uzhastik previously approved these changes Aug 5, 2024
Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 11:15:10 UTC Pre-commit check for acdadb1 has started.
2024-08-05 11:19:02 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-05 13:37:53 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13349 13272 0 20 40 17

🟢 2024-08-05 13:39:24 UTC Build successful.
🟢 2024-08-05 13:39:54 UTC ydbd size 5.4 GiB changed* by +10.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: a1f1ecc merge: acdadb1 diff diff %
ydbd size 5 825 869 880 Bytes 5 825 880 312 Bytes +10.2 KiB +0.000%
ydbd stripped size 1 463 178 928 Bytes 1 463 179 888 Bytes +960 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 11:15:44 UTC Pre-commit check for acdadb1 has started.
2024-08-05 11:19:34 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-05 11:27:46 UTC Build successful.

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 11:17:24 UTC Pre-commit check for acdadb1 has started.
2024-08-05 11:20:33 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-05 13:34:36 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
74864 61260 0 2 13592 10

2024-08-05 13:44:37 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-05 13:51:53 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14 (only retried tests) 6 0 0 0 8

🟢 2024-08-05 13:58:49 UTC Build successful.
🟢 2024-08-05 13:59:21 UTC ydbd size 8.1 GiB changed* by +6.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: a1f1ecc merge: acdadb1 diff diff %
ydbd size 8 666 159 312 Bytes 8 666 165 712 Bytes +6.2 KiB +0.000%
ydbd stripped size 471 459 624 Bytes 471 459 368 Bytes -256 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@kardymonds kardymonds enabled auto-merge (squash) August 5, 2024 14:07
@kardymonds kardymonds disabled auto-merge August 5, 2024 14:07
@kardymonds kardymonds enabled auto-merge (squash) August 6, 2024 09:40
@kardymonds kardymonds merged commit 18b7d76 into ydb-platform:main Aug 6, 2024
10 of 12 checks passed
kardymonds added a commit to kardymonds/ydb that referenced this pull request Aug 6, 2024
kardymonds added a commit to kardymonds/ydb that referenced this pull request Aug 6, 2024
kardymonds added a commit to kardymonds/ydb that referenced this pull request Aug 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants