Skip to content

YQL-17087: Enable output channel spilling only if task has two or more outputs #1043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Jan 29, 2024
Merged

YQL-17087: Enable output channel spilling only if task has two or more outputs #1043

merged 9 commits into from
Jan 29, 2024

Conversation

Darych
Copy link
Contributor

@Darych Darych commented Jan 16, 2024

No description provided.

@Darych Darych changed the title Enable output channel spilling only if task has two or more outputs YQL-17087: Enable output channel spilling only if task has two or more outputs Jan 16, 2024
rvu1024
rvu1024 requested changes Jan 16, 2024
View reviewed changes
Copy link
Member

@rvu1024 rvu1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Как будет реагировать сейчас код на ситуацию, когда spilling service выключен, а в запросе ставят прагму dq.SpillingEngine="file"?

@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 16, 2024 •
edited
Loading

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-release-asan: some tests FAILED for commit 2015d4b.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15870 15694 0 63 110 3

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 2015d4b.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
59890 50632 0 47 9206 5

🔴 linux-x86_64-release-asan: some tests FAILED for commit 2015d4b.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15870 15694 0 63 110 3

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 2015d4b.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
59890 50632 0 47 9206 5

@Darych
Copy link
Contributor Author

Darych commented Jan 17, 2024

Как будет реагировать сейчас код на ситуацию, когда spilling service выключен, а в запросе ставят прагму dq.SpillingEngine="file"?

Если в запросе реально начнет применяться спиллинг, то запрос зависнет, потому что данные никогда не запишутся, а следовательно не прочитаются.

@Darych Darych force-pushed the YQL-17087-opt-enable-spilling branch from 2015d4b to a78a7d5 Compare January 18, 2024 09:39
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 18, 2024 •
edited
Loading

⚪ 2024-01-24 07:51:11 UTC Pre-commit check for a107a41 has started.
⚪ 2024-01-24 07:51:14 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-01-24 08:31:01 UTC Build successful.
⚪ 2024-01-24 08:31:15 UTC Tests are running...
🟢 2024-01-24 10:11:46 UTC Tests successful.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
60025 50782 0 0 9227 16

@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 18, 2024 •
edited
Loading

⚪ 2024-01-24 07:53:55 UTC Pre-commit check for a107a41 has started.
⚪ 2024-01-24 07:53:57 UTC Build linux-x86_64-release-asan is running...
🟢 2024-01-24 08:38:15 UTC Build successful.
⚪ 2024-01-24 08:38:28 UTC Tests are running...
🔴 2024-01-24 10:25:59 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15929 15763 0 36 99 31

@Darych
Copy link
Contributor Author

Darych commented Jan 22, 2024

Как будет реагировать сейчас код на ситуацию, когда spilling service выключен, а в запросе ставят прагму dq.SpillingEngine="file"?

Добавил флажок для отслеживания Undelivered. Spilling actor будет следить за undelivered и в случае недоставки ответит EvError.

Darych added 3 commits January 22, 2024 12:41
@Darych
Enable output channel spilling only if task has two or more outputs
e244c53
@Darych
Address comments
40f9120
@Darych
Add handling if spilling service is not started.
a2af99e 
TDqLocalFileSpillingActor sends events with flag to traack undelivery.
Add unit tests for this.
Fix LOG macroses in channel_storage_actor.cpp to handle ActorSystem pointer also.
@Darych Darych force-pushed the YQL-17087-opt-enable-spilling branch from 3d67e3c to a2af99e Compare January 22, 2024 12:42
@Darych Darych requested a review from a team as a code owner January 22, 2024 12:42
@lll-phill-lll
Copy link
Member

А теперь получается, что включение спиллинга не всегда включает спиллинг?

@Darych
Copy link
Contributor Author

Darych commented Jan 24, 2024

А теперь получается, что включение спиллинга не всегда включает спиллинг?

Для YDB пока еще всегда. Для обратной совместимости с прошлой версией.
В DQ уже спиллинг в каналах будет происходить только когда из стейджа выходит больше одного канала.

Darych added 4 commits January 24, 2024 07:31
@Darych
Enable output channel spilling only if task has two or more outputs
1be50d8
@Darych
Address comments
5de39dd
@Darych
Add handling if spilling service is not started.
3a06cf0 
TDqLocalFileSpillingActor sends events with flag to traack undelivery.
Add unit tests for this.
Fix LOG macroses in channel_storage_actor.cpp to handle ActorSystem pointer also.
@Darych
Return backward compatibility to YDB for spilling
a107a41
@Darych Darych force-pushed the YQL-17087-opt-enable-spilling branch from fe4adb7 to a107a41 Compare January 24, 2024 07:48
rvu1024
rvu1024 previously approved these changes Jan 24, 2024
View reviewed changes
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 25, 2024 •
edited
Loading

⚪ 2024-01-25 14:58:09 UTC Pre-commit check for ab81780 has started.
⚪ 2024-01-25 14:58:12 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-01-25 15:38:42 UTC Build successful.
⚪ 2024-01-25 15:38:58 UTC Tests are running...
🔴 2024-01-25 17:14:46 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
60106 50848 0 1 9235 22

@vitstn vitstn mentioned this pull request Feb 7, 2024
Merged
@GrigoriyPA GrigoriyPA mentioned this pull request Feb 7, 2024
Merged
This was referenced Feb 7, 2024
Merged
Merged
Merged
@ildar-khisambeev ildar-khisambeev mentioned this pull request Feb 8, 2024
Merged
@vitstn vitstn mentioned this pull request Feb 8, 2024
Merged
@niksaveliev niksaveliev mentioned this pull request Feb 9, 2024
Merged
@GrigoriyPA GrigoriyPA mentioned this pull request Feb 9, 2024
Merged
@pavelvelikhov pavelvelikhov mentioned this pull request Feb 9, 2024
Merged
@niksaveliev niksaveliev mentioned this pull request Feb 12, 2024
Merged
@starlinskiy starlinskiy mentioned this pull request Feb 12, 2024
Closed
@vitstn vitstn mentioned this pull request Feb 12, 2024
Merged
@GrigoriyPA GrigoriyPA mentioned this pull request Feb 12, 2024
Merged
@vitstn vitstn mentioned this pull request Feb 12, 2024
Merged
@starlinskiy starlinskiy mentioned this pull request Feb 12, 2024
Merged
@pavelvelikhov pavelvelikhov mentioned this pull request Feb 13, 2024
Merged
This was referenced Feb 13, 2024
Merged
Merged
Merged
@pavelvelikhov pavelvelikhov mentioned this pull request Feb 15, 2024
Merged
@niksaveliev niksaveliev mentioned this pull request Feb 16, 2024
Merged
@vitstn vitstn mentioned this pull request Feb 16, 2024
Merged
This was referenced Feb 17, 2024
Merged
Merged
Merged
Merged
Merged
@aakulaga-ydb aakulaga-ydb mentioned this pull request Feb 24, 2024
Merged
@GrigoriyPA GrigoriyPA mentioned this pull request Feb 26, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gridnevvvit gridnevvvit gridnevvvit approved these changes

@rvu1024 rvu1024 rvu1024 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Darych @lll-phill-lll @gridnevvvit @rvu1024