Skip to content

Scheme shard support for cdc stream per index #4650

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
May 22, 2024
Merged

Scheme shard support for cdc stream per index #4650

merged 5 commits into from
May 22, 2024

Conversation

@dcherednik
Copy link
Member

@dcherednik dcherednik commented May 17, 2024

Changelog category

  • Improvement

Additional information

...

@github-actions
Copy link

github-actions bot commented May 17, 2024 •
edited
Loading

⚪ 2024-05-17 13:01:33 UTC Pre-commit check for a8afd05 has started.
⚪ 2024-05-17 13:01:35 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-17 13:40:36 UTC Build successful.
⚪ 2024-05-17 13:42:24 UTC Tests are running...
🔴 2024-05-17 15:19:33 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38057 33224 0 17 4807 9

@github-actions
Copy link

github-actions bot commented May 17, 2024 •
edited
Loading

⚪ 2024-05-17 13:01:37 UTC Pre-commit check for a8afd05 has started.
⚪ 2024-05-17 13:01:39 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-17 13:42:05 UTC Build successful.
⚪ 2024-05-17 13:43:54 UTC Tests are running...
🔴 2024-05-17 15:41:22 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10673 10498 0 39 114 22

@github-actions
Copy link

github-actions bot commented May 17, 2024 •
edited
Loading

⚪ 2024-05-17 13:01:41 UTC Pre-commit check for a8afd05 has started.
⚪ 2024-05-17 13:01:44 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-17 13:39:12 UTC Build successful.

@dcherednik dcherednik added the rebase-and-check Rebase PR with the current base branch and check label May 17, 2024
@github-actions github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label May 17, 2024
@github-actions
Copy link

github-actions bot commented May 17, 2024 •
edited
Loading

⚪ 2024-05-17 16:49:53 UTC Pre-commit check for 5edbfcd has started.
⚪ 2024-05-17 16:49:56 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-17 16:51:21 UTC Build successful.
⚪ 2024-05-17 16:53:06 UTC Tests are running...
🔴 2024-05-17 18:27:32 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38058 33236 0 16 4794 12

@github-actions
Copy link

github-actions bot commented May 17, 2024 •
edited
Loading

⚪ 2024-05-17 16:49:54 UTC Pre-commit check for 5edbfcd has started.
⚪ 2024-05-17 16:49:57 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-17 16:51:21 UTC Build successful.

@github-actions
Copy link

github-actions bot commented May 17, 2024 •
edited
Loading

⚪ 2024-05-17 17:01:31 UTC Pre-commit check for 5edbfcd has started.
⚪ 2024-05-17 17:01:33 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-17 17:03:03 UTC Build successful.
⚪ 2024-05-17 17:04:51 UTC Tests are running...
🔴 2024-05-17 19:17:54 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10671 10551 0 35 71 14

CyberROFL
CyberROFL reviewed May 17, 2024
View reviewed changes
ydb/core/protos/flat_scheme_op.proto Outdated
optional uint64 RetentionPeriodSeconds = 3 [default = 86400]; // 1d by default
optional uint32 TopicPartitions = 4;
oneof IndexMode {
bool AllIndexes = 5; // Create topic per each index
Copy link
Member

@CyberROFL CyberROFL May 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Есть смысл передавать AllIndexes = false? Чем это отличается от IndexMode == NOT_SET?

Copy link
Member Author

@dcherednik dcherednik May 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ничем, но просто поле enum нельзя положить в oneof, или как ты предлагаешь?

Copy link
Member

@CyberROFL CyberROFL May 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Предлагаю google.protobuf.empty AllIndexes, если достаточно (и всегда будет) самого факта. Или хотя бы TAllIndexes AllIndexes, где TAllIndexes — это пустое сообщение (если его предполагается когда-нибудь расширять).

ydb/core/protos/flat_scheme_op.proto
optional uint32 TopicPartitions = 4;
oneof IndexMode {
bool AllIndexes = 5; // Create topic per each index
string IndexName = 6;
Copy link
Member

@CyberROFL CyberROFL May 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Не очень понял, для чего нужен этот режим?

Copy link
Member Author

@dcherednik dcherednik May 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ну мы можем или создать стримы для всех индексов, или только для определенных. Кроме того нам надо передать TCreateCdcStream в сабоперацию и удобно иметь тут же информацию об индексе для которой создана сабоперация. Есть идеи как это выразить лучше?

Copy link
Member

@CyberROFL CyberROFL May 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Какой сценарий создания стрима только для определенного (даже не списка) индекса?

Copy link
Member Author

@dcherednik dcherednik May 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ну к примеру у тебя часть индексов используется для местной аналитики и не нужна на реплике, а часть для OLTP запросов. Или же наоборот. Список наверно лучше тут.

@dcherednik dcherednik force-pushed the cdc_stream_table_with_index_2_with_cancelable_pq_3 branch from 7e6b712 to 924a023 Compare May 21, 2024 11:28
@dcherednik dcherednik force-pushed the cdc_stream_table_with_index_2_with_cancelable_pq_3 branch from 924a023 to 4af1121 Compare May 21, 2024 11:51
dcherednik
dcherednik commented May 21, 2024
View reviewed changes
Copy link
Member Author

@dcherednik dcherednik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Обновил pr и оставил комментарии.

@github-actions
Copy link

github-actions bot commented May 21, 2024 •
edited
Loading

⚪ 2024-05-21 12:17:34 UTC Pre-commit check for 535cd94 has started.
⚪ 2024-05-21 12:17:35 UTC Build linux-x86_64-relwithdebinfo is running...
⚫ 2024-05-21 12:54:06 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented May 21, 2024 •
edited
Loading

⚪ 2024-05-21 12:18:16 UTC Pre-commit check for 535cd94 has started.
⚪ 2024-05-21 12:18:17 UTC Build linux-x86_64-release-clang14 is running...
⚫ 2024-05-21 12:54:07 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented May 21, 2024 •
edited
Loading

⚪ 2024-05-21 12:18:34 UTC Pre-commit check for 535cd94 has started.
⚪ 2024-05-21 12:18:37 UTC Build linux-x86_64-release-asan is running...
⚫ 2024-05-21 12:54:22 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented May 21, 2024 •
edited
Loading

⚪ 2024-05-21 12:55:04 UTC Pre-commit check for d236d96 has started.
⚪ 2024-05-21 12:55:05 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-21 13:04:02 UTC Build successful.
⚪ 2024-05-21 13:05:43 UTC Tests are running...
🔴 2024-05-21 15:06:06 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10674 10595 0 26 37 16

@github-actions
Copy link

github-actions bot commented May 21, 2024 •
edited
Loading

⚪ 2024-05-21 12:57:00 UTC Pre-commit check for d236d96 has started.
⚪ 2024-05-21 12:57:02 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-21 13:04:21 UTC Build successful.

@github-actions
Copy link

github-actions bot commented May 21, 2024 •
edited
Loading

⚪ 2024-05-21 13:03:54 UTC Pre-commit check for d236d96 has started.
⚪ 2024-05-21 13:03:57 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-21 13:11:42 UTC Build successful.
⚪ 2024-05-21 13:13:31 UTC Tests are running...
🔴 2024-05-21 14:46:56 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38123 33331 0 1 4785 6

@github-actions
Copy link

github-actions bot commented May 22, 2024 •
edited
Loading

⚪ 2024-05-22 13:57:01 UTC Pre-commit check for ab225b5 has started.
⚪ 2024-05-22 13:57:02 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-22 14:37:57 UTC Build successful.
⚪ 2024-05-22 14:39:30 UTC Tests are running...
🔴 2024-05-22 16:32:38 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10668 10613 0 20 24 11

@github-actions
Copy link

github-actions bot commented May 22, 2024 •
edited
Loading

⚪ 2024-05-22 13:58:29 UTC Pre-commit check for ab225b5 has started.
⚪ 2024-05-22 13:58:32 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-22 14:33:20 UTC Build successful.

@github-actions
Copy link

github-actions bot commented May 22, 2024 •
edited
Loading

⚪ 2024-05-22 13:59:29 UTC Pre-commit check for ab225b5 has started.
⚪ 2024-05-22 13:59:30 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-22 14:41:45 UTC Build successful.
⚪ 2024-05-22 14:43:22 UTC Tests are running...
🔴 2024-05-22 16:24:37 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38132 33335 0 1 4785 11

@dcherednik dcherednik merged commit 636451e into ydb-platform:main May 22, 2024
@serbel324 serbel324 mentioned this pull request May 24, 2024
Merged
This was referenced May 27, 2024
Closed
Closed
Merged
MrLolthe1st pushed a commit to MrLolthe1st/ydb that referenced this pull request May 28, 2024
@dcherednik @MrLolthe1st
Scheme shard support for cdc stream per index (ydb-platform#4650)
b8ac6ee
@niksaveliev niksaveliev mentioned this pull request May 29, 2024
Merged
This was referenced May 29, 2024
Merged
Merged
This was referenced May 30, 2024
Merged
Closed
Closed
Merged
This was referenced Jun 7, 2024
Merged
Merged
@niksaveliev niksaveliev mentioned this pull request Jun 17, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CyberROFL CyberROFL CyberROFL approved these changes

Assignees

No one assigned

Labels

improvement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dcherednik @CyberROFL