Add `ListNodes` cache in `DynamicNameserver` #12694

pixcc · 2024-12-17T23:39:35Z

Changelog entry

Add caching for ListNodes request in DynamicNameserver

Changelog category

Performance improvement

Additional information

Prior to this change, the entire node list was copied for every ListNodes request. This change adds caching of the node list, which is now distributed by pointer. The node list is re-cached when changes are made.

github-actions · 2024-12-17T23:43:24Z

⚪ 2024-12-17 23:43:23 UTC Pre-commit check linux-x86_64-relwithdebinfo for 83459a8 has started.
⚪ 2024-12-17 23:43:35 UTC Artifacts will be uploaded here
⚪ 2024-12-17 23:46:51 UTC ya make is running...
🔴 2024-12-17 23:50:02 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-12-17T23:43:41Z

⚪ 2024-12-17 23:43:40 UTC Pre-commit check linux-x86_64-release-asan for 83459a8 has started.
⚪ 2024-12-17 23:43:51 UTC Artifacts will be uploaded here
⚪ 2024-12-17 23:46:59 UTC ya make is running...
⚫ 2024-12-18 00:02:30 UTC Check cancelled

github-actions · 2024-12-18T00:09:00Z

⚪ 2024-12-18 00:08:59 UTC Pre-commit check linux-x86_64-release-asan for 17674cc has started.
⚪ 2024-12-18 00:09:12 UTC Artifacts will be uploaded here
⚪ 2024-12-18 00:12:34 UTC ya make is running...
🔴 2024-12-18 00:22:47 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-12-18T00:18:50Z

⚪ 2024-12-18 00:18:49 UTC Pre-commit check linux-x86_64-relwithdebinfo for 17674cc has started.
⚪ 2024-12-18 00:19:00 UTC Artifacts will be uploaded here
⚪ 2024-12-18 00:22:07 UTC ya make is running...
🔴 2024-12-18 00:38:15 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-12-18T01:28:19Z

⚪ 2024-12-18 01:28:19 UTC Pre-commit check linux-x86_64-release-asan for bb38f2b has started.
⚪ 2024-12-18 01:28:30 UTC Artifacts will be uploaded here
⚪ 2024-12-18 01:31:44 UTC ya make is running...
🔴 2024-12-18 01:38:07 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-12-18T01:28:23Z

⚪ 2024-12-18 01:28:23 UTC Pre-commit check linux-x86_64-relwithdebinfo for bb38f2b has started.
⚪ 2024-12-18 01:28:34 UTC Artifacts will be uploaded here
⚪ 2024-12-18 01:31:42 UTC ya make is running...
🔴 2024-12-18 01:38:19 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-12-18T01:52:54Z

⚪ 2024-12-18 01:52:53 UTC Pre-commit check linux-x86_64-relwithdebinfo for 672eb08 has started.
⚪ 2024-12-18 01:52:58 UTC Artifacts will be uploaded here
⚪ 2024-12-18 01:56:16 UTC ya make is running...
🔴 2024-12-18 02:03:40 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-12-18T01:56:07Z

⚪ 2024-12-18 01:56:07 UTC Pre-commit check linux-x86_64-release-asan for 672eb08 has started.
⚪ 2024-12-18 01:56:18 UTC Artifacts will be uploaded here
⚪ 2024-12-18 01:59:27 UTC ya make is running...
🔴 2024-12-18 02:09:57 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-12-18T02:26:42Z

⚪ 2024-12-18 02:26:42 UTC Pre-commit check linux-x86_64-relwithdebinfo for 00e2ef6 has started.
⚪ 2024-12-18 02:26:54 UTC Artifacts will be uploaded here
⚪ 2024-12-18 02:30:15 UTC ya make is running...
🔴 2024-12-18 02:37:27 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-12-18T02:27:12Z

⚪ 2024-12-18 02:27:11 UTC Pre-commit check linux-x86_64-release-asan for 00e2ef6 has started.
⚪ 2024-12-18 02:27:23 UTC Artifacts will be uploaded here
⚪ 2024-12-18 02:30:35 UTC ya make is running...
🔴 2024-12-18 02:38:03 UTC Build failed, see the logs. Also see fail summary

ydb/library/actors/core/interconnect.h

ydb/core/fq/libs/actors/nodes_manager.cpp

ydb/core/mind/bscontroller/impl.h

ydb/core/mind/dynamic_nameserver.cpp

ydb/core/mind/dynamic_nameserver_impl.h

github-actions · 2024-12-18T10:12:53Z

⚪ 2024-12-18 10:12:52 UTC Pre-commit check linux-x86_64-release-asan for 7cf0ae6 has started.
⚪ 2024-12-18 10:13:04 UTC Artifacts will be uploaded here
⚪ 2024-12-18 10:16:15 UTC ya make is running...
🔴 2024-12-18 10:39:01 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-12-18T10:14:42Z

⚪ 2024-12-18 10:14:42 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7cf0ae6 has started.
⚪ 2024-12-18 10:15:18 UTC Artifacts will be uploaded here
⚪ 2024-12-18 10:19:20 UTC ya make is running...
⚫ 2024-12-18 10:44:04 UTC Check cancelled

github-actions · 2024-12-18T10:45:44Z

⚪ 2024-12-18 10:45:44 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2322d60 has started.
⚪ 2024-12-18 10:45:55 UTC Artifacts will be uploaded here
⚪ 2024-12-18 10:49:03 UTC ya make is running...
🟡 2024-12-18 11:51:34 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
31298	28342	0	8	2832	116

⚪ 2024-12-18 11:54:28 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-18 12:06:27 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
173 (only retried tests)	62	0	0	0	111

🟢 2024-12-18 12:06:35 UTC Build successful.
🟢 2024-12-18 12:06:56 UTC ydbd size 2.1 GiB changed* by +35.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `3d7fe12`	merge: `2322d60`	diff	diff %
ydbd size	2 253 716 912 Bytes	2 253 753 520 Bytes	+35.8 KiB	+0.002%
ydbd stripped size	483 049 840 Bytes	483 054 576 Bytes	+4.6 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-18T10:47:00Z

⚪ 2024-12-18 10:47:00 UTC Pre-commit check linux-x86_64-release-asan for 2322d60 has started.
⚪ 2024-12-18 10:47:42 UTC Artifacts will be uploaded here
⚪ 2024-12-18 10:51:30 UTC ya make is running...
🟡 2024-12-18 12:07:09 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14248	14128	0	59	15	46

🟢 2024-12-18 12:08:30 UTC Build successful.
🟢 2024-12-18 12:09:02 UTC ydbd size 3.6 GiB changed* by +54.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `3d7fe12`	merge: `2322d60`	diff	diff %
ydbd size	3 894 518 104 Bytes	3 894 573 544 Bytes	+54.1 KiB	+0.001%
ydbd stripped size	1 363 834 448 Bytes	1 363 854 864 Bytes	+19.9 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

ydb/core/viewer/viewer_ut.cpp

github-actions · 2024-12-18T15:29:14Z

⚪ 2024-12-18 15:29:14 UTC Pre-commit check linux-x86_64-release-asan for 81556aa has started.
⚪ 2024-12-18 15:30:01 UTC Artifacts will be uploaded here
⚪ 2024-12-18 15:33:28 UTC ya make is running...
🟡 2024-12-18 17:05:10 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14254	14127	0	67	12	48

🟢 2024-12-18 17:06:17 UTC ydbd size 3.6 GiB changed* by +50.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `47dde79`	merge: `81556aa`	diff	diff %
ydbd size	3 895 061 568 Bytes	3 895 112 824 Bytes	+50.1 KiB	+0.001%
ydbd stripped size	1 364 015 184 Bytes	1 364 031 440 Bytes	+15.9 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2024-12-18 17:06:19 UTC Check cancelled

github-actions · 2024-12-18T15:30:31Z

⚪ 2024-12-18 15:30:30 UTC Pre-commit check linux-x86_64-relwithdebinfo for 81556aa has started.
⚪ 2024-12-18 15:30:42 UTC Artifacts will be uploaded here
⚪ 2024-12-18 15:34:06 UTC ya make is running...
⚫ 2024-12-18 17:07:04 UTC Check cancelled

github-actions · 2024-12-18T17:08:04Z

⚪ 2024-12-18 17:08:03 UTC Pre-commit check linux-x86_64-relwithdebinfo for 76e9c42 has started.
⚪ 2024-12-18 17:08:37 UTC Artifacts will be uploaded here
⚪ 2024-12-18 17:12:23 UTC ya make is running...
🟡 2024-12-18 18:36:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
31317	28361	0	7	2831	118

⚪ 2024-12-18 18:40:13 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-12-18 18:52:29 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
171 (only retried tests)	57	0	3	0	111

⚪ 2024-12-18 18:52:37 UTC ya make is running... (failed tests rerun, try 3)
🟢 2024-12-18 19:04:42 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
160 (only retried tests)	50	0	0	0	110

🟢 2024-12-18 19:04:49 UTC Build successful.
🟢 2024-12-18 19:05:12 UTC ydbd size 2.1 GiB changed* by +35.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `ae2a5a5`	merge: `76e9c42`	diff	diff %
ydbd size	2 254 434 712 Bytes	2 254 471 256 Bytes	+35.7 KiB	+0.002%
ydbd stripped size	483 155 760 Bytes	483 160 432 Bytes	+4.6 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-18T17:08:41Z

⚪ 2024-12-18 17:08:40 UTC Pre-commit check linux-x86_64-release-asan for 76e9c42 has started.
⚪ 2024-12-18 17:09:20 UTC Artifacts will be uploaded here
⚪ 2024-12-18 17:12:35 UTC ya make is running...
🟡 2024-12-18 18:59:05 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14258	14138	0	59	13	48

🟢 2024-12-18 19:00:55 UTC Build successful.
🟢 2024-12-18 19:01:23 UTC ydbd size 3.6 GiB changed* by +46.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `ae2a5a5`	merge: `76e9c42`	diff	diff %
ydbd size	3 895 758 264 Bytes	3 895 805 416 Bytes	+46.0 KiB	+0.001%
ydbd stripped size	1 364 174 288 Bytes	1 364 186 448 Bytes	+11.9 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

Add ListNodes cache in DynamicNameserver

1da63a3

pixcc self-assigned this Dec 17, 2024

pixcc requested review from a team as code owners December 17, 2024 23:39

github-actions bot added performance and removed performance labels Dec 17, 2024

pixcc requested review from CyberROFL and snaury December 17, 2024 23:40

github-actions bot added performance and removed performance labels Dec 17, 2024

Fix build

f7b8277

Fix build

92309cc

Fix build

acd8a4e

Fix build

c880ef8

snaury reviewed Dec 18, 2024

View reviewed changes

Fix review issues and build

cf6aea3

pixcc requested a review from snaury December 18, 2024 10:10

Fix build

2f46766

snaury reviewed Dec 18, 2024

View reviewed changes

ydb/core/viewer/viewer_ut.cpp Outdated Show resolved Hide resolved

Get rid of const_cast

e631604

pixcc requested a review from snaury December 18, 2024 15:31

vitstn previously approved these changes Dec 18, 2024

View reviewed changes

snaury previously approved these changes Dec 18, 2024

View reviewed changes

Fix tests

5018737

pixcc dismissed stale reviews from snaury and vitstn via 5018737 December 18, 2024 17:05

pixcc requested review from snaury and vitstn December 18, 2024 17:05

snaury approved these changes Dec 18, 2024

View reviewed changes

vitstn approved these changes Dec 18, 2024

View reviewed changes

uzhastik approved these changes Dec 18, 2024

View reviewed changes

adameat approved these changes Dec 18, 2024

View reviewed changes

pixcc merged commit f3c9a60 into ydb-platform:main Dec 18, 2024
10 checks passed

pixcc added a commit to pixcc/ydb that referenced this pull request Dec 18, 2024

Add ListNodes cache in DynamicNameserver (ydb-platform#12694)

3b6f686

pixcc added a commit that referenced this pull request Dec 19, 2024

24-3: Add ListNodes cache in DynamicNameserver (#12694) (#12747)

11eb58f

GrigoriyPA mentioned this pull request Dec 25, 2024

Fixed empty error from CMS create tenant request #12977

Merged

This was referenced Jun 3, 2025

Enable kafka port for local ydb container #19218

Closed

Enable kafka port for local ydb container #19220

Closed

Add ListNodes cache in DynamicNameserver #12694

Add ListNodes cache in DynamicNameserver #12694

Uh oh!

Conversation

pixcc commented Dec 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog entry

Changelog category

Additional information

Uh oh!

github-actions bot commented Dec 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Add `ListNodes` cache in `DynamicNameserver` #12694

Add `ListNodes` cache in `DynamicNameserver` #12694

pixcc commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading