Add streamlookup monitoring (dynamic counters) #10280

yumkam · 2024-10-10T10:31:15Z

Changelog entry

Adds dq_tasks/operation={query_id}/component=Lookup{,Src}/* sensors

Changelog category

Not for changelog (changelog entry is not required)

Additional information

...

github-actions · 2024-10-10T10:35:09Z

⚪ 2024-10-10 10:35:09 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5a23df8 has started.
⚪ 2024-10-10 10:35:19 UTC Artifacts will be uploaded here
⚪ 2024-10-10 10:38:22 UTC ya make is running...
🔴 2024-10-10 10:47:09 UTC Build failed, see the logs.

github-actions · 2024-10-10T10:35:50Z

⚪ 2024-10-10 10:35:50 UTC Pre-commit check linux-x86_64-release-asan for 5a23df8 has started.
⚪ 2024-10-10 10:36:00 UTC Artifacts will be uploaded here
⚪ 2024-10-10 10:39:07 UTC ya make is running...
⚫ 2024-10-10 10:49:20 UTC Check cancelled

github-actions · 2024-10-10T10:55:04Z

⚪ 2024-10-10 10:55:03 UTC Pre-commit check linux-x86_64-relwithdebinfo for f88c8b6 has started.
⚪ 2024-10-10 10:55:14 UTC Artifacts will be uploaded here
⚪ 2024-10-10 10:58:22 UTC ya make is running...
⚫ 2024-10-10 11:04:35 UTC Check cancelled

github-actions · 2024-10-10T10:55:09Z

⚪ 2024-10-10 10:55:09 UTC Pre-commit check linux-x86_64-release-asan for f88c8b6 has started.
⚪ 2024-10-10 10:55:20 UTC Artifacts will be uploaded here
⚪ 2024-10-10 10:58:16 UTC ya make is running...
⚫ 2024-10-10 11:04:40 UTC Check cancelled

github-actions · 2024-10-10T11:07:53Z

⚪ 2024-10-10 11:07:53 UTC Pre-commit check linux-x86_64-relwithdebinfo for c30cba5 has started.
⚪ 2024-10-10 11:08:52 UTC Artifacts will be uploaded here
⚪ 2024-10-10 11:12:28 UTC ya make is running...
🔴 2024-10-10 11:25:30 UTC Build failed, see the logs.

github-actions · 2024-10-10T11:08:08Z

⚪ 2024-10-10 11:08:08 UTC Pre-commit check linux-x86_64-release-asan for c30cba5 has started.
⚪ 2024-10-10 11:08:39 UTC Artifacts will be uploaded here
⚪ 2024-10-10 11:12:07 UTC ya make is running...
🔴 2024-10-10 11:18:44 UTC Build failed, see the logs.

github-actions · 2024-10-10T11:29:52Z

⚪ 2024-10-10 11:29:51 UTC Pre-commit check linux-x86_64-release-asan for 57ad24a has started.
⚪ 2024-10-10 11:30:02 UTC Artifacts will be uploaded here
⚪ 2024-10-10 11:33:10 UTC ya make is running...
⚫ 2024-10-10 11:35:49 UTC Check cancelled

StreamLookupTransformCount is same as GenericLookupCount StreamLookupTransformKeys is same as GenericLookupKeys StreamLookupTransformTimeMs is almost same as GenericLookupAnswerTimeMs (plus expenses on actor message passing, and it expected to be small)

github-actions · 2024-10-10T11:31:52Z

⚪ 2024-10-10 11:31:51 UTC Pre-commit check linux-x86_64-relwithdebinfo for 57ad24a has started.
⚪ 2024-10-10 11:32:02 UTC Artifacts will be uploaded here
⚪ 2024-10-10 11:35:01 UTC ya make is running...
⚫ 2024-10-10 11:35:46 UTC Check cancelled

github-actions · 2024-10-10T11:38:56Z

⚪ 2024-10-10 11:38:55 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8cfc3a4 has started.
⚪ 2024-10-10 11:38:59 UTC Artifacts will be uploaded here
⚪ 2024-10-10 11:42:01 UTC ya make is running...
⚫ 2024-10-10 11:42:50 UTC Check cancelled

github-actions · 2024-10-10T11:39:20Z

⚪ 2024-10-10 11:39:19 UTC Pre-commit check linux-x86_64-release-asan for 8cfc3a4 has started.
⚪ 2024-10-10 11:39:29 UTC Artifacts will be uploaded here
⚪ 2024-10-10 11:42:26 UTC ya make is running...
⚫ 2024-10-10 11:42:49 UTC Check cancelled

github-actions · 2024-10-10T11:46:06Z

⚪ 2024-10-10 11:46:06 UTC Pre-commit check linux-x86_64-relwithdebinfo for c470553 has started.
⚪ 2024-10-10 11:46:16 UTC Artifacts will be uploaded here
⚪ 2024-10-10 11:49:24 UTC ya make is running...
⚫ 2024-10-10 12:01:44 UTC Check cancelled

github-actions · 2024-10-10T11:46:07Z

⚪ 2024-10-10 11:46:06 UTC Pre-commit check linux-x86_64-release-asan for c470553 has started.
⚪ 2024-10-10 11:46:17 UTC Artifacts will be uploaded here
⚪ 2024-10-10 11:49:17 UTC ya make is running...
⚫ 2024-10-10 12:01:44 UTC Check cancelled

Lookup{,Src}* -> component[Lookup{,Src}].*

github-actions · 2024-10-16T09:44:42Z

⚪ 2024-10-16 09:44:42 UTC Pre-commit check linux-x86_64-relwithdebinfo for e55f8c3 has started.
⚪ 2024-10-16 09:44:53 UTC Artifacts will be uploaded here
⚪ 2024-10-16 09:47:53 UTC ya make is running...
🟢 2024-10-16 10:56:27 UTC ydbd size 2.8 GiB changed* by +64.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `fff7685`	merge: `e55f8c3`	diff	diff %
ydbd size	3 010 921 712 Bytes	3 010 987 808 Bytes	+64.5 KiB	+0.002%
ydbd stripped size	477 552 408 Bytes	477 561 496 Bytes	+8.9 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2024-10-16 10:56:29 UTC Check cancelled

github-actions · 2024-10-16T09:44:46Z

⚪ 2024-10-16 09:44:46 UTC Pre-commit check linux-x86_64-release-asan for e55f8c3 has started.
⚪ 2024-10-16 09:44:57 UTC Artifacts will be uploaded here
⚪ 2024-10-16 09:47:58 UTC ya make is running...
⚫ 2024-10-16 10:56:07 UTC Check cancelled

ydb/tests/fq/generic/streaming/test_join.py

github-actions · 2024-10-16T10:58:08Z

⚪ 2024-10-16 10:58:07 UTC Pre-commit check linux-x86_64-relwithdebinfo for 52016ed has started.
⚪ 2024-10-16 10:58:12 UTC Artifacts will be uploaded here
⚪ 2024-10-16 11:01:24 UTC ya make is running...
🟡 2024-10-16 12:06:26 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
74968	60990	0	4	13871	103

⚪ 2024-10-16 12:13:28 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-16 12:22:51 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
106 (only retried tests)	7	0	1	0	98

⚪ 2024-10-16 12:23:00 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-10-16 12:32:07 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
99 (only retried tests)	1	0	1	0	97

🟢 2024-10-16 12:32:14 UTC Build successful.
🟢 2024-10-16 12:32:38 UTC ydbd size 2.8 GiB changed* by +64.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `030f818`	merge: `52016ed`	diff	diff %
ydbd size	3 010 928 160 Bytes	3 010 994 320 Bytes	+64.6 KiB	+0.002%
ydbd stripped size	477 553 432 Bytes	477 562 584 Bytes	+8.9 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-16T10:58:52Z

⚪ 2024-10-16 10:58:51 UTC Pre-commit check linux-x86_64-release-asan for 52016ed has started.
⚪ 2024-10-16 10:58:56 UTC Artifacts will be uploaded here
⚪ 2024-10-16 11:01:54 UTC ya make is running...
🟡 2024-10-16 12:32:24 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
12099	11840	0	77	73	109

🟢 2024-10-16 12:33:25 UTC Build successful.
🟡 2024-10-16 12:34:02 UTC ydbd size 5.7 GiB changed* by +187.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `030f818`	merge: `52016ed`	diff	diff %
ydbd size	6 099 890 304 Bytes	6 100 082 336 Bytes	+187.5 KiB	+0.003%
ydbd stripped size	1 521 081 712 Bytes	1 521 116 528 Bytes	+34.0 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-16T15:24:24Z

⚪ 2024-10-16 15:24:23 UTC Pre-commit check linux-x86_64-release-asan for 055049a has started.
⚪ 2024-10-16 15:24:34 UTC Artifacts will be uploaded here
⚪ 2024-10-16 15:27:41 UTC ya make is running...
🟡 2024-10-16 17:05:34 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
12107	11868	0	57	79	103

🟢 2024-10-16 17:06:37 UTC Build successful.
🟡 2024-10-16 17:07:12 UTC ydbd size 5.7 GiB changed* by +191.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `8739de4`	merge: `055049a`	diff	diff %
ydbd size	6 129 681 552 Bytes	6 129 877 672 Bytes	+191.5 KiB	+0.003%
ydbd stripped size	1 529 354 544 Bytes	1 529 393 456 Bytes	+38.0 KiB	+0.003%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-16T15:25:44Z

⚪ 2024-10-16 15:25:44 UTC Pre-commit check linux-x86_64-relwithdebinfo for 055049a has started.
⚪ 2024-10-16 15:25:55 UTC Artifacts will be uploaded here
⚪ 2024-10-16 15:29:04 UTC ya make is running...
🟡 2024-10-16 16:40:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
74982	60999	0	10	13874	99

⚪ 2024-10-16 16:47:48 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-10-16 16:59:16 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
112 (only retried tests)	13	0	0	1	98

🟢 2024-10-16 16:59:24 UTC Build successful.
🟢 2024-10-16 16:59:43 UTC ydbd size 2.8 GiB changed* by +64.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `8739de4`	merge: `055049a`	diff	diff %
ydbd size	3 028 486 472 Bytes	3 028 552 568 Bytes	+64.5 KiB	+0.002%
ydbd stripped size	480 129 624 Bytes	480 138 712 Bytes	+8.9 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

…toring

github-actions · 2024-10-17T12:32:12Z

⚪ 2024-10-17 12:32:11 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5c257c4 has started.
⚪ 2024-10-17 12:32:23 UTC Artifacts will be uploaded here
⚪ 2024-10-17 12:35:26 UTC ya make is running...
🟡 2024-10-17 14:08:58 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
74945	60969	0	6	13874	96

⚪ 2024-10-17 14:16:05 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-10-17 14:27:31 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
104 (only retried tests)	14	0	0	0	90

🟢 2024-10-17 14:27:39 UTC Build successful.
🟢 2024-10-17 14:27:59 UTC ydbd size 2.8 GiB changed* by +67.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `be635ce`	merge: `5c257c4`	diff	diff %
ydbd size	3 028 306 248 Bytes	3 028 375 616 Bytes	+67.7 KiB	+0.002%
ydbd stripped size	480 012 920 Bytes	480 022 264 Bytes	+9.1 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-17T12:32:55Z

⚪ 2024-10-17 12:32:54 UTC Pre-commit check linux-x86_64-release-asan for 5c257c4 has started.
⚪ 2024-10-17 12:33:07 UTC Artifacts will be uploaded here
⚪ 2024-10-17 12:36:20 UTC ya make is running...
🟡 2024-10-17 14:56:27 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
12100	11541	0	132	375	52

🟢 2024-10-17 14:57:32 UTC Build successful.
🟡 2024-10-17 14:58:06 UTC ydbd size 5.7 GiB changed* by +184.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `f33f3fe`	merge: `5c257c4`	diff	diff %
ydbd size	6 129 656 288 Bytes	6 129 844 720 Bytes	+184.0 KiB	+0.003%
ydbd stripped size	1 529 234 448 Bytes	1 529 265 296 Bytes	+30.1 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

(cherry picked from commit 707927e)

…7892 #10280 (#10341) Co-authored-by: dmasloff <74042473+dmasloff@users.noreply.github.com> Co-authored-by: Hor911 <hor911@ydb.tech>

github-actions bot added the not-for-changelog label Oct 10, 2024

yumkam requested review from Hor911, uzhastik, vitalyisaev2 and zverevgeny October 10, 2024 10:31

yumkam added 6 commits October 10, 2024 13:33

compute actor: pass counters to input transform

9df7778

input transform: pass in task counters

012981a

stream lookup transform: add dynamic counters

7c60f31

generic lookup: add dynamic counters

ddf5570

tests/tools/fq_runner/kikimr_metrics: avoid failure on missing "value"

322e48c

tests/fq/generic/streaming/test_join: dump counters in debug mode

4967983

yumkam force-pushed the add-streamlookup-monitoring branch from d0208d1 to 4967983 Compare October 10, 2024 10:33

streamlookup: fix batch counting

d81e085

yumkam added 2 commits October 10, 2024 14:28

fix genericlookup ut

4522de1

fix python style

49c4fdc

yumkam force-pushed the add-streamlookup-monitoring branch from 74474f3 to 49c4fdc Compare October 10, 2024 11:28

Remove duplicated counters

11d659e

StreamLookupTransformCount is same as GenericLookupCount StreamLookupTransformKeys is same as GenericLookupKeys StreamLookupTransformTimeMs is almost same as GenericLookupAnswerTimeMs (plus expenses on actor message passing, and it expected to be small)

fix build (unused variable)

d359dd3

uzhastik previously approved these changes Oct 16, 2024

View reviewed changes

Move/rename sensors to component

4044d1f

Lookup{,Src}* -> component[Lookup{,Src}].*

yumkam dismissed uzhastik’s stale review via 4044d1f October 16, 2024 09:41

uzhastik reviewed Oct 16, 2024

View reviewed changes

ydb/tests/fq/generic/streaming/test_join.py Outdated Show resolved Hide resolved

tests: tuple->list

2714c29

yumkam requested a review from uzhastik October 16, 2024 10:57

github-actions bot added not-for-changelog and removed not-for-changelog labels Oct 16, 2024

fix python style

894b751

rvu1024 previously approved these changes Oct 17, 2024

View reviewed changes

Merge remote-tracking branch 'origin/main' into add-streamlookup-moni…

10c14ed

…toring

yumkam dismissed rvu1024’s stale review via 10c14ed October 17, 2024 12:29

rvu1024 approved these changes Oct 17, 2024

View reviewed changes

yumkam enabled auto-merge (squash) October 17, 2024 16:49

uzhastik approved these changes Oct 18, 2024

View reviewed changes

yumkam merged commit 707927e into ydb-platform:main Oct 18, 2024
10 checks passed

yumkam added a commit to yumkam/ydb that referenced this pull request Oct 18, 2024

Add streamlookup monitoring (dynamic counters) (ydb-platform#10280)

9de7a0b

(cherry picked from commit 707927e)

yumkam added a commit that referenced this pull request Oct 18, 2024

Streamlookup stable backports #7782 #9758 #9396 #10283 #10489 #10508 #…

9a9711d

…7892 #10280 (#10341) Co-authored-by: dmasloff <74042473+dmasloff@users.noreply.github.com> Co-authored-by: Hor911 <hor911@ydb.tech>

This was referenced Oct 22, 2024

YQ RD pass UV from parser to filter #10721

Merged

YQ-3784 support computation graph invalidation in purecalc #10799

Merged

GrigoriyPA mentioned this pull request Oct 29, 2024

YQ RD pass UV from parser to filter #11078

Closed

shnikd mentioned this pull request Nov 7, 2024

Support QueryMeta and diagnostics #11371

Merged

Add streamlookup monitoring (dynamic counters) #10280

Add streamlookup monitoring (dynamic counters) #10280

Uh oh!

Conversation

yumkam commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog entry

Changelog category

Additional information

Uh oh!

github-actions bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yumkam commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Oct 17, 2024 •

edited

Loading

github-actions bot commented Oct 17, 2024 •

edited

Loading