Skip to content

add missing monitoring metrics for column shards #7314

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Aug 7, 2024
Merged

add missing monitoring metrics for column shards #7314

merged 15 commits into from
Aug 7, 2024

Conversation

swalrus1
Copy link
Collaborator

@swalrus1 swalrus1 commented Jul 31, 2024 •
edited
Loading

Track the following Monitoring metrics for column shards:

table.columnshard.used_core_percents
table.columnshard.read.bytes
table.columnshard.read.rows
table.columnshard.write.rows
table.columnshard.write.bytes
table.columnshard.erase.rows
table.columnshard.erase.bytes

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 31, 2024 •
edited
Loading

⚪ 2024-07-31 13:18:07 UTC Pre-commit check for b9a09e4 has started.
⚪ 2024-07-31 13:22:21 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-07-31 13:37:01 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 31, 2024 •
edited
Loading

⚪ 2024-07-31 13:18:49 UTC Pre-commit check for b9a09e4 has started.
⚪ 2024-07-31 13:22:07 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-31 13:33:10 UTC Build failed. see the logs.

@swalrus1 swalrus1 force-pushed the add-monitoring-metrics branch from 211aed8 to c35dccf Compare August 2, 2024 07:31
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 07:33:12 UTC Pre-commit check for 0b79bb7 has started.
⚫ 2024-08-02 07:35:32 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 07:34:19 UTC Pre-commit check for 0b79bb7 has started.
⚫ 2024-08-02 07:35:39 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 07:34:31 UTC Pre-commit check for 0b79bb7 has started.
⚫ 2024-08-02 07:35:40 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 07:37:08 UTC Pre-commit check for 1cc92bb has started.
⚪ 2024-08-02 07:40:03 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-02 09:30:07 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39650 34317 0 4 5306 23

⚪ 2024-08-02 09:34:40 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-02 09:42:32 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
25 (only retried tests) 13 0 0 0 12

🟢 2024-08-02 09:45:58 UTC Build successful.
🟢 2024-08-02 09:46:32 UTC ydbd size 8.1 GiB changed* by +50.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 6416406 merge: 1cc92bb diff diff %
ydbd size 8 676 085 112 Bytes 8 676 136 712 Bytes +50.4 KiB +0.001%
ydbd stripped size 472 182 088 Bytes 472 186 440 Bytes +4.2 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 07:38:53 UTC Pre-commit check for 1cc92bb has started.
⚪ 2024-08-02 07:41:45 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-02 07:49:13 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 07:39:18 UTC Pre-commit check for 1cc92bb has started.
⚪ 2024-08-02 07:42:22 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-02 09:47:20 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10897 10855 0 7 23 12

🟢 2024-08-02 09:48:32 UTC Build successful.
🟡 2024-08-02 09:49:03 UTC ydbd size 5.4 GiB changed* by +108.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 153df90 merge: 1cc92bb diff diff %
ydbd size 5 843 615 264 Bytes 5 843 725 816 Bytes +108.0 KiB +0.002%
ydbd stripped size 1 464 332 496 Bytes 1 464 354 640 Bytes +21.6 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 14:19:13 UTC Pre-commit check for 774ab30 has started.
⚪ 2024-08-02 14:22:33 UTC Check linux-x86_64-release-clang14 is running...
🔴 2024-08-02 14:26:04 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 14:20:15 UTC Pre-commit check for 774ab30 has started.
⚪ 2024-08-02 14:24:27 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-08-02 14:28:23 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 14:20:40 UTC Pre-commit check for 774ab30 has started.
⚪ 2024-08-02 14:23:38 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-02 14:27:18 UTC Build failed. see the logs.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 14:49:30 UTC Pre-commit check for 18e2365 has started.
⚪ 2024-08-02 14:52:37 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-02 15:01:15 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 14:49:42 UTC Pre-commit check for 18e2365 has started.
⚪ 2024-08-02 14:53:35 UTC Check linux-x86_64-release-asan is running...
⚫ 2024-08-02 15:06:13 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 14:50:34 UTC Pre-commit check for 18e2365 has started.
⚪ 2024-08-02 14:53:52 UTC Check linux-x86_64-relwithdebinfo is running...
⚫ 2024-08-02 15:06:04 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 15:07:46 UTC Pre-commit check for eb4e5d6 has started.
⚪ 2024-08-02 15:11:32 UTC Check linux-x86_64-release-asan is running...
⚫ 2024-08-02 15:19:20 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 15:09:39 UTC Pre-commit check for eb4e5d6 has started.
⚪ 2024-08-02 15:13:32 UTC Check linux-x86_64-release-clang14 is running...
⚫ 2024-08-02 15:19:16 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 15:10:30 UTC Pre-commit check for eb4e5d6 has started.
⚪ 2024-08-02 15:13:47 UTC Check linux-x86_64-relwithdebinfo is running...
⚫ 2024-08-02 15:19:18 UTC Check cancelled

@swalrus1 swalrus1 force-pushed the add-monitoring-metrics branch from 761130e to 8c5596e Compare August 2, 2024 15:18
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 15:21:04 UTC Pre-commit check for f5d4485 has started.
⚪ 2024-08-02 15:24:07 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-02 15:30:47 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 15:22:55 UTC Pre-commit check for f5d4485 has started.
⚪ 2024-08-02 15:26:50 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-02 17:36:31 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10896 10849 0 12 22 13

🟢 2024-08-02 17:37:45 UTC Build successful.
🟡 2024-08-02 17:38:15 UTC ydbd size 5.4 GiB changed* by +120.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: ca41215 merge: f5d4485 diff diff %
ydbd size 5 824 578 784 Bytes 5 824 701 752 Bytes +120.1 KiB +0.002%
ydbd stripped size 1 463 004 432 Bytes 1 463 030 928 Bytes +25.9 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024 •
edited
Loading

⚪ 2024-08-02 15:23:22 UTC Pre-commit check for f5d4485 has started.
⚪ 2024-08-02 15:27:13 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-02 17:25:44 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39661 34341 0 2 5307 11

⚪ 2024-08-02 17:30:11 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-02 17:37:30 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15 (only retried tests) 7 0 0 0 8

🟢 2024-08-02 17:40:50 UTC Build successful.
🟢 2024-08-02 17:41:23 UTC ydbd size 8.1 GiB changed* by +64.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: e9a7b28 merge: f5d4485 diff diff %
ydbd size 8 664 088 744 Bytes 8 664 154 592 Bytes +64.3 KiB +0.001%
ydbd stripped size 471 413 320 Bytes 471 417 800 Bytes +4.4 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@swalrus1 swalrus1 linked an issue Aug 2, 2024 that may be closed by this pull request
Add monitoring metrics for column shards #6628
Closed
@swalrus1 swalrus1 marked this pull request as ready for review August 2, 2024 16:06
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 7, 2024 •
edited
Loading

⚪ 2024-08-07 10:09:52 UTC Pre-commit check for df9a82d has started.
⚪ 2024-08-07 10:12:50 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-07 10:27:32 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 7, 2024 •
edited
Loading

⚪ 2024-08-07 10:10:17 UTC Pre-commit check for df9a82d has started.
⚪ 2024-08-07 10:13:17 UTC Check linux-x86_64-release-asan is running...
⚫ 2024-08-07 10:58:07 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 7, 2024 •
edited
Loading

⚪ 2024-08-07 10:10:21 UTC Pre-commit check for df9a82d has started.
⚪ 2024-08-07 10:13:18 UTC Check linux-x86_64-relwithdebinfo is running...
⚫ 2024-08-07 10:58:05 UTC Check cancelled

swalrus1
swalrus1 commented Aug 7, 2024
View reviewed changes
ydb/core/tablet/tablet_counters_aggregator.cpp
ColumnShardScanRows_->Set(ColumnShardScannedRows_->Val());
ColumnShardScanBytes_->Set(ColumnShardScannedBytes_->Val());
ColumnShardBulkUpsertRows_->Set(ColumnShardUpsertBlobsWritten_->Val());
ColumnShardBulkUpsertBytes_->Set(ColumnShardUpsertBytesWritten_->Val());
ColumnShardWriteRows_->Set(ColumnShardOperationsRowsWritten_->Val());
Copy link
Collaborator Author

@swalrus1 swalrus1 Aug 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

В ColumnShardWrite* и ColumnShardBulkUpsert* одинаковые значения, потому что сейчас непонятно, как на стороне шарда различать записи через bulk upload от остальных -- когда поддержим транзакции, надо будет разделить эти метрики

ivanmorozov333
ivanmorozov333 reviewed Aug 7, 2024
View reviewed changes
ydb/core/tx/columnshard/columnshard__write.cpp Outdated
@@ -128,8 +128,12 @@ void TColumnShard::Handle(TEvPrivate::TEvWriteBlobsResult::TPtr& ev, const TActo
Counters.GetCSCounters().OnFailedWriteResponse(EWriteFailReason::PutBlob);
wBuffer.RemoveData(aggr, StoragesManager->GetInsertOperator());
} else {
ui64 rows = 0;
for (auto& batch : aggr->GetSplittedBlobs()) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

сделай aggr->GetRecordsCount()

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 7, 2024 •
edited
Loading

⚪ 2024-08-07 10:59:34 UTC Pre-commit check for 5c1ef13 has started.
⚪ 2024-08-07 11:04:13 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-07 13:12:54 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10904 10852 0 15 25 12

🟢 2024-08-07 13:14:08 UTC Build successful.
🟡 2024-08-07 13:14:39 UTC ydbd size 5.4 GiB changed* by +103.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: c319be4 merge: 5c1ef13 diff diff %
ydbd size 5 831 649 440 Bytes 5 831 755 336 Bytes +103.4 KiB +0.002%
ydbd stripped size 1 464 892 400 Bytes 1 464 909 584 Bytes +16.8 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 7, 2024 •
edited
Loading

⚪ 2024-08-07 11:01:10 UTC Pre-commit check for 5c1ef13 has started.
⚪ 2024-08-07 11:04:26 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-07 11:11:05 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 7, 2024 •
edited
Loading

⚪ 2024-08-07 11:01:24 UTC Pre-commit check for 5c1ef13 has started.
⚪ 2024-08-07 11:04:19 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-07 12:54:08 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39678 34352 0 5 5308 13

⚪ 2024-08-07 12:58:53 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-07 13:07:50 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
19 (only retried tests) 10 0 0 0 9

🟢 2024-08-07 13:11:26 UTC Build successful.
🟢 2024-08-07 13:12:00 UTC ydbd size 8.1 GiB changed* by +65.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: fb30ef5 merge: 5c1ef13 diff diff %
ydbd size 8 672 777 904 Bytes 8 672 844 696 Bytes +65.2 KiB +0.001%
ydbd stripped size 471 991 272 Bytes 471 995 752 Bytes +4.4 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@swalrus1 swalrus1 merged commit 7ad9e1e into ydb-platform:main Aug 7, 2024
10 of 12 checks passed
@StekPerepolnen StekPerepolnen mentioned this pull request Aug 8, 2024
Merged
This was referenced Aug 8, 2024
Merged
Merged
@swalrus1 swalrus1 deleted the add-monitoring-metrics branch September 3, 2024 15:22
@swalrus1 swalrus1 mentioned this pull request Sep 3, 2024
Closed
swalrus1 added a commit to swalrus1/ydb that referenced this pull request Sep 3, 2024
@swalrus1
add missing monitoring metrics for column shards (ydb-platform#7314)
e30f0c0
@swalrus1 swalrus1 mentioned this pull request Sep 3, 2024
Closed
swalrus1 added a commit to swalrus1/ydb that referenced this pull request Sep 3, 2024
@swalrus1
add missing monitoring metrics for column shards (ydb-platform#7314)
b4b11c1
swalrus1 added a commit to swalrus1/ydb that referenced this pull request Sep 4, 2024
@swalrus1
add missing monitoring metrics for column shards (ydb-platform#7314)
9fd65a6
This was referenced Sep 4, 2024
Closed
Closed
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 9, 2024
@swalrus1 @zverevgeny
add missing monitoring metrics for column shards (ydb-platform#7314)
1232116
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 10, 2024
@swalrus1 @zverevgeny
add missing monitoring metrics for column shards (ydb-platform#7314)
500d755
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 11, 2024
@swalrus1 @zverevgeny
add missing monitoring metrics for column shards (ydb-platform#7314)
70c0341
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 12, 2024
@swalrus1 @zverevgeny
add missing monitoring metrics for column shards (ydb-platform#7314)
110607b
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 14, 2024
@swalrus1 @zverevgeny
add missing monitoring metrics for column shards (ydb-platform#7314)
97223ae
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 15, 2024
@swalrus1 @zverevgeny
add missing monitoring metrics for column shards (ydb-platform#7314)
b0b32a0
@zverevgeny zverevgeny mentioned this pull request Sep 16, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ivanmorozov333 ivanmorozov333 ivanmorozov333 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add monitoring metrics for column shards
2 participants
@swalrus1 @ivanmorozov333