Skip to content

HLL in YT statistics #8184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 10, 2024
Merged

HLL in YT statistics #8184

merged 1 commit into from
Sep 10, 2024

Conversation

@alephonea
Copy link
Contributor

@alephonea alephonea commented Aug 22, 2024 •
edited
Loading

Use StatisticsCache and TYtLoadColumnarStatsTransformer. EYtSettingType::ExtendedStatColumns triggers loading extended columnar statistics, protected by pragma.

Change log messages in ExecPathStat() by including full RichYPath, with columns.
Put sort columns and per-column data weight into TYtProviderStatistic. These fields will be used later in the cbo cost function.

@alephonea alephonea requested a review from a team as a code owner August 22, 2024 21:12
@github-actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 21:16:15 UTC Pre-commit check for b8988d4 has started.
⚪ 2024-08-22 21:19:14 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-22 21:36:37 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 21:16:28 UTC Pre-commit check for b8988d4 has started.
⚪ 2024-08-22 21:19:22 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-08-22 22:41:59 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
73869 60198 0 24 13627 20

🟢 2024-08-22 22:49:00 UTC Build successful.
🟡 2024-08-22 22:49:37 UTC ydbd size 8.2 GiB changed* by +1.7 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 4a97ccb merge: b8988d4 diff diff %
ydbd size 8 753 334 272 Bytes 8 755 105 984 Bytes +1.7 MiB +0.020%
ydbd stripped size 476 022 536 Bytes 476 064 904 Bytes +41.4 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 21:16:33 UTC Pre-commit check for b8988d4 has started.
⚪ 2024-08-22 21:19:25 UTC Check linux-x86_64-release-asan is running...
⚫ 2024-08-22 23:06:04 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 23:09:28 UTC Pre-commit check for 4ae555b has started.
⚪ 2024-08-22 23:12:17 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-22 23:19:00 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 23:09:43 UTC Pre-commit check for 4ae555b has started.
⚪ 2024-08-22 23:12:29 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-23 00:42:27 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11943 11859 0 32 36 16

🟢 2024-08-23 00:43:29 UTC Build successful.
🟡 2024-08-23 00:43:59 UTC ydbd size 5.5 GiB changed* by +1.3 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 4a97ccb merge: 4ae555b diff diff %
ydbd size 5 890 879 200 Bytes 5 892 206 984 Bytes +1.3 MiB +0.023%
ydbd stripped size 1 479 181 744 Bytes 1 479 332 208 Bytes +146.9 KiB +0.010%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Aug 22, 2024 •
edited
Loading

⚪ 2024-08-22 23:10:23 UTC Pre-commit check for 4ae555b has started.
⚪ 2024-08-22 23:13:18 UTC Check linux-x86_64-relwithdebinfo is running...
🔴 2024-08-23 00:24:57 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
73869 60201 0 16 13628 24

🟢 2024-08-23 00:32:03 UTC Build successful.
🟡 2024-08-23 00:32:39 UTC ydbd size 8.2 GiB changed* by +1.7 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 4a97ccb merge: 4ae555b diff diff %
ydbd size 8 753 334 272 Bytes 8 755 107 048 Bytes +1.7 MiB +0.020%
ydbd stripped size 476 022 536 Bytes 476 065 096 Bytes +41.6 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Aug 26, 2024 •
edited
Loading

⚪ 2024-08-26 23:42:29 UTC Pre-commit check for d1185b2 has started.
⚪ 2024-08-26 23:45:19 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-27 00:02:41 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Aug 26, 2024 •
edited
Loading

⚪ 2024-08-26 23:42:30 UTC Pre-commit check for d1185b2 has started.
⚪ 2024-08-26 23:45:25 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-27 01:06:01 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
74021 60318 0 6 13681 16

⚪ 2024-08-27 01:12:53 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-27 01:19:48 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
25 (only retried tests) 15 0 0 0 10

🟢 2024-08-27 01:19:56 UTC Build successful.
🟡 2024-08-27 01:20:32 UTC ydbd size 8.2 GiB changed* by +1.7 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: d9ff14c merge: d1185b2 diff diff %
ydbd size 8 796 903 120 Bytes 8 798 675 896 Bytes +1.7 MiB +0.020%
ydbd stripped size 479 778 312 Bytes 479 820 872 Bytes +41.6 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Aug 26, 2024 •
edited
Loading

⚪ 2024-08-26 23:42:45 UTC Pre-commit check for d1185b2 has started.
⚪ 2024-08-26 23:45:39 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-27 01:39:32 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11983 11841 0 56 62 24

🟢 2024-08-27 01:40:35 UTC Build successful.
🟡 2024-08-27 01:41:32 UTC ydbd size 5.5 GiB changed* by +1.3 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: d9ff14c merge: d1185b2 diff diff %
ydbd size 5 926 967 776 Bytes 5 928 291 504 Bytes +1.3 MiB +0.022%
ydbd stripped size 1 488 845 840 Bytes 1 488 992 272 Bytes +143.0 KiB +0.010%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Aug 28, 2024 •
edited
Loading

⚪ 2024-08-28 19:40:45 UTC Pre-commit check linux-x86_64-release-asan for 8abd065 has started.
⚫ 2024-08-28 19:43:01 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Aug 28, 2024

⚫ 2024-08-28 19:42:54 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Aug 28, 2024 •
edited
Loading

⚪ 2024-08-28 19:47:00 UTC Pre-commit check linux-x86_64-release-asan for 4253e0b has started.
⚪ 2024-08-28 19:49:49 UTC ya make is running...
🔴 2024-08-28 21:30:17 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11981 11893 0 35 32 21

🟢 2024-08-28 21:31:21 UTC Build successful.
🟡 2024-08-28 21:31:53 UTC ydbd size 5.5 GiB changed* by +1.2 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: e5fc449 merge: 4253e0b diff diff %
ydbd size 5 936 324 264 Bytes 5 937 580 896 Bytes +1.2 MiB +0.021%
ydbd stripped size 1 490 960 048 Bytes 1 491 088 400 Bytes +125.3 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Aug 28, 2024 •
edited
Loading

⚪ 2024-08-28 19:49:00 UTC Pre-commit check linux-x86_64-release-clang14 for 4253e0b has started.
⚪ 2024-08-28 19:51:48 UTC ya make is running...
🟢 2024-08-28 20:07:22 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Aug 28, 2024 •
edited
Loading

⚪ 2024-08-28 19:49:19 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4253e0b has started.
⚪ 2024-08-28 19:52:11 UTC ya make is running...
🟡 2024-08-28 21:12:08 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
74051 60335 0 17 13682 17

⚪ 2024-08-28 21:19:13 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-08-28 21:26:28 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
37 (only retried tests) 11 0 15 0 11

⚪ 2024-08-28 21:26:36 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-08-28 21:34:08 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
31 (only retried tests) 5 0 15 0 11

🟢 2024-08-28 21:34:15 UTC Build successful.
🟡 2024-08-28 21:34:53 UTC ydbd size 8.2 GiB changed* by +1.7 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: e5fc449 merge: 4253e0b diff diff %
ydbd size 8 809 687 632 Bytes 8 811 418 856 Bytes +1.7 MiB +0.020%
ydbd stripped size 480 401 608 Bytes 480 436 616 Bytes +34.2 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

rvu1024
rvu1024 reviewed Sep 3, 2024
View reviewed changes
@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 14:02:41 UTC Pre-commit check linux-x86_64-release-asan for deea3ff has started.
⚪ 2024-09-04 14:06:46 UTC ya make is running...
🔴 2024-09-04 16:08:18 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12097 11933 0 60 78 26

🟢 2024-09-04 16:09:24 UTC Build successful.
🟡 2024-09-04 16:09:59 UTC ydbd size 5.5 GiB changed* by +1.2 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: dfa0721 merge: deea3ff diff diff %
ydbd size 5 957 674 224 Bytes 5 958 908 912 Bytes +1.2 MiB +0.021%
ydbd stripped size 1 490 132 880 Bytes 1 490 257 008 Bytes +121.2 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 14:05:56 UTC Pre-commit check linux-x86_64-release-clang14 for deea3ff has started.
⚪ 2024-09-04 14:09:01 UTC ya make is running...
🟢 2024-09-04 14:26:52 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 14:06:17 UTC Pre-commit check linux-x86_64-relwithdebinfo for deea3ff has started.
⚪ 2024-09-04 14:09:14 UTC ya make is running...
🟡 2024-09-04 15:32:38 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
74265 60490 0 18 13736 21

⚪ 2024-09-04 15:40:38 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-09-04 15:48:38 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
49 (only retried tests) 19 0 15 0 15

⚪ 2024-09-04 15:48:46 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-09-04 15:57:17 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
35 (only retried tests) 6 0 15 0 14

🟢 2024-09-04 15:57:24 UTC Build successful.
🟡 2024-09-04 15:58:03 UTC ydbd size 8.3 GiB changed* by +1.6 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: dfa0721 merge: deea3ff diff diff %
ydbd size 8 922 565 592 Bytes 8 924 269 864 Bytes +1.6 MiB +0.019%
ydbd stripped size 481 974 600 Bytes 482 010 696 Bytes +35.2 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 5, 2024 •
edited
Loading

⚪ 2024-09-05 09:27:31 UTC Pre-commit check linux-x86_64-release-clang14 for 1ea2986 has started.
⚪ 2024-09-05 09:30:36 UTC ya make is running...
⚫ 2024-09-05 09:41:31 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 5, 2024 •
edited
Loading

⚪ 2024-09-05 09:29:45 UTC Pre-commit check linux-x86_64-release-asan for 1ea2986 has started.
⚪ 2024-09-05 09:32:51 UTC ya make is running...
⚫ 2024-09-05 09:41:31 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 5, 2024 •
edited
Loading

⚪ 2024-09-05 09:30:00 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1ea2986 has started.
⚪ 2024-09-05 09:33:01 UTC ya make is running...
⚫ 2024-09-05 09:41:33 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 5, 2024 •
edited
Loading

⚪ 2024-09-05 09:42:59 UTC Pre-commit check linux-x86_64-release-asan for 1da3e93 has started.
⚪ 2024-09-05 09:46:02 UTC ya make is running...
🔴 2024-09-05 11:51:58 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12104 11975 0 41 51 37

🟢 2024-09-05 11:53:04 UTC Build successful.
🟡 2024-09-05 11:53:40 UTC ydbd size 5.6 GiB changed* by +1.3 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 732bf12 merge: 1da3e93 diff diff %
ydbd size 6 019 637 112 Bytes 6 021 000 160 Bytes +1.3 MiB +0.023%
ydbd stripped size 1 507 074 288 Bytes 1 507 202 128 Bytes +124.8 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 5, 2024 •
edited
Loading

⚪ 2024-09-05 09:43:03 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1da3e93 has started.
⚪ 2024-09-05 09:46:12 UTC ya make is running...
🟡 2024-09-05 11:09:49 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
74290 60514 0 5 13751 20

⚪ 2024-09-05 11:17:02 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-09-05 11:24:47 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38 (only retried tests) 19 0 1 0 18

⚪ 2024-09-05 11:24:55 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-09-05 11:31:45 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
25 (only retried tests) 7 0 1 0 17

🟢 2024-09-05 11:31:52 UTC Build successful.
🟡 2024-09-05 11:32:28 UTC ydbd size 8.4 GiB changed* by +1.8 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 732bf12 merge: 1da3e93 diff diff %
ydbd size 9 001 104 776 Bytes 9 002 943 376 Bytes +1.8 MiB +0.020%
ydbd stripped size 486 652 968 Bytes 486 696 424 Bytes +42.4 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 5, 2024 •
edited
Loading

⚪ 2024-09-05 09:43:03 UTC Pre-commit check linux-x86_64-release-clang14 for 1da3e93 has started.
⚪ 2024-09-05 09:45:56 UTC ya make is running...
🟢 2024-09-05 09:54:58 UTC Build successful.

rvu1024
rvu1024 reviewed Sep 9, 2024
View reviewed changes
@github-actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 07:31:14 UTC Pre-commit check linux-x86_64-release-clang14 for ecb2de0 has started.
⚪ 2024-09-10 07:34:18 UTC ya make is running...
🟢 2024-09-10 07:50:51 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 07:31:14 UTC Pre-commit check linux-x86_64-relwithdebinfo for ecb2de0 has started.
⚪ 2024-09-10 07:34:27 UTC ya make is running...
🟡 2024-09-10 09:02:26 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
74351 60542 0 4 13779 26

⚪ 2024-09-10 09:09:51 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-10 09:18:59 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
42 (only retried tests) 23 0 0 0 19

🟢 2024-09-10 09:19:06 UTC Build successful.
🟡 2024-09-10 09:19:46 UTC ydbd size 8.4 GiB changed* by +1.7 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: dd142f6 merge: ecb2de0 diff diff %
ydbd size 9 024 182 440 Bytes 9 026 016 992 Bytes +1.7 MiB +0.020%
ydbd stripped size 487 703 400 Bytes 487 742 824 Bytes +38.5 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 10, 2024 •
edited
Loading

⚪ 2024-09-10 07:31:29 UTC Pre-commit check linux-x86_64-release-asan for ecb2de0 has started.
⚪ 2024-09-10 07:35:32 UTC ya make is running...
🔴 2024-09-10 09:34:14 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12122 11989 0 41 56 36

🟢 2024-09-10 09:35:17 UTC Build successful.
🟡 2024-09-10 09:37:59 UTC ydbd size 5.6 GiB changed* by +1.3 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: dd142f6 merge: ecb2de0 diff diff %
ydbd size 6 036 912 800 Bytes 6 038 276 080 Bytes +1.3 MiB +0.023%
ydbd stripped size 1 511 084 976 Bytes 1 511 212 816 Bytes +124.8 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@alephonea alephonea merged commit a7d2186 into ydb-platform:main Sep 10, 2024
@GrigoriyPA GrigoriyPA mentioned this pull request Sep 10, 2024
Merged
@shnikd shnikd mentioned this pull request Sep 11, 2024
Closed
@StekPerepolnen StekPerepolnen mentioned this pull request Sep 17, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rvu1024 rvu1024 rvu1024 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alephonea @rvu1024