Skip to content

YQ-3561 add features for FQ-run #14826

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

GrigoriyPA
Copy link
Collaborator

@GrigoriyPA GrigoriyPA commented Feb 20, 2025

Changelog entry

Added features for FQ-run

Changelog category

  • Improvement

Description for reviewers

@GrigoriyPA GrigoriyPA requested a review from a team as a code owner February 20, 2025 08:40
Copy link

github-actions bot commented Feb 20, 2025

🟢 2025-02-20 08:58:00 UTC The validation of the Pull Request description is successful.

Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 08:42:06 UTC Pre-commit check linux-x86_64-relwithdebinfo for e30a7be has started.
2025-02-20 08:42:18 UTC Artifacts will be uploaded here
2025-02-20 08:44:45 UTC ya make is running...
🟢 2025-02-20 08:54:06 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9342 6921 0 0 2421 0

🟢 2025-02-20 08:55:01 UTC Build successful.
🟢 2025-02-20 08:55:10 UTC ydbd size 2.1 GiB changed* by +56.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 24d79ff merge: e30a7be diff diff %
ydbd size 2 239 349 000 Bytes 2 239 406 296 Bytes +56.0 KiB +0.003%
ydbd stripped size 474 513 656 Bytes 474 530 296 Bytes +16.2 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 08:45:32 UTC Pre-commit check linux-x86_64-release-asan for e30a7be has started.
2025-02-20 08:46:34 UTC Artifacts will be uploaded here
2025-02-20 08:48:57 UTC ya make is running...
🟢 2025-02-20 08:54:05 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
250 249 0 0 1 0

🟢 2025-02-20 08:54:12 UTC Build successful.
🟢 2025-02-20 08:54:24 UTC ydbd size 3.6 GiB changed* by +44.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 24d79ff merge: e30a7be diff diff %
ydbd size 3 895 444 968 Bytes 3 895 490 568 Bytes +44.5 KiB +0.001%
ydbd stripped size 1 363 998 144 Bytes 1 364 013 344 Bytes +14.8 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@GrigoriyPA GrigoriyPA force-pushed the YQ-3561-add-features-for-FQ-run branch from a66bbe9 to 887af1f Compare February 20, 2025 10:27
Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 10:31:56 UTC Pre-commit check linux-x86_64-release-asan for 38d3331 has started.
2025-02-20 10:32:08 UTC Artifacts will be uploaded here
2025-02-20 10:34:37 UTC ya make is running...
🟢 2025-02-20 10:39:56 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
250 249 0 0 1 0

🟢 2025-02-20 10:40:03 UTC Build successful.
🟢 2025-02-20 10:40:16 UTC ydbd size 3.6 GiB changed* by -96 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash main: 4d7998d merge: 38d3331 diff diff %
ydbd size 3 895 490 664 Bytes 3 895 490 568 Bytes -96 Bytes -0.000%
ydbd stripped size 1 364 013 408 Bytes 1 364 013 344 Bytes -64 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 10:32:13 UTC Pre-commit check linux-x86_64-relwithdebinfo for 38d3331 has started.
2025-02-20 10:33:30 UTC Artifacts will be uploaded here
2025-02-20 10:35:57 UTC ya make is running...
🟡 2025-02-20 10:46:29 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9342 6919 0 2 2421 0

2025-02-20 10:47:21 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-02-20 10:48:11 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
2 (only retried tests) 2 0 0 0 0

🟢 2025-02-20 10:48:18 UTC Build successful.
🟢 2025-02-20 10:48:28 UTC ydbd size 2.1 GiB changed* by 0 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash main: 4d7998d merge: 38d3331 diff diff %
ydbd size 2 239 406 296 Bytes 2 239 406 296 Bytes 0 Bytes 0.000%
ydbd stripped size 474 530 296 Bytes 474 530 296 Bytes 0 Bytes 0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@GrigoriyPA GrigoriyPA requested a review from yumkam February 20, 2025 10:45
Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 10:54:29 UTC Pre-commit check linux-x86_64-release-asan for f274f92 has started.
2025-02-20 10:54:33 UTC Artifacts will be uploaded here
2025-02-20 10:57:01 UTC ya make is running...
🟢 2025-02-20 11:02:16 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
250 249 0 0 1 0

🟢 2025-02-20 11:02:23 UTC Build successful.
🟢 2025-02-20 11:02:36 UTC ydbd size 3.6 GiB changed* by -96 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash main: 4d7998d merge: f274f92 diff diff %
ydbd size 3 895 490 664 Bytes 3 895 490 568 Bytes -96 Bytes -0.000%
ydbd stripped size 1 364 013 408 Bytes 1 364 013 344 Bytes -64 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 10:55:37 UTC Pre-commit check linux-x86_64-relwithdebinfo for f274f92 has started.
2025-02-20 10:55:52 UTC Artifacts will be uploaded here
2025-02-20 10:58:19 UTC ya make is running...
🟢 2025-02-20 11:07:41 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9342 6921 0 0 2421 0

🟢 2025-02-20 11:08:35 UTC Build successful.
🟢 2025-02-20 11:08:44 UTC ydbd size 2.1 GiB changed* by 0 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash main: 4d7998d merge: f274f92 diff diff %
ydbd size 2 239 406 296 Bytes 2 239 406 296 Bytes 0 Bytes 0.000%
ydbd stripped size 474 530 296 Bytes 474 530 296 Bytes 0 Bytes 0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation


fqrun_pid=$(pgrep -u $USER fqrun)

sudo perf record -F 50 --call-graph dwarf -g --proc-map-timeout=10000 --pid $fqrun_pid -v -o profdata -- sleep ${1:-'30'}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

А нужно ли тут sudo? (возможно, в некоторых случаях -- да, но в некоторых случаях -- тот же пользователь, отсутствие параноидальных настроек с ptrace -- работает и без него). (Насколько я понимаю, это просто скопировано из kqprun, так что те же вопросы можно адресовать и к тому скрипту)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Иногда без sudo у меня perf ругался, оставил чтобы работало скорее всегда (если прав на sudo нет, что редко, можно в ручную убрать).

Это ведь не засоряет лишними файлами fs для супер пользователя?

Copy link
Collaborator

@yumkam yumkam Feb 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perf record вроде ничего трогать не должен, и не сильно страшен; только вот он сохраняет perf.data с 0600 root:root, что побуждает и perf script запускать с sudo, а он делает всякое-страшное (и насчёт загрязнения не уверен)
В общем, чем с меньшими правами всякое запускается -- тем меньше шансов случайно отстрелить.
Я бы скорее вставил в скрипт что-то вроде $SUDO perf [...] и оставил выставление SUDO=sudo ./flame-graph.sh на пользователя (если у него без sudo не работает)
Но можно оставить и как есть

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Хорошее замечание, поправлю в следующем PR

uzhastik
uzhastik previously approved these changes Feb 20, 2025
@@ -20,6 +20,7 @@ PEERDIR(
ydb/library/security
ydb/library/yql/providers/pq/provider
ydb/tests/tools/kqprun/runlib
yql/essentials/minikql
)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

рекомендую сортировать списки. util лучше убери, он всегда неявно подключается

Copy link
Collaborator Author

@GrigoriyPA GrigoriyPA Feb 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Я стараюсь везде сортировать список PEERDIR по возрастанию, где тут нарушился порядок? С ходу не видно

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ну вот ты добавил строчку не по алфавиту

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

util тоже не по алфавиту был

Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 11:42:46 UTC Pre-commit check linux-x86_64-release-asan for 142b58f has started.
2025-02-20 11:42:58 UTC Artifacts will be uploaded here
2025-02-20 11:45:17 UTC ya make is running...
2025-02-20 11:46:47 UTC Check cancelled

Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 11:43:34 UTC Pre-commit check linux-x86_64-relwithdebinfo for 142b58f has started.
2025-02-20 11:43:48 UTC Artifacts will be uploaded here
2025-02-20 11:46:18 UTC ya make is running...
2025-02-20 11:46:47 UTC Check cancelled

Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 11:48:57 UTC Pre-commit check linux-x86_64-relwithdebinfo for 81bb3b9 has started.
2025-02-20 11:50:20 UTC Artifacts will be uploaded here
2025-02-20 11:52:44 UTC ya make is running...
🟢 2025-02-20 12:21:07 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9342 6921 0 0 2421 0

🟢 2025-02-20 12:21:56 UTC Build successful.
🟢 2025-02-20 12:22:07 UTC ydbd size 2.1 GiB changed* by +4.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 4d7998d merge: 81bb3b9 diff diff %
ydbd size 2 239 406 296 Bytes 2 239 411 192 Bytes +4.8 KiB +0.000%
ydbd stripped size 474 530 296 Bytes 474 529 528 Bytes -768 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 11:51:06 UTC Pre-commit check linux-x86_64-release-asan for 81bb3b9 has started.
2025-02-20 12:08:13 UTC Artifacts will be uploaded here
2025-02-20 12:10:39 UTC ya make is running...
🟢 2025-02-20 12:37:53 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
250 249 0 0 1 0

🟢 2025-02-20 12:38:00 UTC Build successful.
🟢 2025-02-20 12:38:13 UTC ydbd size 3.6 GiB changed* by +14.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 4d7998d merge: 81bb3b9 diff diff %
ydbd size 3 895 490 664 Bytes 3 895 505 328 Bytes +14.3 KiB +0.000%
ydbd stripped size 1 364 013 408 Bytes 1 364 021 408 Bytes +7.8 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@GrigoriyPA GrigoriyPA merged commit 3ee5135 into ydb-platform:main Feb 20, 2025
12 checks passed
@GrigoriyPA GrigoriyPA deleted the YQ-3561-add-features-for-FQ-run branch February 20, 2025 12:44
GrigoriyPA added a commit to GrigoriyPA/ydb that referenced this pull request Feb 27, 2025
lberserq pushed a commit to lberserq/ydb that referenced this pull request Mar 3, 2025
blinkov pushed a commit that referenced this pull request Mar 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants