Ydb workload log uniform pk distribution #17061
Conversation
|
🟢 |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
Протестировал ydb workload log run {insert, upsert, bulk_upsert} --date_from --date_to - работает ожидаемым образом. PK генерируются равномерно на промежутке при его указании с помощью этих опций |
|
⚪ |
|
⚪ |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
| #ifdef DT_CHR | ||
| #undef DT_CHR | ||
| #endif | ||
| #include <ydb/library/benchmarks/gen/tpch-dbgen/dss.h> |
There was a problem hiding this comment.
Без этого трюка падало с ошибкой - https://paste.yandex-team.ru/bea6a2c6-a803-4b8e-aa40-507d3624429c
There was a problem hiding this comment.
a85d2e9 - при добавлении вот этого диффа(#include <ydb/public/lib/ydb_cli/common/command.h> в хедере + дифф который добавляет методы и их реализации) падает с такой ошибкой. Почему-то редефайнится значение.
Без этого диффа ошибки нет.
| #include <ydb/public/sdk/cpp/include/ydb-cpp-sdk/client/value/value.h> | ||
| #include <ydb/library/accessor/accessor.h> | ||
| #include <library/cpp/getopt/last_getopt.h> | ||
| #include <ydb/public/lib/ydb_cli/common/command.h> |
There was a problem hiding this comment.
Это противоестественная зависимость. Должно быть наоборот.
There was a problem hiding this comment.
А почему? И что значит наоборот? что command.h инклюдит workload_query_generator.h?
There was a problem hiding this comment.
Потому что команды ссылаются на библиотеку, а не наоборот.
- Тут очень вероятно появление циклических зависимостей.
- абстрактная библиотека очень тяжелеет в смысле зависимостей.
…terval validation
Emgariko
force-pushed
the
ydb-workload-log-uniform-pk-distribution
branch
from
377455d to
5dddd6a
Compare
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
Emgariko
force-pushed
the
ydb-workload-log-uniform-pk-distribution
branch
from
61d254c to
65cb9c8
Compare
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
| } | ||
| virtual TString GetWorkloadName() const = 0; | ||
|
|
||
| virtual void Validate() const {}; |
There was a problem hiding this comment.
Валидировать надо с проверкой типа run-а - т.к например такая валидация не нужна для Select-а.
There was a problem hiding this comment.
Убедил, пусть будет Validate. Давай только сделаем его универсальнее, пусть принимает параметры const ECommandType commandType, int workloadType, чтобы его можно было из разных мест звать.
There was a problem hiding this comment.
Хорошо
Добавил эти параметры в Validate.
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
| PrepareForRun(config); | ||
| Params.DbPath = config.Database; | ||
| auto workloadGen = Params.CreateGenerator(); | ||
| Params.Validate(); |
There was a problem hiding this comment.
Не нужна отдельная публичная функция в интерфейсе. Можно прямо в CreateGenerator вставить проверку.
There was a problem hiding this comment.
CreateGenerator вызывается еще в том числе до того как проставились параметры.
Если туда добавлять проверку, валидация ошибочно срабатывает.
There was a problem hiding this comment.
Предлагаю такой вариант оставить, и вмержить хотя бы так.
А дальше сделать рефакторинг. Я пытался разобраться с CreateGenerator - быстро сделать не получилось. Пока пришел к такому решению.
There was a problem hiding this comment.
CreateGenerator вызывается прямо перед твоей проверкой, между ними ничего не происходит. Параметры там одни и те же в обоих местах. Сейчас это выглядит как какой-то костыль, зачем-то торчащий из интерфейса.
There was a problem hiding this comment.
Да, перед моей проверкой он вызывается.
А так же вызывается в других местах. И если добавлять в CreateGenerator валидацию - то при вызове из других мест эта валидация падает, т.к там еще не попарсились параметры.
There was a problem hiding this comment.
(gdb) break TLogWorkloadParams::CreateGenerator
Breakpoint 1 at 0xd28b18f: file /home/emgariko/project/ydb_fork_1/ydb/ydb/library/workload/log/log.cpp, line 489.
(gdb) run -e grpc://localhost:2136 -d /Root/test workload log run upsert --date-from 1 --date-to 2000 --timestamp_deviation 3
Starting program: /home/emgariko/project/ydb_fork_1/ydb/ydb/apps/ydb/ydb -e grpc://localhost:2136 -d /Root/test workload log run upsert --date-from 1 --date-to 2000 --timestamp_deviation 3
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Breakpoint 1, NYdbWorkload::NLog::TLogWorkloadParams::CreateGenerator (this=0x6f83f801040)
at /home/emgariko/project/ydb_fork_1/ydb/ydb/library/workload/log/log.cpp:489
489 return MakeHolder<TLogGenerator>(this);
(gdb) where
#0 NYdbWorkload::NLog::TLogWorkloadParams::CreateGenerator (this=0x6f83f801040)
at /home/emgariko/project/ydb_fork_1/ydb/ydb/library/workload/log/log.cpp:489
#1 0x0000000010ce9f7c in NYdb::NConsoleClient::TWorkloadCommandRoot::TWorkloadCommandRoot (this=0x6f83f787d10, key=...)
at /home/emgariko/project/ydb_fork_1/ydb/ydb/public/lib/ydb_cli/commands/ydb_workload.cpp:431
#2 0x0000000010ce43df in std::__y1::make_unique[abi:fe190000]<NYdb::NConsoleClient::TWorkloadCommandRoot, char const*>(char const*&&) (
__args=@0x7fffffffda48: 0x6f83f6524a9 "log") at /home/emgariko/project/ydb_fork_1/ydb/contrib/libs/cxxsupp/libcxx/include/__memory/unique_ptr.h:621
#3 0x0000000010ce3f75 in NYdb::NConsoleClient::TCommandWorkload::TCommandWorkload (this=0x6f83f7863b0)
at /home/emgariko/project/ydb_fork_1/ydb/ydb/public/lib/ydb_cli/commands/ydb_workload.cpp:52
#4 0x000000000d575f90 in std::__y1::make_unique[abi:fe190000]<NYdb::NConsoleClient::TCommandWorkload>() ()
at /home/emgariko/project/ydb_fork_1/ydb/contrib/libs/cxxsupp/libcxx/include/__memory/unique_ptr.h:621
#5 0x000000000d5748da in NYdb::NConsoleClient::TClientCommandRootCommon::TClientCommandRootCommon (this=0x6f83f6c0780, name=..., settings=...)
at /home/emgariko/project/ydb_fork_1/ydb/ydb/public/lib/ydb_cli/commands/ydb_root_common.cpp:56
#6 0x000000000d56c339 in NYdb::NConsoleClient::TClientCommandRoot::TClientCommandRoot (this=0x6f83f6c0780, name=..., settings=...)
at /-S/ydb/apps/ydb/commands/ydb_root.cpp:16
#7 0x000000000d56c499 in NYdb::NConsoleClient::TYdbClientCommandRoot::TYdbClientCommandRoot (this=0x6f83f6c0780, name=..., settings=...)
at /-S/ydb/apps/ydb/commands/ydb_root.cpp:62
#8 0x000000000d56d364 in MakeHolder<NYdb::NConsoleClient::TYdbClientCommandRoot, std::__y1::basic_string<char, std::__y1::char_traits<char>, std::__y1::allocator<char> >, NYdb::NConsoleClient::TClientSettings&> (args=..., args=...) at /-S/util/generic/ptr.h:370
#9 0x000000000d56cf9d in NYdb::NConsoleClient::NewYdbClient (argc=15, argv=0x7fffffffe388) at /-S/ydb/apps/ydb/commands/ydb_root.cpp:121
#10 0x000000000b302d32 in main (argc=15, argv=0x7fffffffe388) at /-S/ydb/apps/ydb/main.cpp:14
(gdb)
здесь параметры еще не попарсились
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
Co-authored-by: Oleg Geller <iddqd@ydb.tech>
Co-authored-by: Oleg Geller <iddqd@ydb.tech>
liruoko
added
the
release-scope-25-3
4 participants
Changelog entry
Add of date range parameters (--date-to, --date-from to support uniform PK distribution) for ydb workload log run operations including bulk_upsert, insert, and upsert
Changelog category
Description for reviewers
Added
date_from&date_toparams to ydb workload log to support uniformly distributed timestamp generation.Добавил эти параметры в workload run {bulk_upsert, insert, upsert}. Параметры валидируются. --timestamp_deviation и --date-from, --date-to - взаимоисключающие. Нужно передавать либо --timestamp_deviation, либо --date-from, --date-to. Если не указывать ничего из этих опций - используется timestamp_deviation = 0