NActors::TVerifyFormattedRecordWriter::~TVerifyFormattedRecordWriter()

[SloNN]

Version main from 29.05

NActors::TVerifyFormattedRecordWriter::~TVerifyFormattedRecordWriter() 

Additional info available at
https://coredumps.yandex-team.ru/core?core_hash=642733009321404&itype=kikimr

It seems that crash is caused by delete statement, delete from table

[SloNN]

Trying to clear up the table once again

error
overload data error
error
Kikimr cluster or one of its subsystems is overloaded. Tablet 72075186224073756 is overloaded. Table `/Root/XXX`.
Code: 2006
error
overload data error

The cluster seems to be free, node usage is about 2%

[ivanmorozov333]

текущий статус
delete from table - не достаточно для воспроизведения. в реальности - что-то было по интервалам
корки к проблеме отношения не имеют - они с другого кластера
на кластере "проблемном" корок и VERIFY в логах нет с 3-го числа, когда была одна корка на записи - ее посмотрю отдельно. она с удалением не связана и она разовая (на смерть базы не похожа)
как умерла база (если умерла, а была не разовая корка) - не понятно
также, хорошо бы приложить сюда ссылку на кластер и графики - вторая проблема с overload, судя по всему, это, связано с количеством одновременных запросов с KQP - видимо, нужно будет конфиг поправить.

[ivanmorozov333]

единственное что нашел интересного - корки вида

#0 0x4300b in ?? () at libc.so.6
#1 0x21 in NPrivate::Panic(NPrivate::TStaticBuf const&, int, char const*, char const*, char const*, ...) () from /home/slonn/ydb_org/ydb/util/system/yassert.cpp:55
#2 0x25 in NActors::TVerifyFormattedRecordWriter::~TVerifyFormattedRecordWriter() () from /home/slonn/ydb_org/ydb/ydb/library/actors/core/log.cpp:800
#3 0x5 in NKikimr::NOlap::TGranuleMeta::CommitImmediateOnExecute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NKikimr::NOlap::TSnapshot const&, NKikimr::NOlap::TPortionDataAccessor const&, unsigned long) const () from /home/slonn/ydb_org/ydb/ydb/core/tx/columnshard/engines/storage/granule/granule.cpp:321
#4 0xf in NKikimr::NColumnShard::TTxBlobsWritingFinished::DoExecute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) () from /home/slonn/ydb_org/ydb/ydb/core/tx/columnshard/blobs_action/transaction/tx_blobs_written.cpp:48
#5 0xf in NKikimr::NColumnShard::TExtendedTransactionBase::Execute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) () from /home/slonn/ydb_org/ydb/ydb/core/tx/columnshard/tablet/ext_tx_base.cpp:9
#6 0x6 in NKikimr::NTabletFlatExecutor::TExecutor::ExecuteTransaction(NKikimr::NTabletFlatExecutor::TSeat*) () from /home/slonn/ydb_org/ydb/ydb/core/tablet_flat/flat_executor.cpp:1975
#7 0xb in NKikimr::NTabletFlatExecutor::TExecutor::DoExecute(TAutoPtr<NKikimr::NTabletFlatExecutor::ITransaction, TDelete>, NKikimr::NTabletFlatExecutor::TExecutor::ETxMode) () from /home/slonn/ydb_org/ydb/ydb/core/tablet_flat/flat_executor.cpp:1879
#8 0x7 in NKikimr::NTabletFlatExecutor::TExecutor::Execute(TAutoPtr<NKikimr::NTabletFlatExecutor::ITransaction, TDelete>, NActors::TActorContext const&) () from /home/slonn/ydb_org/ydb/ydb/core/tablet_flat/flat_executor.cpp:1893
#9 0x6 in NKikimr::NTabletFlatExecutor::TTabletExecutedFlat::Execute(TAutoPtr<NKikimr::NTabletFlatExecutor::ITransaction, TDelete>, NActors::TActorContext const&) () from /home/slonn/ydb_org/ydb/ydb/core/tablet_flat/tablet_flat_executed.cpp:62
#10 0x5 in NKikimr::NColumnShard::TColumnShard::Handle(TAutoPtr<NActors::TEventHandleNKikimr::NColumnShard::NPrivateEvents::NWrite::TEvWritePortionResult, TDelete>&, NActors::TActorContext const&) () from /home/slonn/ydb_org/ydb/ydb/core/tx/columnshard/columnshard__write.cpp:118
#11 0xc in NKikimr::NColumnShard::TColumnShard::StateWork(TAutoPtr<NActors::IEventHandle, TDelete>&) () from /home/slonn/ydb_org/ydb/ydb/core/tx/columnshard/columnshard_impl.h:428
#12 0x5 in NActors::IActor::Receive(TAutoPtr<NActors::IEventHandle, TDelete>&) () from /home/slonn/ydb_org/ydb/ydb/library/actors/core/actor.cpp:280
#13 0xf in NActors::TExecutorThread::Execute(NActors::TMailbox*, bool) () from /home/slonn/ydb_org/ydb/ydb/library/actors/core/executor_thread.cpp:269
#14 0x10 in NActors::TExecutorThread::ProcessExecutorPool()::$_0::operator()(NActors::TMailbox*, bool) const () from /home/slonn/ydb_org/ydb/ydb/library/actors/core/executor_thread.cpp:460
#15 0x11 in NActors::TExecutorThread::ProcessExecutorPool() () from /home/slonn/ydb_org/ydb/ydb/library/actors/core/executor_thread.cpp:512
#16 0x8 in NActors::TExecutorThread::ThreadProc() () from /home/slonn/ydb_org/ydb/ydb/library/actors/core/executor_thread.cpp:538
#17 0x2 in (anonymous namespace)::TPosixThread::ThreadProxy(void*) () from /home/slonn/ydb_org/ydb/util/system/thread.cpp:244
#18 0x8609 in ?? () at libpthread.so.0
#19 0x11f353 in ?? () at libc.so.6

ими занимаюсь
но они не приводят к смерти базы.

[naspirato]

На версии main 6d02f27 ( от24.06)
запускаю созданы таблицы через workload sipmle_queue
получаю бесконечные

Jun 24 12:12:34 ydb-sas-testing-0007 kikimr_31003[826319]: VERIFY failed (2025-06-24T12:12:34.101839+0300): tablet_id=72075186224038424;event=initialize_shard;load_stage_name=EXECUTE:composite_init;load_stage_name=EXECUTE:composite_init/tables_manager;verification=MaxInternalPathId >= GetInitialMaxInternalPathId(TabletId);fline=tables_manager.cpp:83;
Jun 24 12:12:34 ydb-sas-testing-0007 kikimr_31003[826319]:   ydb/library/actors/core/log.cpp:800
Jun 24 12:12:34 ydb-sas-testing-0007 kikimr_31003[826319]:   ~TVerifyFormattedRecordWriter(): requirement false failed
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 0. /-S/util/system/yassert.cpp:83: NPrivate::InternalPanicImpl(int, char const*, char const*, int, int, int, TBasicStringBuf<char, std::__y1::char_traits<char>>, char const*, unsigned long) @ 0xA367325
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 1. /-S/util/system/yassert.cpp:55: NPrivate::Panic(NPrivate::TStaticBuf const&, int, char const*, char const*, char const*, ...) @ 0xA36013C
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 2. /place/sandbox-data/tasks/4/5/3167558954/ydb/ydb/library/actors/core/log.cpp:800: NActors::TVerifyFormattedRecordWriter::~TVerifyFormattedRecordWriter() @ 0xB24C203
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 3. /-S/ydb/core/tx/columnshard/tables_manager.cpp:83: NKikimr::NColumnShard::TTablesManager::InitFromDB(NKikimr::NIceDb::TNiceDb&) @ 0x14B35CEF
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 4. /-S/ydb/core/tx/columnshard/loading/stages.cpp:159: NKikimr::NColumnShard::NLoading::TTablesManagerInitializer::DoExecute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x14AA9BF1
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 5. /-S/ydb/core/tx/columnshard/tx_reader/abstract.cpp:28: NKikimr::ITxReader::Execute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x149386D8
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 6. /place/sandbox-data/tasks/7/8/3168475887/ydb/ydb/core/tx/columnshard/tx_reader/composite.h:17: NKikimr::TTxCompositeReader::DoExecute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x13E1B88F
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 7. /-S/ydb/core/tx/columnshard/tx_reader/abstract.cpp:28: NKikimr::ITxReader::Execute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x149386D8
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 8. /-S/ydb/core/tx/columnshard/columnshard__init.cpp:82: NKikimr::NColumnShard::TTxInit::Execute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x14A81F67
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 9. /place/sandbox-data/tasks/7/8/3168475887/ydb/ydb/core/tablet_flat/flat_executor.cpp:1976: NKikimr::NTabletFlatExecutor::TExecutor::ExecuteTransaction(NKikimr::NTabletFlatExecutor::TSeat*) @ 0xCD53588
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 10. /place/sandbox-data/tasks/7/8/3168475887/ydb/ydb/core/tablet_flat/flat_executor.cpp:4201: NKikimr::NTabletFlatExecutor::TExecutor::StateWork(TAutoPtr<NActors::IEventHandle, TDelete>&) @ 0xCD3EAB6
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 11. /-S/ydb/library/actors/core/actor.cpp:280: NActors::IActor::Receive(TAutoPtr<NActors::IEventHandle, TDelete>&) @ 0xB204C24
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 12. /-S/ydb/library/actors/core/executor_thread.cpp:269: NActors::TExecutorThread::Execute(NActors::TMailbox*, bool) @ 0xB23DB30
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 13. /-S/ydb/library/actors/core/executor_thread.cpp:460: NActors::TExecutorThread::ProcessExecutorPool()::$_0::operator()(NActors::TMailbox*, bool) const @ 0xB241631
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 14. /-S/ydb/library/actors/core/executor_thread.cpp:512: NActors::TExecutorThread::ProcessExecutorPool() @ 0xB241280
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 15. /-S/ydb/library/actors/core/executor_thread.cpp:538: NActors::TExecutorThread::ThreadProc() @ 0xB241E2A
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 16. /-S/util/system/thread.cpp:245: (anonymous namespace)::TPosixThread::ThreadProxy(void*) @ 0xA36817C
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 17. ??:0: ?? @ 0x7FBD2CEE8608

насыпало уже 73000 таких https://nda.ya.ru/t/cmrC1D9x7FiKte