NActors::TVerifyFormattedRecordWriter::~TVerifyFormattedRecordWriter()

[SloNN]

Version main from 29.05

NActors::TVerifyFormattedRecordWriter::~TVerifyFormattedRecordWriter() 

Additional info available at
https://coredumps.yandex-team.ru/core?core_hash=642733009321404&itype=kikimr

It seems that crash is caused by delete statement, delete from table

[SloNN]

Trying to clear up the table once again

error
overload data error
error
Kikimr cluster or one of its subsystems is overloaded. Tablet 72075186224073756 is overloaded. Table `/Root/XXX`.
Code: 2006
error
overload data error

The cluster seems to be free, node usage is about 2%

[ivanmorozov333]

текущий статус
delete from table - не достаточно для воспроизведения. в реальности - что-то было по интервалам
корки к проблеме отношения не имеют - они с другого кластера
на кластере "проблемном" корок и VERIFY в логах нет с 3-го числа, когда была одна корка на записи - ее посмотрю отдельно. она с удалением не связана и она разовая (на смерть базы не похожа)
как умерла база (если умерла, а была не разовая корка) - не понятно
также, хорошо бы приложить сюда ссылку на кластер и графики - вторая проблема с overload, судя по всему, это, связано с количеством одновременных запросов с KQP - видимо, нужно будет конфиг поправить.

[ivanmorozov333]

единственное что нашел интересного - корки вида

#0 0x4300b in ?? () at libc.so.6
#1 0x21 in NPrivate::Panic(NPrivate::TStaticBuf const&, int, char const*, char const*, char const*, ...) () from /home/slonn/ydb_org/ydb/util/system/yassert.cpp:55
#2 0x25 in NActors::TVerifyFormattedRecordWriter::~TVerifyFormattedRecordWriter() () from /home/slonn/ydb_org/ydb/ydb/library/actors/core/log.cpp:800
#3 0x5 in NKikimr::NOlap::TGranuleMeta::CommitImmediateOnExecute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NKikimr::NOlap::TSnapshot const&, NKikimr::NOlap::TPortionDataAccessor const&, unsigned long) const () from /home/slonn/ydb_org/ydb/ydb/core/tx/columnshard/engines/storage/granule/granule.cpp:321
#4 0xf in NKikimr::NColumnShard::TTxBlobsWritingFinished::DoExecute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) () from /home/slonn/ydb_org/ydb/ydb/core/tx/columnshard/blobs_action/transaction/tx_blobs_written.cpp:48
#5 0xf in NKikimr::NColumnShard::TExtendedTransactionBase::Execute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) () from /home/slonn/ydb_org/ydb/ydb/core/tx/columnshard/tablet/ext_tx_base.cpp:9
#6 0x6 in NKikimr::NTabletFlatExecutor::TExecutor::ExecuteTransaction(NKikimr::NTabletFlatExecutor::TSeat*) () from /home/slonn/ydb_org/ydb/ydb/core/tablet_flat/flat_executor.cpp:1975
#7 0xb in NKikimr::NTabletFlatExecutor::TExecutor::DoExecute(TAutoPtr<NKikimr::NTabletFlatExecutor::ITransaction, TDelete>, NKikimr::NTabletFlatExecutor::TExecutor::ETxMode) () from /home/slonn/ydb_org/ydb/ydb/core/tablet_flat/flat_executor.cpp:1879
#8 0x7 in NKikimr::NTabletFlatExecutor::TExecutor::Execute(TAutoPtr<NKikimr::NTabletFlatExecutor::ITransaction, TDelete>, NActors::TActorContext const&) () from /home/slonn/ydb_org/ydb/ydb/core/tablet_flat/flat_executor.cpp:1893
#9 0x6 in NKikimr::NTabletFlatExecutor::TTabletExecutedFlat::Execute(TAutoPtr<NKikimr::NTabletFlatExecutor::ITransaction, TDelete>, NActors::TActorContext const&) () from /home/slonn/ydb_org/ydb/ydb/core/tablet_flat/tablet_flat_executed.cpp:62
#10 0x5 in NKikimr::NColumnShard::TColumnShard::Handle(TAutoPtr<NActors::TEventHandleNKikimr::NColumnShard::NPrivateEvents::NWrite::TEvWritePortionResult, TDelete>&, NActors::TActorContext const&) () from /home/slonn/ydb_org/ydb/ydb/core/tx/columnshard/columnshard__write.cpp:118
#11 0xc in NKikimr::NColumnShard::TColumnShard::StateWork(TAutoPtr<NActors::IEventHandle, TDelete>&) () from /home/slonn/ydb_org/ydb/ydb/core/tx/columnshard/columnshard_impl.h:428
#12 0x5 in NActors::IActor::Receive(TAutoPtr<NActors::IEventHandle, TDelete>&) () from /home/slonn/ydb_org/ydb/ydb/library/actors/core/actor.cpp:280
#13 0xf in NActors::TExecutorThread::Execute(NActors::TMailbox*, bool) () from /home/slonn/ydb_org/ydb/ydb/library/actors/core/executor_thread.cpp:269
#14 0x10 in NActors::TExecutorThread::ProcessExecutorPool()::$_0::operator()(NActors::TMailbox*, bool) const () from /home/slonn/ydb_org/ydb/ydb/library/actors/core/executor_thread.cpp:460
#15 0x11 in NActors::TExecutorThread::ProcessExecutorPool() () from /home/slonn/ydb_org/ydb/ydb/library/actors/core/executor_thread.cpp:512
#16 0x8 in NActors::TExecutorThread::ThreadProc() () from /home/slonn/ydb_org/ydb/ydb/library/actors/core/executor_thread.cpp:538
#17 0x2 in (anonymous namespace)::TPosixThread::ThreadProxy(void*) () from /home/slonn/ydb_org/ydb/util/system/thread.cpp:244
#18 0x8609 in ?? () at libpthread.so.0
#19 0x11f353 in ?? () at libc.so.6

ими занимаюсь
но они не приводят к смерти базы.

[naspirato]

На версии main 6d02f27 ( от24.06)
запускаю созданы таблицы через workload sipmle_queue
получаю бесконечные

Jun 24 12:12:34 ydb-sas-testing-0007 kikimr_31003[826319]: VERIFY failed (2025-06-24T12:12:34.101839+0300): tablet_id=72075186224038424;event=initialize_shard;load_stage_name=EXECUTE:composite_init;load_stage_name=EXECUTE:composite_init/tables_manager;verification=MaxInternalPathId >= GetInitialMaxInternalPathId(TabletId);fline=tables_manager.cpp:83;
Jun 24 12:12:34 ydb-sas-testing-0007 kikimr_31003[826319]:   ydb/library/actors/core/log.cpp:800
Jun 24 12:12:34 ydb-sas-testing-0007 kikimr_31003[826319]:   ~TVerifyFormattedRecordWriter(): requirement false failed
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 0. /-S/util/system/yassert.cpp:83: NPrivate::InternalPanicImpl(int, char const*, char const*, int, int, int, TBasicStringBuf<char, std::__y1::char_traits<char>>, char const*, unsigned long) @ 0xA367325
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 1. /-S/util/system/yassert.cpp:55: NPrivate::Panic(NPrivate::TStaticBuf const&, int, char const*, char const*, char const*, ...) @ 0xA36013C
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 2. /place/sandbox-data/tasks/4/5/3167558954/ydb/ydb/library/actors/core/log.cpp:800: NActors::TVerifyFormattedRecordWriter::~TVerifyFormattedRecordWriter() @ 0xB24C203
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 3. /-S/ydb/core/tx/columnshard/tables_manager.cpp:83: NKikimr::NColumnShard::TTablesManager::InitFromDB(NKikimr::NIceDb::TNiceDb&) @ 0x14B35CEF
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 4. /-S/ydb/core/tx/columnshard/loading/stages.cpp:159: NKikimr::NColumnShard::NLoading::TTablesManagerInitializer::DoExecute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x14AA9BF1
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 5. /-S/ydb/core/tx/columnshard/tx_reader/abstract.cpp:28: NKikimr::ITxReader::Execute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x149386D8
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 6. /place/sandbox-data/tasks/7/8/3168475887/ydb/ydb/core/tx/columnshard/tx_reader/composite.h:17: NKikimr::TTxCompositeReader::DoExecute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x13E1B88F
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 7. /-S/ydb/core/tx/columnshard/tx_reader/abstract.cpp:28: NKikimr::ITxReader::Execute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x149386D8
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 8. /-S/ydb/core/tx/columnshard/columnshard__init.cpp:82: NKikimr::NColumnShard::TTxInit::Execute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x14A81F67
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 9. /place/sandbox-data/tasks/7/8/3168475887/ydb/ydb/core/tablet_flat/flat_executor.cpp:1976: NKikimr::NTabletFlatExecutor::TExecutor::ExecuteTransaction(NKikimr::NTabletFlatExecutor::TSeat*) @ 0xCD53588
Jun 24 12:12:35 ydb-sas-testing-0007 kikimr_31003[826319]: 10. /place/sandbox-data/tasks/7/8/3168475887/ydb/ydb/core/tablet_flat/flat_executor.cpp:4201: NKikimr::NTabletFlatExecutor::TExecutor::StateWork(TAutoPtr<NActors::IEventHandle, TDelete>&) @ 0xCD3EAB6
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 11. /-S/ydb/library/actors/core/actor.cpp:280: NActors::IActor::Receive(TAutoPtr<NActors::IEventHandle, TDelete>&) @ 0xB204C24
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 12. /-S/ydb/library/actors/core/executor_thread.cpp:269: NActors::TExecutorThread::Execute(NActors::TMailbox*, bool) @ 0xB23DB30
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 13. /-S/ydb/library/actors/core/executor_thread.cpp:460: NActors::TExecutorThread::ProcessExecutorPool()::$_0::operator()(NActors::TMailbox*, bool) const @ 0xB241631
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 14. /-S/ydb/library/actors/core/executor_thread.cpp:512: NActors::TExecutorThread::ProcessExecutorPool() @ 0xB241280
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 15. /-S/ydb/library/actors/core/executor_thread.cpp:538: NActors::TExecutorThread::ThreadProc() @ 0xB241E2A
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 16. /-S/util/system/thread.cpp:245: (anonymous namespace)::TPosixThread::ThreadProxy(void*) @ 0xA36817C
Jun 24 12:12:36 ydb-sas-testing-0007 kikimr_31003[826319]: 17. ??:0: ?? @ 0x7FBD2CEE8608

насыпало уже 73000 таких https://nda.ya.ru/t/cmrC1D9x7FiKte

[Vladilen]

Есть ещё такие похожие выстрелы на текущем main (от 04.07)

0?? () libc.so.6
1NPrivate::Panic(NPrivate::TStaticBuf const&, int, char const*, char const*, char const*, ...) () [util/system/yassert.cpp](https://a.yandex-team.ru/arc/trunk/arcadia/util/system/yassert.cpp?#L55) +55
2NActors::TVerifyFormattedRecordWriter::~TVerifyFormattedRecordWriter() () /-S/ydb/library/actors/core/log.cpp +800
3NKikimr::NArrow::NSSA::NGraph::NExecution::TCompiledGraph::TCompiledGraph(NKikimr::NArrow::NSSA::NGraph::NOptimization::TGraph const&, NKikimr::NArrow::NSSA::IColumnResolver const&) () /-S/ydb/core/formats/arrow/program/graph_execute.cpp +148
4NKikimr::NArrow::NSSA::NGraph::NOptimization::TGraph::Compile() () [contrib/libs/cxxsupp/libcxx/include/__memory/allocator.h](https://a.yandex-team.ru/arc/trunk/arcadia/contrib/libs/cxxsupp/libcxx/include/__memory/allocator.h?#L151) +151
5NKikimr::NArrow::NSSA::NGraph::NOptimization::TGraph::TBuilder::Finish() () /-S/ydb/core/formats/arrow/program/graph_optimization.h +240
6NKikimr::NOlap::TProgramContainer::ParseProgram(NKikimr::NArrow::NSSA::IColumnResolver const&, NKikimrSSA::TProgram const&) () /-S/ydb/core/tx/program/builder.h +61
7NKikimr::NOlap::TProgramContainer::Init(NKikimr::NArrow::NSSA::IColumnResolver const&, NKikimrSSA::TProgram const&) () /-S/ydb/core/tx/program/program.cpp +47
8NKikimr::NOlap::TProgramContainer::Init(NKikimr::NArrow::NSSA::IColumnResolver const&, NKikimrSSA::TOlapProgram const&) () /-S/ydb/core/tx/program/program.cpp +71
9NKikimr::NOlap::TProgramContainer::Init(NKikimr::NArrow::NSSA::IColumnResolver const&, NKikimrSchemeOp::EOlapProgramType, TBasicString<char, std::__y1::char_traits<char>>) () /-S/ydb/core/tx/program/program.cpp +96
10NKikimr::NOlap::NReader::IScannerConstructor::ParseProgram(NKikimr::NOlap::TVersionedIndex const*, NKikimrSchemeOp::EOlapProgramType, TBasicString<char, std::__y1::char_traits<char>> const&, NKikimr::NOlap::NReader::TReadDescription&, NKikimr::NArrow::NSSA::IColumnResolver const&) const () /-S/ydb/core/tx/columnshard/engines/reader/abstract/constructor.cpp +24
11NKikimr::NOlap::NReader::NSimple::TIndexScannerConstructor::ParseProgram(NKikimr::NOlap::TVersionedIndex const*, NKikimrTxDataShard::TEvKqpScan const&, NKikimr::NOlap::NReader::TReadDescription&) const () /-S/ydb/core/tx/columnshard/engines/reader/simple_reader/constructor/constructor.cpp +14
12NKikimr::NOlap::NReader::TTxScan::Complete(NActors::TActorContext const&) () /-S/ydb/core/tx/columnshard/engines/reader/transaction/tx_scan.cpp +122
13NKikimr::NTabletFlatExecutor::TSeat::Complete(NActors::TActorContext const&, bool) () /-S/ydb/core/tablet_flat/flat_exec_seat.cpp +18
14NKikimr::NTabletFlatExecutor::CompleteRoTransaction(std::__y1::unique_ptr<NKikimr::NTabletFlatExecutor::TSeat, std::__y1::default_delete<NKikimr::NTabletFlatExecutor::TSeat>>, NActors::TActorContext const&, NKikimr::NTabletFlatExecutor::TExecutorCounters*, NKikimr::TTabletCountersWithTxTypes*) () /-S/ydb/core/tablet_flat/flat_executor_txloglogic.cpp +69
15NKikimr::NTabletFlatExecutor::TLogicRedo::Confirm(unsigned int, NActors::TActorContext const&, NActors::TActorId const&) () /-S/ydb/core/tablet_flat/flat_executor_txloglogic.cpp +298
16NKikimr::NTabletFlatExecutor::TExecutor::Handle(TAutoPtr<NActors::TEventHandle<NKikimr::TEvTablet::TEvCommitResult>, TDelete>&, NActors::TActorContext const&) () /-S/ydb/core/tablet_flat/flat_executor.cpp +3175
17NKikimr::NTabletFlatExecutor::TExecutor::StateWork(TAutoPtr<NActors::IEventHandle, TDelete>&) () /-S/ydb/core/tablet_flat/flat_executor.cpp +4216
18NActors::IActor::Receive(TAutoPtr<NActors::IEventHandle, TDelete>&) () /-S/ydb/library/actors/core/actor.cpp +280
19NActors::TExecutorThread::Execute(NActors::TMailbox*, bool) () /-S/ydb/library/actors/core/executor_thread.cpp +268
20NActors::TExecutorThread::ProcessExecutorPool()::$_0::operator()(NActors::TMailbox*, bool) const () /-S/ydb/library/actors/core/executor_thread.cpp +458
21NActors::TExecutorThread::ProcessExecutorPool() () /-S/ydb/library/actors/core/executor_thread.cpp +510
22NActors::TExecutorThread::ThreadProc() () /-S/ydb/library/actors/core/executor_thread.cpp +536
23(anonymous namespace)::TPosixThread::ThreadProxy(void*) () [util/system/thread.cpp](https://a.yandex-team.ru/arc/trunk/arcadia/util/system/thread.cpp?#L245) +245
24?? () libpthread.so.0
25?? () libc.so.6

https://nda.ya.ru/t/iKYFLY3t7G65ce
https://nda.ya.ru/t/APFwQFtJ7G65eV
https://nda.ya.ru/t/Qez-vTJL7G65yb