DataShard: VERIFY failed: ydb/core/tx/datashard/datashard_write_operation.cpp:385: SetTxBody(): requirement !WriteRequest failed

[snaury]

Stack trace when the new EvWrite api is enabled:

VERIFY failed (2024-09-26T13:35:18.136902+0300):
  ydb/core/tx/datashard/datashard_write_operation.cpp:385
  SetTxBody(): requirement !WriteRequest failed
0. util/system/yassert.cpp:83: NPrivate::InternalPanicImpl(int, char const*, char const*, int, int, int, TBasicStringBuf<char, std::__y1::char_traits<char>>, char const*, unsigned long) @ 0xAF8DBBC
1. util/system/yassert.cpp:55: NPrivate::Panic(NPrivate::TStaticBuf const&, int, char const*, char const*, char const*, ...) @ 0xAF878E6
2. ydb/core/tx/datashard/datashard_write_operation.cpp:385: NKikimr::NDataShard::TWriteOperation::SetTxBody(TBasicString<char, std::__y1::char_traits<char>> const&) @ 0x162CAD17
3. ydb/core/tx/datashard/datashard_write_operation.cpp:504: NKikimr::NDataShard::TWriteOperation::RestoreTxData(NKikimr::NDataShard::TDataShard*, NKikimr::NTable::TDatabase&) @ 0x162CC8C7
4. ydb/core/tx/datashard/datashard_pipeline.h:286: NKikimr::NDataShard::TPipeline::RestoreWriteTx(NKikimr::NDataShard::TWriteOperation*, NKikimr::NTabletFlatExecutor::TTransactionContext&) @ 0x164375FC
5. ydb/core/tx/datashard/execute_write_unit.cpp:221: NKikimr::NDataShard::TExecuteWriteUnit::Execute(TIntrusivePtr<NKikimr::NDataShard::TOperation, TDefaultIntrusivePtrOps<NKikimr::NDataShard::TOperation>>, NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x164375FC
6. ydb/core/tx/datashard/datashard_pipeline.cpp:1830: NKikimr::NDataShard::TPipeline::RunExecutionPlan(TIntrusivePtr<NKikimr::NDataShard::TOperation, TDefaultIntrusivePtrOps<NKikimr::NDataShard::TOperation>>, TVector<NKikimr::NDataShard::EExecutionUnitKind, std::__y1::allocator<NKikimr::NDataShard::EExecutionUnitKind>>&, NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x16175123
7. ydb/core/tx/datashard/datashard__progress_tx.cpp:73: NKikimr::NDataShard::TDataShard::TTxProgressTransaction::Execute(NKikimr::NTabletFlatExecutor::TTransactionContext&, NActors::TActorContext const&) @ 0x165603F8
8. ydb/core/tablet_flat/flat_executor.cpp:1715: NKikimr::NTabletFlatExecutor::TExecutor::ExecuteTransaction(TAutoPtr<NKikimr::NTabletFlatExecutor::TSeat, TDelete>, NActors::TActorContext const&) @ 0xDA19EDF
9. ydb/core/tablet_flat/flat_executor.cpp:2636: NKikimr::NTabletFlatExecutor::TExecutor::Handle(TAutoPtr<NActors::TEventHandle<NKikimr::NTabletFlatExecutor::TExecutor::TEvPrivate::TEvActivateExecution>, TDelete>&, NActors::TActorContext const&) @ 0xDA266CA
10. ydb/core/tablet_flat/flat_executor.cpp:3938: NKikimr::NTabletFlatExecutor::TExecutor::StateWork(TAutoPtr<NActors::IEventHandle, TDelete>&) @ 0xDA08B2D
11. ydb/library/actors/core/executor_thread.cpp:251: NActors::TGenericExecutorThread::TProcessingResult NActors::TGenericExecutorThread::Execute<NActors::TMailboxTable::TReadAsFilledMailbox>(NActors::TMailboxTable::TReadAsFilledMailbox*, unsigned int, bool) @ 0xBF47846
12. ydb/library/actors/core/executor_thread.cpp:441: NActors::TGenericExecutorThread::ProcessExecutorPool(NActors::IExecutorPool*)::$_0::operator()(unsigned int, bool) const @ 0xBF3BD02
13. ydb/library/actors/core/executor_thread.cpp:493: NActors::TGenericExecutorThread::ProcessExecutorPool(NActors::IExecutorPool*) @ 0xBF3B6D4
14. ydb/library/actors/core/executor_thread.cpp:524: NActors::TExecutorThread::ThreadProc() @ 0xBF3C58F
15. util/system/thread.cpp:244: (anonymous namespace)::TPosixThread::ThreadProxy(void*) @ 0xAF922CE
16. ??:0: ?? @ 0x7EFD9490E608
17. ??:0: ?? @ 0x7EFD9482E352

I think we may have some missing cleanup between ReleaseTxData and RestoreTxData.

[snaury]

The error seems to be here:

ydb/ydb/core/tx/datashard/datashard_write_operation.cpp

Line 419 in 987a8eb

if (!WriteTx || IsTxDataReleased())

We assign to ReleasedTxDataSize (which makes it non-zero), and then immediately check IsTxDataReleased() which is now true, so we perform no cleanup at all. TActiveTransaction calls DataTx->IsTxDataRelease() which is a different check entirely.

[snaury]

To reproduce we would probably need to induce a page fault during a distributed non-volatile write...