[Filestore] NodeRef stays locked until tablet restart after InvalidNodeRefUponCompleteUnlinkNode upon OpLog replay right after enabling DirectoryCreationInShardsEnabled #7344

We hit AppCriticalEvents/InvalidNodeRefUponCompleteUnlinkNode right after enabling directory creation in shards for a filesystem. After that CreateHandle(O_CREAT) for the same (parent, name) was getting E_REJECTED forever - until the leader tablet restart.

Timeline (vhost profile log, the client creates and unlinks the same name once a minute):

16:11:27.673544Z  CreateHandle  0.002090s  S_OK        {parent_node_id=2, node_name=NAME, flags=38, ...}
16:11:27.679095Z  UnlinkNode    0.000957s  S_OK        {parent_node_id=2, node_name=NAME}
16:12:29.903219Z  CreateHandle  1.224134s  S_OK        {parent_node_id=2, node_name=NAME, flags=38, ...}
16:12:31.829491Z  UnlinkNode    0.049024s  E_REJECTED  {parent_node_id=2, node_name=NAME}
16:12:32.378675Z  UnlinkNode    0.324290s  E_REJECTED  {parent_node_id=2, node_name=NAME}
16:12:33.703092Z  UnlinkNode    0.000190s  S_OK        {parent_node_id=2, node_name=NAME}
16:13:27.673266Z  GetNodeAttr   0.000192s  E_FS_NOENT  {parent_node_id=2, node_name=NAME, ...}
16:13:27.673550Z  CreateHandle  0.000100s  E_REJECTED  {parent_node_id=2, node_name=NAME, flags=38, ...}
16:13:28.173746Z  CreateHandle  0.000278s  E_REJECTED  {parent_node_id=2, node_name=NAME, flags=38, ...}
... retried with backoff for 20+ minutes, all E_REJECTED

Crit event on the leader:

2026-09-28T16:12:32.716788Z :NFS_SERVER INFO: CRITICAL_EVENT:AppCriticalEvents/InvalidNodeRefUponCompleteUnlinkNode: CompleteUnlinkNode: Headers { ClientId: "..." SessionId: "..." RequestId: 1502489140 ... } FileSystemId: "FILESYSTEM" NodeId: 2 Name: "NAME", Error: E_FS_NOENT invalid target node #2 "NAME"

What happened:

1. UnlinkNode arrives while DirectoryCreationInShardsEnabled is still false. In this mode the NodeRef is deleted right away in ExecuteTx_UnlinkNode, in the same tx we write the OpLog entry and the DupCache entry:

   nbs/cloud/filestore/libs/storage/tablet/tablet_actor_unlinknode.cpp

   Lines 495 to 524 in e77458c

   	if (!GetFileSystem().GetDirectoryCreationInShardsEnabled()) {
   	UnlinkExternalNode(
   	*db,
   	args.ParentNodeId,
   	args.Name,
   	args.ChildRef->ShardId,
   	args.ChildRef->ShardNodeName,
   	args.ChildRef->MinCommitId,
   	args.CommitId);
   	}
   	// OpLogEntryId doesn't have to be a CommitId - it's just convenient to
   	// use CommitId here in order not to generate some other unique ui64
   	args.OpLogEntry.SetEntryId(args.CommitId);
   	auto* shardRequest = args.OpLogEntry.MutableUnlinkNodeInShardRequest();
   	shardRequest->MutableHeaders()->CopyFrom(args.Request.GetHeaders());
   	shardRequest->SetFileSystemId(args.ChildRef->ShardId);
   	shardRequest->SetNodeId(RootNodeId);
   	shardRequest->SetName(args.ChildRef->ShardNodeName);
   	shardRequest->SetUnlinkDirectory(args.Request.GetUnlinkDirectory());
   	shardRequest->MutableOriginalRequest()->CopyFrom(args.Request);
   	const bool serialized = args.ProfileLogRequest.SerializeToString(
   	args.OpLogEntry.MutableProfileLogRequest());
   	if (!serialized) {
   	ReportBrokenProfileLogRequest(TStringBuilder()
   	<< "Written OpLogEntry: "
   	<< args.OpLogEntry.ShortUtf8DebugString().Quote());
   	}
   	WriteOpLogEntry(*db, args.OpLogEntry);

2. ConfigureShards sets DirectoryCreationInShardsEnabled = true and the tablet commits suicide, the shard unlink is not completed yet so the OpLog entry stays:

   nbs/cloud/filestore/libs/storage/tablet/tablet_actor_updateconfig.cpp

   Lines 480 to 519 in e77458c

   	config.SetDirectoryCreationInShardsEnabled(
   	args.Request.GetDirectoryCreationInShardsEnabled());
   	config.SetForceDirectoryCreationInShards(
   	args.Request.GetForceDirectoryCreationInShards());
   	config.SetStrictFileSystemSizeEnforcementEnabled(
   	args.Request.GetStrictFileSystemSizeEnforcementEnabled());
   	LOG_INFO(
   	ctx,
   	TFileStoreComponents::TABLET,
   	"%s Configuring shards in ExecuteTx_ConfigureShards, target config: %s",
   	LogTag.c_str(),
   	config.ShortDebugString().c_str());
   	UpdateConfig(*db, *Config, config, GetThrottlingConfig());
   	}
   	void TIndexTabletActor::CompleteTx_ConfigureShards(
   	const TActorContext& ctx,
   	TTxIndexTablet::TConfigureShards& args)
   	{
   	LOG_INFO(ctx, TFileStoreComponents::TABLET,
   	"%s Configured shards, new shard list: %s, new file shard list: %s",
   	LogTag.c_str(),
   	JoinSeq(",", GetFileSystem().GetShardFileSystemIds()).c_str(),
   	JoinSeq(",", GetFileSystem().GetFileShardFileSystemIds()).c_str());
   	auto response =
   	std::make_unique<TEvIndexTablet::TEvConfigureShardsResponse>();
   	NCloud::Reply(ctx, *args.RequestInfo, std::move(response));
   	LOG_INFO(
   	ctx,
   	TFileStoreComponents::TABLET,
   	"%s Suiciding after shard reconfiguration to force clients to fetch"
   	" a new shard list via CreateSession method",
   	LogTag.c_str());
   	Suicide(ctx);

3. upon restart ReplayOpLog decides how to process the entry based on the current value of the flag, not on the value the entry was written with. So it locks the NodeRef and sets shouldUnlockUponCompletion = true:

   nbs/cloud/filestore/libs/storage/tablet/tablet_actor_oplog.cpp

   Lines 92 to 106 in e77458c

   	bool hasOriginalRequest = request.HasOriginalRequest();
   	bool shouldUnlockUponCompletion = hasOriginalRequest
   	&& GetFileSystem().GetDirectoryCreationInShardsEnabled();
   	if (shouldUnlockUponCompletion) {
   	// There is a need to unlock the node ref after the operation is
   	// completed, because the node ref should have been locked
   	const auto& originalRequest = request.GetOriginalRequest();
   	const bool locked = TryLockNodeRef(
   	{originalRequest.GetNodeId(), originalRequest.GetName()});
   	if (!locked) {
   	ReportFailedToLockNodeRef(TStringBuilder()
   	<< "UnlinkNodeInShardRequest: "
   	<< request.ShortUtf8DebugString().Quote());
   	}
   	}

4. after the shard responds we go to CompleteUnlinkNode:

   nbs/cloud/filestore/libs/storage/tablet/tablet_actor_unlinknode.cpp

   Lines 928 to 946 in e77458c

   	if (std::holds_alternative<NProto::TUnlinkNodeResponse>(res) &&
   	msg->ShouldUnlockUponCompletion)
   	{
   	const auto& originalRequest = msg->OriginalRequest;
   	auto response = std::get<NProto::TUnlinkNodeResponse>(res);
   	if (msg->RequestInfo) {
   	AddInFlightRequest<TEvService::TUnlinkNodeMethod>(
   	*msg->RequestInfo);
   	}
   	ExecuteTx<TCompleteUnlinkNode>(
   	ctx,
   	msg->RequestInfo,
   	originalRequest,
   	std::move(msg->ProfileLogRequest),
   	msg->OpLogEntryId,
   	response,
   	false /* isExplicitRequest */);

5. PrepareTx_CompleteUnlinkNode doesn't find the NodeRef (it was deleted at step 1) -> E_FS_NOENT:

   nbs/cloud/filestore/libs/storage/tablet/tablet_actor_unlinknode.cpp

   Lines 699 to 702 in e77458c

   	if (!args.ChildRef) {
   	args.Error = ErrorInvalidTarget(args.ParentNodeId, args.Name);
   	return true;
   	}

6. ExecuteTx_CompleteUnlinkNode deletes the OpLog entry and reports the crit event:

   nbs/cloud/filestore/libs/storage/tablet/tablet_actor_unlinknode.cpp

   Lines 725 to 744 in e77458c

   	DeleteOpLogEntry(*db, args.OpLogEntryId);
   	// If the original response was an error or prepare stage failed, we don't
   	// need to do anything
   	if (FAILED(args.Error.GetCode())) {
   	// It should be impossible to achieve this state
   	LOG_ERROR(
   	ctx,
   	TFileStoreComponents::TABLET,
   	"%s CompleteUnlinkNode is in invalid state: %s, original request: "
   	"%s",
   	LogTag.c_str(),
   	FormatError(args.Error).c_str(),
   	args.Request.ShortDebugString().c_str());
   	ReportInvalidNodeRefUponCompleteUnlinkNode(
   	TStringBuilder()
   	<< "CompleteUnlinkNode: " << args.Request.ShortDebugString()
   	<< ", Error: " << FormatError(args.Error));
   	return;
   	}

7. CompleteTx_CompleteUnlinkNode unlocks the NodeRef only if there is no error, so the lock taken at step 3 is never released:

   nbs/cloud/filestore/libs/storage/tablet/tablet_actor_unlinknode.cpp

   Lines 783 to 793 in e77458c

   	if (!HasError(args.Error)) {
   	LOG_DEBUG(
   	ctx,
   	TFileStoreComponents::TABLET,
   	"%s Unlinking node in shard upon CompleteUnlinkNode: %s, %s",
   	LogTag.c_str(),
   	args.ChildRef->ShardId.c_str(),
   	args.ChildRef->ShardNodeName.c_str());
   	UnlockNodeRef({args.ParentNodeId, args.Name});
   	}

8. the retried UnlinkNode gets S_OK from DupCache, so the client thinks that everything is fine
9. any subsequent CreateHandle with E_CREATE for the same name fails to lock the NodeRef and gets E_REJECTED:

   nbs/cloud/filestore/libs/storage/tablet/tablet_actor_createhandle.cpp

   Lines 173 to 191 in e77458c

   	bool isNodeRefLocked = false;
   	if (msg->Record.GetName() &&
   	HasFlag(msg->Record.GetFlags(), NProto::TCreateHandleRequest::E_CREATE))
   	{
   	isNodeRefLocked =
   	TryLockNodeRef({msg->Record.GetNodeId(), msg->Record.GetName()});
   	if (!isNodeRefLocked) {
   	auto error = MakeError(
   	E_REJECTED,
   	TStringBuilder()
   	<< "node ref " << msg->Record.GetNodeId() << " "
   	<< msg->Record.GetName() << " is locked for CreateHandle");
   	auto response = std::make_unique<TResponse>(error);
   	NCloud::Reply(ctx, *ev, std::move(response));
   	onReply(error);
   	return;
   	}
   	}

The lock is in-memory only and the OpLog entry is already deleted at step 6, so a leader tablet restart fixes the state. Nothing is lost - the file is unlinked both in the leader and in the shard, but the name can't be reused and the client hangs in retries.

Proposed fix

1. Persist the unlink completion mode in TUnlinkNodeInShardRequest and use it in TIndexTabletActor and TUnlinkNodeInShardActor to ensure correct oplog replay after configuration changes.
2. Preserve backward compatibility by deriving the mode for legacy entries where the field is absent.
3. Remove backward compatibility code when the changes are fully deployed.