Searcher acquired on transport thread

[DaveCTurner]

Today SearchService#rewriteShardRequest runs on a transport thread (Names.SAME) and calls Engine#acquireSearcher. If the engine is a frozen engine then this may involve opening a directory reader, which does a fair amount of IO and therefore should not happen on a transport thread.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Engine)

[ywelsch]

I think this is caused by the changes in #51708 where we now always rewrite the request, even in later phases. FrozenEngine has special handling for the case where the acquired searcher has source type "can_match", but the searcher acquired here is used for more than just rewriting.

@jimczi How should we fix this?