Description
Modifications of Runtime Fields in the mapping should invalidate the request cache. Currently, search requests that target runtime fields are eligible to the request cache but they are not removed when a modification occurs. We consider this behavior as a non-released bug.
We discussed several solutions to this problem:
- Add a serializable mapping version in the shard search request cache key.
We already expose a mapping version that is incremented on each mapping update. However we don't force a single mapping version per request. The mapping service can be updated in the middle of the parsing of a search request so multiple versions of the mapping can co-exist in a single shard search request.
We could force a single version during the parsing but that would require to add a way to clone a mapping service so that's not a low hanging fruit.
- Add versioning for runtime fields that is plugged in the shard search request cache key.
With one version per field we could associate each runtime field used in the search request with its version.
Although, extracting all fields that appear in a search request is not trivial. We don't have visitors for aggregations or query builders so that would require adding one. The advantage of this option is that caching would be invalidated only for search requests that target a modified runtime field.
- Invalidate all request cache entries whenever there is an update in the mapping.
Updating the mapping requires a cluster update so this event should be rare even with runtime fields.
We have plans to add the ability to plug runtime fields on top of the mapping (in the search request or in datastreams)
in order to add more flexibility. For these reasons, a simple solution could be to invalidate the shard request cache entries whenever there is an update in the mapping. We already have this invalidation when the searcher reopens so that should be straightforward to add.