[Feature Request] Tie breaker for search with search_after pagination

[Arpit-Bandejiya]

Is your feature request related to a problem? Please describe

When we do the sorting by datetime and have the recurrent values.

#1 "2024-01-03 19:57:38"
#2 "2024-01-03 19:57:38"
...
#3 "2024-01-04 19:57:39"
#4 "2024-01-04 19:57:39"

This makes leaks of the docs while paginating using search_after parameter. According to the dataset above imagine first 10K docs ends with #1 value, so the next 10K will start with #3. #2 is missed.

This feature is requested by other users as well: https://stackoverflow.com/questions/76042569/can-i-imitate-a-tie-breaker-field-in-opensearch-with-search-after-pagination

Describe the solution you'd like

We need to introduce an default tie_breaker_fields for the PIT with search_after.

Related component

Search:Query Capabilities

Describe alternatives you've considered

No response

Additional context

No response

[msfroh]

@Arpit-Bandejiya -- Does this work if you sort by timestamp and _id, then search_after with both? That should provide a unique sort, right?

In theory, I suppose we could add _id as an implicit tie breaker.

[bharath-techie]

Hi @msfroh ,
By default, many users don't seem to index '_id' as a different doc values field. So , they get loaded as field data and has a impact on heap usage.