Add the ability to do post-indexing index-time sort on frozen/closed indices

[dakrone]

It would be nice to be able to take advantage of the storage savings from
index-time sorting, however, in a lot of cases the sorting criteria is not known
at index time, or the indexing cost is too high for the throughput required.

One way that we might be able to address this is to enhance the merging process
to perform an offline sort followed by a forcemerge. This could hopefully be
done while an index was frozen or closed.

This would allow us to build a functionality for this into ILM and have it
automatically done during the index's lifecycle.

/cc @s1monw maybe you can comment on the feasibility and details on doing the
offline sort and force merge?

[elasticmachine]

Pinging @elastic/es-distributed

[jpountz]

We might be able to sort the index by hooking Lucene's SortingLeafReader in the merge process. However this would only sort the index, not update the index metadata. Said otherwise we would have the space saving benefits, but not the search-time sort speedups since collectors would not realize that the index is sorted. My gut feeling is that it would be enough for most users.

Getting index metadata updated would be more complicated since IndewWriter forbids changing it on an existing index. So we would have to add a hack that rewrites the index metadata without Lucene's consent, which feels dangerous to me, or move the data to a different index in the same spirit as what the shrink/split operations are doing.

[dakrone]

Said otherwise we would have the space saving benefits, but not the search-time sort speedups since collectors would not realize that the index is sorted. My gut feeling is that it would be enough for most users.

I agree, I think that is what @eskibars is interested in for this use case.

[eskibars]

I agree, I think that is what @eskibars is interested in for this use case.

It is indeed!

[AtlantaPepsi]

Hi @jpountz this issue sounds very interesting to me, would you mind giving me some suggestions on where to start? It seems to me that ElasticsearchConcurrentMergeScheduler is in charge of the merging process? Should the sorting be done after doMerge()?

[s1monw]

I opened a PR on Lucene to allow this to happen after the fact. apache/lucene-solr#1789 it produces a full-fledged sorted index with updated metadata

[jpountz]

I have been using the new Lucene API to play with the impact of index sorting on disk usage, but unfortunately it proved slower than reindexing the content. On the dataset I was playing with, the bottleneck was the sorting of stored fields, which requires random lookups into stored fields and performed very slowly.

[henningandersen]

I wonder if this issue is superseded by the tsdb project, which also aims to lower storage cost through index sorting, i.e., can we close this? It is also not clear to me in which cases we would enable such an offline (disruptive I think) merge.