Re-consider whether _forcemerge should honor max_num_segments over maxMergedSegmentSize

[jpountz]

Currently _forcemerge chooses to honor max_num_segments over the maximum segment size. So if the maximum segment size is 5GB (default) and you ask a 10GB shard to be force-merged then you will get a 10GB segment. It makes it hard to have a one-size-fits-all policy regarding the value of max_num_segments that should be provided: merging shards of 10GB down to 1 segment is probably fine, merging shards of 300GB down to 1 segment maybe not.

Maybe one way to work around this issue would be to have a configurable maximum segment size for forced merges as well, and to honor the maximum segment size over max_num_segments. For instance say the maximum segment size for forced merges is 50GB then a 300GB shard would more likely be reduced to 6-7 segments.

This would probably be helpful for ILM.

[elasticmachine]

Pinging @elastic/es-core-features

[bra-fsn]

I just wanted to open a feature request for this today because I'm struggling with the exploding number of segments with a lot of deleted documents lingering around in very big shards, but the current forcemerge isn't really useful to solve this problem.
What you've described fits well for this case: if I want to, I could merge a big shard into one big segment, but I could break that out to smaller ones by specifying a max_segment_size.
I guess it's easy to do and provides a huge value over the way it works currently.
Please implement this!

[jpountz]

@bra-fsn I don't think this idea would help much with reclaiming deletes. What version are you using? Since Elasticsearch 6.5 we have better logic for reclamation of deleted documents in segments that reached the maximum segment size (5GB by default). Before 6.5 it wouldn't be unlikely that 50% of your documents would be deletes for an update-heavy use-case. (If you have follow-up questions, please raise them at discuss.elastic.co and ping me there, so that we can keep this issue focus on the problem that is described.)

[bra-fsn]

Running 6.6.2. Out of 20 billion (primary) docs I have around 1.5B deleted and this is with I do forcemerges on smaller indices regularly. I upgraded to 6.6.0 in February and the situation is really better with that version:

But getting rid of them is just a nice side effect (sorry for mentioning it), what I'm really up to is to aggressively reduce the number of segments in heavily written/updated indices.
And having what you've described would be a good tool for it.

[dakrone]

We discussed this and decided that with the regular policies for beats and logstash sitting around the 50gb mark, this isn't something we're likely to hit. For now we are going to wait and see whether any users have issues force merging to a single large segment.

[bra-fsn]

@dakrone: we -and I know others- have. We would like to merge larger indices occasionally to free up memory and resources, but currently the way it works prohibits us to do this.
BTW, if you could solve this, you could remove this warning from the forcemerge docs:

Warning
Force merge should only be called against read-only indices. Running force merge against a read-write index can cause very large segments to be produced (>5Gb per segment), and the merge policy will never consider it for merging again until it mostly consists of deleted docs. This can cause very large segments to remain in the shards.

Currently it's really limited to read only indices.

[mbudge]

I have a use case for this. One of my indexes has a policy to create a new index after the index contains 120,000 documents in 3 shards working out at 40GB per shard. A lot of the documents have scripted updates with painless through the ingest pipeline so the index size grows quite large.

At the moment I have to periodically shut the ingest down to reindex each index to a new index with the same read/search alias, then delete the source index. Would be easier to do a force merge with max segment size, so I could keep doing that to reduce index size by removing the physical deletes. That or I create a new microservice backed by sql to store the updates (which would be another component to maintain..)

[craigglennie]

Wanted to bump this up. We'd like to be able to expunge deletes without risking that segments end up larger than the max segment size limit. This would let us optionally intervene to reclaim disk space (our indexes are large, never go read-only, and have lots of updates) but otherwise let normal merging consider all the segments.