Can we avoid force-merging all shard copies?

[jpountz]

@ywelsch mentioned to me that one could easily run a force-merge on a read-only index on a single copy by doing the following sequence of operations:

• call the index clone API and set the number of replicas to 0 on the clone
• force-merge the clone
• set the number of replicas to the original value on the clone
• flip the alias so that the clone takes the place of the original index in the data stream
• delete the original index

This would only require half the CPU compared to running a plain force-merge, which is a great deal. Should we move the force-merge ILM action to this sequence of operations instead of just calling the _force_merge API?

Interestingly it wouldn't require more temporary storage. Since the index clone API uses symlinks, it doesn't need additional storage initially, so we would only need 2x the size of a shard of temporary storage right after increasing the number of replicas of the clone to 1. This is the same as running the _force_merge API since merging needs the same amount of temporary storage as the size of a shard copy, and this is needed of both shard copies.

For the searchable-snapshots ILM action, which optionally performs a forced merge, we could do something better by adding an option to _force_merge to only merge primaries, and using it with the searchable-snapshots action. This would work since only primaries are used to take snapshots. And this would only require half the temporary storage that it needs today.

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[DaveCTurner]

My main concern with this as a direct replacement for the force-merge ILM action is the additional node-to-node data transfer volume that it would generate. It'd be preferable to snapshot the result of the force-merge and then recover the replica from the snapshot, since data transfer via the snapshot repository is effectively free (we're going to put the data there eventually anyway).

[jpountz]

Thanks for bringing up this point.

This makes me wonder: force-merge already implicitly marks indices as read-only. If we're also going to couple force-merging with taking a snapshot, then is there a good reason for marking an index read-only, force-merging it, snapshotting it, and then not making it a searchable snapshot? Said otherwise, should our docs recommend users to use the searchable_snapshot action instead of force_merge?

[DaveCTurner]

That would assume that users (a) have a cold or frozen tier and (b) are doing the force-merge while moving data into such a tier. I think there's still value in having force-merged shards in the warm tier too, although IMO in most cases it does indeed make sense to go straight to a searchable snapshot.

[jpountz]

That would assume that users (a) have a cold or frozen tier

Is it true? You can use the searchable snapshot action in any tier you'd like, can't you?

[VimCommando]

Is it true? You can use the searchable snapshot action in any tier you'd like, can't you?

Not quite, the warm tier does not allow searchable snapshots. Hot and cold can use fully-mounted indices and frozen only uses partially-mounted indices.

There two index lifecycle management (ILM) triggers for force merge right now are an explicit action or a searchable snapshot.

Explicit force merge ILM action

By convention a lot of ILM policies use the warm phase to run the force merge. Assuming each tier has its own dedicated hardware, this allows force merges to not impact ingestion performance and use the second-fastest hardware in the cluster. The explicit action happens in the phase it is defined.

There are some operational benefits to force merging data still in the hot or warm tiers:

1. If an index use-case frequently updates document, this reclaims the deleted documents
2. Once force merged to a single segment, search no longer needs to build global ordinals
3. Fewer segment files can slightly improve I/O (more sequential reads)

I will not overstate the performance benefit, it is a slight optimization, not a significant one. Depending on how many index rollovers there are (days of retention) and how many different index series there are (many Elastic Agent integrations) this does add up.

In this scenario weighing the additional data transfer cost @DaveCTurner mentions is definitely important. Most ingest-heavy workloads only run with 1 replica.

Most search workloads with high replica counts tend not to need the force merge anywhere near as often (if ever). Plus we've introduced a more flexible only_expunge_deletes.

Searchable snapshot force merge

This is where only force merging the primary shard makes a lot of sense.

This force merging occurs in the phase that the index is in prior to the searchable_snapshot action. For example, if using a searchable_snapshot action in the hot phase, the force merge will be performed on the hot nodes. If using a searchable_snapshot action in the cold phase, the force merge will be performed on whatever tier the index is prior to the cold phase (either hot or warm).

This is the sequence I see as highly inefficient:

1. An ILM policy uses searchable_snapshot in the cold phase (using a fully-mounted index)
2. An index moves from the warm to cold phase
3. A force merge triggers on both the primary and replica shards
4. The newly force-merged segment generates a new searchable snapshot (from the primary shard)
5. The cold node creates a restored-* index from the snapshot
6. Both the primary and replica shards are removed from the warm node

The problem is in steps 3 and 6, because fully-mounted indices don't use replicas. Step 3's force merge in the warm tier is completed on both the primary and replica shards, requiring two nodes to execute the task. Then in step 6 both shards are deleted as soon as the snapshot+restore is done. Making that force merged replica shard's lifetime exceptionally short.

The end result is spending a lot of CPU and I/O time to force merge replica shards that could have a lifespan shorter than the force merge itself. On a cluster force merging terabytes of shards a day, this is a huge resource sink with no real benefit.

We do have an issue to skip the force merge step in searchable_snapshot if an index was already explicitly force merged earlier. But this does not address the "force merge it and throw it away" behavior above.

[joegallo]

I'm removing the team-discuss label from some older Team:Data Management issues -- we've had plenty of time to discuss them, but we haven't, so the label isn't serving its purpose. Feel free to delete this comment and/or re-add the team-discuss label.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)