Throttling incoming indexing when Lucene merges fall behind

[mikemccand]

Lucene has low-level protection that blocks incoming segment-producing threads (indexing threads, NRT reopen threads, commit, etc.) when there are too many merges running.

But this is too harsh for Elasticsearch, so it's entirely disabled, but this means merges can fall far behind under heavy indexing, and this results in too many segments in the index, which causes all sorts of problems (slow version lookups, too much RAM, etc.).

So we need to do something "softer"; Simon has a good starting patch, which I tested and confirmed (after https://issues.apache.org/jira/browse/LUCENE-5644 is fixed) at least in one use-case that it prevents too many segments in the index:

Before Simon's + Lucene's fix: http://people.apache.org/~mikemccand/lucenebench/base.html

Same test with the fix: http://people.apache.org/~mikemccand/lucenebench/throttled.html

Segment counts stay essentially flat.

Here's Simon's prototype patch: s1monw@2de96f9

[nik9000]

It looks like Simon's prototype pauses the indexing thread if too many merges are in flight. I'm not 100% clear on the code path that gets here. Will that pause indexing or pause refreshing or both? It'd be neat to slow down just the refreshing and let indexing be slowed down by the refresh backlog logic. Or am I crazy?

[s1monw]

@nik9000 internally the IndexWriter has several threads states (8 by default) that we index into. If we limit to a single threads we only use on of the states and make sure we max out the RAM buffer and write the least amount of segments. This means we 1. reduce the number of segments to merge and 2. make sure flushes are only done if really needed. I think we can't slow down refreshes otherwise folks will see odd results since you don't get new documents. You also want to refresh to publish merged segments to further reduce the number of segments. We will do the right thing and provide backpressure on indexing not on refresh. Hope that makes sense?

[nik9000]

We will do the right thing and provide backpressure on indexing not on refresh. Hope that makes sense?

I'd honestly forgotten about flushes. Its what I get for only playing on the other side. Anyway, I'm happy so long as back pressure is provided on indexing.

[mikemccand]

I tested the current throttling branch with the refresh=-1 case, and we have problems because the "abandoned" thread states will never flush until a full flush ... workaround is you must use a refresh to get them flushed.

[mikemccand]

I'm inclined to simply document that index throttling won't kick in if you use SerialMergeScheduler.

SMS only allows one merge to run at a time, so apps that are doing heavy bulk indexing really should not be using it.

[mikemccand]

OK I reviewed these changes with Simon. We decided we don't need to add a separate "kill switch" for this because you can just set max_merge_count higher to avoid throttling. But we also decided not to document this new setting on the index-modules-merges docs: it's a very advanced setting, and playing with it could easily mess up merges.