Should we lower the default merge IO throttle rate?

[mikemccand]

In #6018 we are fixing a serious bug in the IO RateLimiting API we use from Lucene ... causing IO throttling to be effectively 8 MB/sec (way too low).

So, should we now bring the merge throttling back down from the current 50 MB/sec given that we previously increased it from 20 MB/sec not knowing about this issue?

The problem is, even at 30 MB/sec rate limiting, with #6018 fixed, I can quite easily get merges to fall behind with 4 indexing threads.

But then in that same case, when testing with #6066 the merges are able to keep up, but at the cost of periodically making indexing single threaded.

[nik9000]

Right now it'd be a 625% increase instead of the planned 250% increase. I think that big of a bump might be too much. I guess it depends who the defaults are targeted for. I imagine folks with lots of spindles and/or ssds care enough to tweak the defaults. That leaves cloud folks and those of us with physical hardware but few spindles.

For what it is worth I use two standard small hard drives on each of my nodes with software striping raid and iostat reports 10% utilization when they write 10MB/s each.

[rmuir]

I don't understand the logic at this point behind throttling merges at all.

With #6066 the term 'Throttling merges' really just implies 'Throttling indexing'.

So why rate-limit indexing to some arbitrary bytes/second? Surely not all users want to make this throughput/latency tradeoff just to tune Pnn search metrics, e.g. some may be doing high ingest and things like analytics instead that are less sensitive to small latency changes in search.

Why rate-limit indexing more than its already been rate-limited? We are discussing adjusting the dosage for a patient that may no longer need the medicine: perhaps #5882 was the cure.

Why rate-limit indexing via this indirect, confusing path (via merging). Why not just have the option to rate-limit indexing directly for users that want this?

Why rate-limit to bytes/second anyway? This is a bad measure, it doesn't adapt to different use cases or hardware.

It seems pretty expert to make this tradeoff, so maybe it should just be disabled by default?

[clintongormley]

@rmuir The original idea behind merge throttling was to prevent a 5GB merge (which could be sparked off by a single indexing request) from consuming all resources and killing search, and perhaps even making the box unresponsive. This was a scenario I saw regularly in production before merge throttling was introduced.

[rmuir]

Then maybe searches should be throttled too? After all a badly formed wildcard or rangequery can easily do the same thing.

[clintongormley]

Searches are something that the user has control over. Merges are not. Also, there's PR #4586 which wouldn't throttle searches, but would allow the user to timeout long running searches

[rmuir]

The merge policy can be controlled too. Like i said, the math here behind throttling by default to an arbitrary bytes/second makes little sense.

Sure for some specific hardware or specific use case, maybe it can make things better, but by default for everyone?

How can you sure it was a single merge causing the issue for you and not #5882 ? Maybe your hardware just did not have enough i/o concurrency.

[clintongormley]

How can you sure it was a single merge causing the issue for you and not #5882 ?

I had this problem several times and usually the stats output showed only one merge, but this was a long time ago, so no I can't be sure.

Maybe your hardware just did not have enough i/o concurrency.

Oh for sure it didn't! But I wasn't running on terrible boxes, in fact they were the biggest boxes I had, and certainly bigger than lots of our users are using now.

Sure for some specific hardware or specific use case, maybe it can make things better, but by default for everyone?

Trade offs. For people with fast disks, setting merge throttling too low gives them slow performance. For people with slow disks, setting merge throttling too high can cause nodes to fall out of the cluster. I'd prefer the defaults to minimise cluster instability. If you see slow throughput, it's easy enough to change the throttling. If your node is unstable, it can be more difficult to diagnose.

Clearly there is no single default that will work for everybody. We need to provide good example settings for different hardware setups. Even better would be some stress test tool which would make recommendations based on the user's actual hardware.

[rmuir]

Clearly there is no single default that will work for everybody. We need to provide good example settings for different hardware setups. Even better would be some stress test tool which would make recommendations based on the user's actual hardware.

Sure, but why do it in an indirect way (via merging), instead of directly rate-limiting all writes? This is easier for the user to understand, and its whats implied by #6066

[clintongormley]

Fair enough

[kimchy]

remember also that merge scheduler and policy are per Lucene index, and in ES, we can have several shards on a node, and merge throttling is the only way we have today to control it across all those shards.

Writes are "rate" limited already, btw, because we have a bounded thread pool that defaults to the number of cores, you can't have more threads than that. Rate limiting on number of operations feels weird, it needs to happen due to backpressure building.

[rmuir]

Search performance is also impacted by the number of segments... today this throttling (combined with disabling lucene's safety) causes segment explosion, so its probably having the exact opposite impact.

Keeping the index in a healthy state will also naturally improve search performance! These are the reasons why I question if anything should happen here by default.

[rmuir]

Writes are "rate" limited already, btw, because we have a bounded thread pool that defaults to the number of cores

Yes, thats nice because it adapts to the concurrency of the hardware of the machine and how fast it can process requests. MB/S does not.

[s1monw]

So why rate-limit indexing to some arbitrary bytes/second? Surely not all users want to make this throughput/latency tradeoff just to tune Pnn search metrics, e.g. some may be doing high ingest and things like analytics instead that are less sensitive to small latency changes in search.

the numbers were not set arbitrarily - I did some measurements in a incremental indexing scenario on a real world search index (~1k docs / min update rate) and the 20mb at that point were the setting that removed the spikes from the response rate. It was targeted to let housekeeping do it's work but don't go crazy with merging even not for a short time at the cost of having more segments. Given that the intention was to let searches go wild and reduce IO consumption for "segment housekeeping" I think it's a valid default? I mean I'd not recommend it if you do bulk indexing while not searching at all which is IMO not the main usecase of ES?