completionStats are slow

[johnskopis]

Elasticsearch version (bin/elasticsearch --version):
6.5.2

Plugins installed: []
elasticsearch-prometheus-exporter

JVM version (java -version):
default docker version

OS version (uname -a if on a Unix-like system):
default docker version

Description of the problem including expected versus actual behavior:
Occasionally we see the cluster turn red. Looking at a thread dump I can see these threads running:
https://gist.github.com/johnskopis/bb3012a2f2b56304cc31d2861b6bfe0

Seems to me the management threadpool becomes full. Does this sound plausible?

This seems to be from monitoring using elasticsearch-prometheus-exporter.

Seems to be expensive to calculate completion stats:

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html

Do the stats need to run in management TP?

Can it be less expensive?

Can the default change from all fields to no fields?

Any other ideas?

Thanks

[jkakavas]

Hi @johnskopis !

Thanks very much for your interest in Elasticsearch.

This appears to be a user question, and we'd like to direct these kinds
of things to the forums. If you can stop
by there, we'd appreciate it. This allows us to use GitHub for verified
bug reports, feature requests, and pull requests.

Also please make sure that the gist URL you include is valid and public
as the current one returns a 404.

There's an active community in the forums
that should be able to help get an answer to your question. As such, I
hope you don't mind that I close this.

[johnskopis]

https://gist.github.com/johnskopis/bb3012a42f2b56304cc31d2861b6bfe0 is the correct link. that's copypasta fail

[johnskopis]

Also, concurrent requests for index stats causing the cluster to go RED seems like a bug. I can open on the forums but I expect you can reproduce by having a large/busy cluster and requesting indexstats as highlighted in thread dump...

[johnskopis]

https://discuss.elastic.co/t/concurrent-indices-stats-requests-cause-cluster-to-go-red/161385