Support queries against too many indices

[jordansissel]

Some users are running into problems where Kibana 3 will expand a year's worth of indices to a single query (sweeeet!) but that GET request exceeds some webserver's ability to process them (awwwww), and the queries fail. Apache and Elasticsearch each have different limits.

[jordansissel]

I don't have preference on solution, but two ideas come to mind:

1. Split queries that are expected to be very long into multiple queries
2. or, create a temporary alias that matches the index list and query that alias.

[rashidkpc]

Problem: Backend does not support long index lists.

1. Kibana already splits any index list it can into multiple requests. Eg, terms facets must be executed in a single request or the results will be wrong(er)
2. Could be ok, would like to see TTLs on aliases if we're going this direction. Probably doesn't make sense to have the front end attempting to manage the state of temporary objects

3rd option
3) Switch to using the indices filter.(http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/query-dsl-indices-filter.html). This has 2 issues:

• It breaks the security-via-url-pattern thing
• AFAIK Elasticsearch does not currently optimize the request to only hit shards with indices in the list.

[jordansissel]

Agree it's probably not worth solving in Kibana itself. ES needs to help here. I'll file a ticket against ES if there already isn't one.

[otisg]

@jordansissel did you end up filing that ticket? I just looked at ES issues, but couldn't find it.

[jordansissel]

@otisg Filed elasticsearch#7298 just now

[elvarb]

Should be a part of the documentation on how to modify this setting for each of the most popular web servers.

I use IIS ARR to reverse proxy elasticsearch and had to modify some registry settings to expend the limitation.

[spalger]

Since Kibana 4 uses mgets for it's requests, we've "solved" this. 😃

[jordansissel]