Disable searching across all indices

[avleen]

Hitting /_search causes elasticsearch to search all types, all indices.
With very large indices this is a problem.
We currently have ~30Tb of data across multiple logstash indices.

If someone talks to ES (eg, using sense or a other method) and doesn't specify the index name, the cluster effectively grinds to a half until we restart it.

Can we have an option to require specifying at least one index name to the search endpoint?

Thanks!

[martijnvg]

The setting 'allow_no_indices' requires at least one available index to be specified:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/multi-index.html

[martijnvg]

The above comment doesn't make sense. The index expansion should be disabled in this your case. However there're only two values for 'expand_wildcards', either 'open' for open indices and 'closed' for closed indices. If you don't have closed indices setting this to 'close' will make specifying no index (and * and _all as index) not execute the search request. I think a third option 'none' should be added, which disables index expansion.

[avleen]

Doesn't 'allow_no_indices' have to be set in the query?
We want a servers side setting to prevent this problem where people don't
use 'allow_no_indices' with their query.

This can too easily happen by accident right now.
On Jun 12, 2014 5:08 AM, "Martijn van Groningen" notifications@github.com
wrote:

The setting 'allow_no_indices' requires at least one available index to be
specified:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/multi-index.html

Reply to this email directly or view it on GitHub
#6470 (comment)
.

[martijnvg]

Yes, the 'allow_no_indices' and other multi index settings have to be set in the search request.

[avleen]

Right, that's exactly the problem :-)
Here's an analogy to the problem I face, where I have many people who will
want to query Elasticsearch directly:

Imagine if the rm command, when run with no arguments, defaulted to
defaulted to wiping your hard drive.
I can tell people to always use arguments, I can remind them constantly,
but it only takes one person one time to run rm without any arguments,
expecting everything to be safe.. Oops.

Without server-side enforcement here to protect the cluster, we have to
assume that everyone know and will always be able to use the right
arguments, which isn't possible.

On Thu, Jun 12, 2014 at 12:57 PM, Martijn van Groningen <
notifications@github.com> wrote:

Yes, the 'allow_no_indices' and other multi index settings have to be set
in the search request.

Reply to this email directly or view it on GitHub
#6470 (comment)
.

[martijnvg]

Right, I understand :-) a server side setting disallowing a search without index or type specified makes sense.

[tomryanx]

I'm also looking to enforce well formed queries by disabling _all and wildcard expansion on the index name. I've disabled creation of _all on the index level, but over-eager queries are still able to search every index (when they should be restricted by date range and specify all relevant indices). Any update on this one?

[clintongormley]

I'm on the fence about this one. It seems simple to add, but then everybody has a different policy that they want to enforce, so a tiny feature becomes complex. This is already easily solved by putting a proxy like nginx in front of ES and specifying the exact rules that you want to implement. I'm not sure that we should go down this road...

[nik9000]

It seems simple to add, but then everybody has a different policy that they want to enforce, so a tiny feature becomes complex.

I'm not sure the slippery slope argument applies here - its probably a bad idea to query all indexes in a decently large cluster and if that is simple to make an option then its probably worth it. Anyone with any more complex set of requirements can break out nginx.

OTOH if it isn't a simple option to implement just telling people to break out nginx if they want any filtering isn't that bad. Its not like Elasticsearch needs to be the one to enforce this. It'd just be convenient if it did.