Query TimeAllowed in Elastic Server Side

[aloneldi]

Hi ,

I'm started new project in Elastic Search .

My previous one was on SOLR where I used the parameter timeAllowed in the server side and overwrite in the client side when needed to terminate long running queries and return partial results to avoid SOLR crashes .

It seems that this crusiel feature is missing in Elastic search .
I saw the parameter TerminateAfter . But as far as I understand it can apply only in the client side .
Also it seems that it will not work when running heavy aggregation query which return only small number of rows .

Also saw some workarounds like Circuit Breakers which return an error to the client .

The TimeAllowed in SOLR seems to be based on Lucene or maybe I'm wrong .

https://lucene.apache.org/core/4_4_0/core/org/apache/lucene/search/TimeLimitingCollector.html

It looks to me as a great feature to add which will help elasatic search stability and avoid from users to run bad queries on production .

Thanks

Alon

[clintongormley]

Hi @aloneldi

We already use TimeLimitingCollector, see https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/search/query/QueryPhase.java#L374

I think you're looking for the timeout parameter.

Please ask questions like these in the forum instead: https://discuss.elastic.co/

The github issues list is reserved for bug reports and feature requests only.

thanks

[aloneldi]

Hi Clinton ,

No it's not timeout parameter .
Timeout parameter only timeout the client request in the server side the request is keep running .
I'm looking for equivalent SOLR time-allowed parameter which can be configured in server side and return partial result incase the query exceeded its limit .

Alon
נשלח מה-iPhone שלי

‫ב-14 בפבר׳ 2016, בשעה 00:43, ‏‏Clinton Gormley ‏notifications@github.com כתב/ה:‬

Hi @aloneldi

We already use TimeLimitingCollector, see https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/search/query/QueryPhase.java#L374

I think you're looking for the timeout parameter.

Please ask questions like these in the forum instead: https://discuss.elastic.co/

The github issues list is reserved for bug reports and feature requests only.

thanks

—
Reply to this email directly or view it on GitHub.

[clintongormley]

No it's not timeout parameter .

Yes it is. See the docs: https://www.elastic.co/guide/en/elasticsearch/reference/2.2/search-request-body.html

timeout
A search timeout, bounding the search request to be executed within the specified time value and bail with the hits accumulated up to that point when expired. Defaults to no timeout.

Timeout parameter only timeout the client request in the server side the request is keep running .

No, depends on the client but they usually implement this timeout as the request_timeout.

[aloneldi]

Hi Clinton .

I mean the timeAllowed Paremeter in SOLR is not timeout parameter in
Elastic .

in SOLR there is timeout which terminate the session in the client side
but keep running the request in the server side .

TimeAllowed in SOLR in the other hand can be configured in the server
side to protect the server incase user run a wild query and also in the
client side incase you want to extend it for specific .

TimeAllowed in SOLR also terminate the search operation in the server side .

I'm looking for the same protection type in Elastic . In the Server side
and that the query will be terminated in case it exceed a specific
threshold

https://wiki.apache.org/solr/CommonQueryParameters#timeAllowed

Thanks

Alon

On 14/02/2016 5:00 PM, Clinton Gormley wrote:

No it's not timeout parameter .

Yes it is. See the docs:
https://www.elastic.co/guide/en/elasticsearch/reference/2.2/search-request-body.html

|timeout|
A search timeout, bounding the search request to be executed
within the specified time value and bail with the hits accumulated
up to that point when expired. Defaults to no timeout.

Timeout parameter only timeout the client request in the server
side the request is keep running .

No, depends on the client but they usually implement this timeout as
the |request_timeout|.

—
Reply to this email directly or view it on GitHub
#16410 (comment).

[clintongormley]

in SOLR there is timeout which terminate the session in the client side
but keep running the request in the server side .

This depends on the client and is usually called request_timeout.

TimeAllowed in SOLR in the other hand can be configured in the server
side to protect the server incase user run a wild query and also in the
client side incase you want to extend it for specific .

TimeAllowed in SOLR also terminate the search operation in the server side .

This can be set per request with the timeout parameter or globally on the server with the search.default_search_timeout setting.

See https://www.elastic.co/guide/en/elasticsearch/reference/2.2/search.html#global-search-timeout

[aloneldi]

Thanks Clinton .

Does it mean that using this parameter with value 30 (seconds) will
terminate the query in the server side in case it exceed 30 seconds as
well ?

Because its not clear in the docs nor in Forums .

https://www.elastic.co/guide/en/elasticsearch/guide/current/empty-search.html

"

It should be noted that this|timeout|does nothalt the execution of the
query; it merely tells the coordinating node to return the results
collected/so far/and to close the connection. In the background, other
shards may still be processing the query even though results have been sent.

Use the time-out because it is important to your SLA, not because you
want to abort the execution of long-running queries."

On 15/02/2016 2:08 PM, Clinton Gormley wrote:

in SOLR there is timeout which terminate the session in the client
side
but keep running the request in the server side .

This depends on the client and is usually called |request_timeout|.

TimeAllowed in SOLR in the other hand can be configured in the server
side to protect the server incase user run a wild query and also
in the
client side incase you want to extend it for specific .

TimeAllowed in SOLR also terminate the search operation in the
server side .

This can be set per request with the |timeout| parameter or globally
on the server with the |search.default_search_timeout| setting.

See
https://www.elastic.co/guide/en/elasticsearch/reference/2.2/search.html#global-search-timeout

—
Reply to this email directly or view it on GitHub
#16410 (comment).

[clintongormley]

This works in the same way. The collector expires after the timeout and so it returns partial results. This doesn't necessarily stop parts of the query running on the node, eg a heavy rewrite on a wildcard query. The only way to interrupt things like this is to add checks, which would also make them all slow.

See #9156 and #4586 for background

[aloneldi]

So what you are saying is that in SOLR when using timeAllowed there
are cases in which the query continue to run ?

As SOLR have Timeout parameter which can be set in the request and
TimeAllowed .

in All Forums I see that there no issues with TimeAllowed .

Where For Elastic There are many .

My Main issue is when query return many rows and I need to display to
the fire 100-1000 rows .

When using also sort by time stamp the query would run to get all data
and then return the the first 100 rows .

When my Database has billion rows and the user put filter which return
90% of the data this would take a lot of time and many times carshed the
JVM
with out of memory error .

Using TimeAllowed in solr to 30 Seconds we could be sure that the user
will get data and that the query will terminate in 30 seconds.

So your suggestion to use *search.default_search_timeout *would solve
this issue in Elastic ?

Thanks

Alon

On 15/02/2016 5:05 PM, Clinton Gormley wrote:

This works in the same way. The collector expires after the timeout
and so it returns partial results. This doesn't necessarily stop parts
of the query running on the node, eg a heavy rewrite on a wildcard
query. The only way to interrupt things like this is to add checks,
which would also make them all slow.

See #9156 #9156 and
#4586 #4586 for background

—
Reply to this email directly or view it on GitHub
#16410 (comment).

[clintongormley]

So what you are saying is that in SOLR when using timeAllowed there are cases in which the query continue to run ?

I don't know enough about SOLR to be certain. A colleague believes that SOLR 5 also added the ability to interrupt operations on the terms dictionary, but that there were performance issues with this approach. i don't know if it is true or if the situation has changed. Either way, Elasticsearch doesn't do this.

My Main issue is when query return many rows and I need to display to the fire 100-1000 rows .

Then set the size limit to 1000.

When my Database has billion rows and the user put filter which return 90% of the data this would take a lot of time and many times carshed the JVM with out of memory error .

This won't happen (unless you set size to something stupid like MAXINT, and we now prevent you from doing this, see https://www.elastic.co/guide/en/elasticsearch/reference/2.2/breaking_21_search_changes.html#_from_size_limits)

Using TimeAllowed in solr to 30 Seconds we could be sure that the user will get data and that the query will terminate in 30 seconds.

This doesn't ensure you get the right results, just some results.

This conversation is definitely one for the forums, and not the issues list.

[aloneldi]

Thanks Clinton .

On 15/02/2016 6:17 PM, Clinton Gormley wrote:

So what you are saying is that in SOLR when using /timeAllowed/
there are cases in which the query continue to run ?

I don't know enough about SOLR to be certain. A colleague believes
that SOLR 5 also added the ability to interrupt operations on the
terms dictionary, but that there were performance issues with this
approach. i don't know if it is true or if the situation has changed.
Either way, Elasticsearch doesn't do this.

My Main issue is when query return many rows and I need to display
to the fire 100-1000 rows .

Then set the |size| limit to 1000.

When my Database has billion rows and the user put filter which
return 90% of the data this would take a lot of time and many
times carshed the JVM with out of memory error .

This won't happen (unless you set |size| to something stupid like
MAXINT, and we now prevent you from doing this, see
https://www.elastic.co/guide/en/elasticsearch/reference/2.2/breaking_21_search_changes.html#_from_size_limits)

Using TimeAllowed in solr to 30 Seconds we could be sure that the
user will get data and that the query will terminate in 30 seconds.

This doesn't ensure you get the right results, just some results.

This conversation is definitely one for the forums, and not the issues
list.

—
Reply to this email directly or view it on GitHub
#16410 (comment).

[delongwu]

curl -XPUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"search.default_search_timeout" : "2s"
}
}'