Limit the timeout of scroll requests

[jpountz]

Individual scrolls requests should be consumed quickly in order not to hold references on to old segments for too long. Right now we accept arbitrary timeouts but we should be more defensive and put a limit. 5 minutes should be plenty for everyone?

[nik9000]

Reindex's throttling relies on setting the timeout higher enough to wait out the throttle period. Five minutes is probably fine but if we are going to do this we'll have to touch reindex too to make the error message it sends more sensible or to change how it implements throttling.

…

On Mon, Feb 20, 2017, 11:49 AM Adrien Grand ***@***.***> wrote: Individual scrolls requests should be consumed quickly in order not to hold references on to old segments for too long. Right now we accept arbitrary timeouts but we should be more defensive and put a limit. 5 minutes should be plenty for everyone? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#23268>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANLoqL8LEVbIGPD3CkfWdjzQnMuutxvks5recQigaJpZM4MGaG2> .

[martinscholz83]

Hi @jpountz, what about this

public Scroll(StreamInput in) throws IOException {
        this.keepAlive = new TimeValue(in);
        if(keepAlive.seconds() > 300)
            throw new IllegalArgumentException("Keep alive time should be less than 5 minutes");
    }

    /**
     * Constructs a new scroll of the provided keep alive.
     */
    public Scroll(TimeValue keepAlive) {
        if(keepAlive.seconds() > 300)
            throw new IllegalArgumentException("Keep alive time should be less than 5 minutes");
        this.keepAlive = Objects.requireNonNull(keepAlive, "keepAlive must not be null");
    }

If the amount of data is too big, try to use Spark + ES.
but if it uses Spark, input-output ratio, may not be too high.

[pratiksanglikar]

Is anyone working on this issue?
If not, can I pick up this issue as my first contribution?

[abeyad]

@pratiksanglikar I don't believe anyone is currently working on it, you are welcome to pick this up. Thank you!

[pratiksanglikar]

@abeyad Thank you!
I'll pick up this issue.