server : add n_discard parameter to specify the number of tokens to discard when context is shifted

[kaetemi]

Currently the context shift just shifts by half of what's not kept. A side effect is that sentences (and sometimes weird Unicode token sequences) get cut halfway. I found that whenever odd sentences got cut in half this had a noticeable negative effect on the model output. By adding n_discard as a parameter we can specify at application level where the natural or safe boundary of the context is (e.g. on newlines), so the context can be shifted more sanely in a more controlled manner.

[ngxson]

We don't have this option in main.cpp so it's quite strange to only implement it in server.cpp

I found that whenever odd sentences got cut in half this had a noticeable negative effect on the model output.

In this such case, can we set the n_keep to an odd value, so that the number of remaining tokens become even?

Could you also provide an example?

[kaetemi]

@ngxson Sorry, I might not have been clear there. I meant odd as in having a meaning that would be significantly changed if the sentence ends up getting cut in half. I don't think it inherently makes sense in main, because the value needs to be changed on the fly to whatever the natural boundary is based on the whatever is getting added to the prompt.

As an example.
Keep: John likes apples.
Context: He tries to sell them. Mary buys them and eats them all. John is happy now.

After context shift could become:
Keep: John likes apples.
Context: and eats them all. John is happy now.

Which changes the whole meaning of the context. Adjusting keep size does not stop sentences from getting cut in half.

By having the application specify where it's safe to context shift, this is largely avoided.