Bug: Phi-3 4K output broken after 2000~ tokens (Reproducible)

[Amadeus-AI]

What happened?

To reproduce:
Download the official released gguf model from huggingface/microsoft.
Run server.exe -m Phi3-mini-4k.gguf -c 4096

When input prompt < ~2048: Output fine. (but output starts getting weird right after it hits ~2048 in total)
When input prompt > ~2048: Output weird.

The weird output seems like what we expect to see when the context is more than the model support, but happens in ~2048, which seems like there are some bugs.

Also tested Llama3-8B, works fine with input prompt < 8192 as expected (with -c 8192), also works fine with input prompt < 4096 as expected (with -c 4096).

Name and Version

version: 3015 (74b239b)
built with MSVC 19.39.33523.0 for x64

Tried both cuda and avx2 version.

Also tried latest version built it myself @ Intel SYCL
version: 3075 (3d7ebf6)
built with IntelLLVM 2024.1.0

What operating system are you seeing the problem on?

Win10, Win11

Relevant log output

Before ~2000 tokens and after

[matteoserva]

I can confirm this. I tried to ask it to summarize an article in italian. Everything is fine until it hits the 2000 tokens wall. After that it outputs garbage.
The model uses a sliding windows attention of 2048 tokens. It might be related.

[Galunid]

Can you try 6369bf0 and 201cc11 to see if there's a difference? First one should work alright, second should break.

[Amadeus-AI]

@Galunid
version: 2960 (6369bf0)
built with IntelLLVM 2024.1.0

Still break

[Galunid]

I can reproduce, it seems there's some issue with the initial implementation in #6852

[ggerganov]

It's most likely the missing sliding window, as pointed out earlier

[Galunid]

@jggc This topic relates to Phi-3 model that has degradation in quality before it runs out of context, so I marked your comments as off-topic. Quality degradation after you run out is expected and from what I understood that is the case here.

[jggc]

Indeed my behavior is slightly different but still degradation WITHIN the context length. I posted in this thread instead of opening a new issue since it had enough similarities that I thought it might be related.

I'll rephrase to make things clearer :

1. Start server
2. Call /completion with a short prompt such as "What is 2+2"
3. Response is OK
4. Call /completion with a long prompt exceeding context length
5. It fails will generate garbage, as expected in this context
6. Call /completion with a short prompt again "What is 2+2"
7. Get garbage output, this is not expected. Model state should not be broken in the server after a single prompt exceeded the context length in the session.

At this point, no matter what I do I won't get sensible responses until I restart the server.

@Galunid Let me know if I should open a new bug. It is reproducible, I could write a gist.

[ngxson]

Interestingly, phi-3-small use a combination of sliding window + block sparse attention. So even we got a hack for sliding window (used by gemma 2), it will still be messy if we want proper support for phi-3

Link to paper: https://arxiv.org/pdf/2404.14219

[njsyw1997]

ONNX runtime has the same bug. This might be a reference for us if they can fix it.
microsoft/onnxruntime-genai#552

[CASE-R]

Commenting to see if there has been an update/solution to this before it gets closed for activity? We've faced this issue for a month now and using the 128K context models is problematic due to available hardware