ggml-cuda : support stablelm rope

[slaren]

Allows full offload of stablelm.

Looks ok, but needs more testing to make sure that the neox rope isn't broken for other cases.

Device 0: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6

model	size	params	backend	ngl	test	t/s
stablelm 3B mostly Q8_0	2.77 GiB	2.80 B	CUDA	34	pp 512	1176.33 ± 47.30
stablelm 3B mostly Q8_0	2.77 GiB	2.80 B	CUDA	34	tg 128	74.37 ± 0.49
stablelm 3B mostly Q8_0	2.77 GiB	2.80 B	CUDA	35	pp 512	6788.78 ± 66.06
stablelm 3B mostly Q8_0	2.77 GiB	2.80 B	CUDA	35	tg 128	140.22 ± 0.35

build: 5844493 (1551)

[ggerganov]

Thanks for looking into this. I'll probably find sometime to test this tomorrow, but if other people confirm in the meantime we can merge

[slaren]

The ppl with falcon 7b f16 is the same as in master, so at least that one isn't broken by this change.

Final estimate: PPL = 7.0087 +/- 0.04184

[Galunid]

Sometimes it hits GGML_ASSERT: ggml-cuda.cu:7759: ggml_is_contiguous(src0) for me.
I cannot debug since I run this on runpod, but seed is included in run command, so it should be easy to reproduce

$ ./main -m models/stabilityai-stablelm-3b-4e1t-Q8_0.gguf -p "The best music is " -ngl 35 --seed 1700600253

[...]

The best music is icky.
I can’t stand the words that are so popular nowadays, “that was a good song,” or “the lyrics were bad/good.” I don’t care if it was an emotional song, or just plain old fun to listen to; those words shouldn’t even exist in anyone’s vocabulary! When you put your music on and hear a new song and say that the lyrics are bad, then you haven’t listened carefully enough. It takes time for these songs to sink into your ears. You can’t just like it or dislike it when you first listen because there is always more to know about a particular song.
I love music. There isn’t anything better than listening to an album and going through the motions of hearing every note in order to understand the lyrics and appreciate what they are saying. When I listen, I have to do something else at the same time like read or draw because otherwise it would just be too much to take in all at once!
Music is the only way people can express themselves without words coming out of their mouths. It’s a universal language that everyone understands even if they don’t know what the person says but still feel something within them after hearing that song or piece of music from afar away through headphones or speakers.
I like to think about this quote by Steve Jobs: “Music is the ultimate form of expression; it can be heard and felt at once.”
The best songs are those that you hear and you’re not quite sure what they mean, but it sounds good. Music should make your ears tingle or something!
I love music because there is a song for every emotion: happy, sad, angry… The way words fit together to make up lyrics has always fascinated me – so many different ways we can describe the same thing and how some songs just seem like they’re telling stories about people who aren’t even real. I think that makes it all more interesting!
I love music because of the emotion in it, not the words or the beat. It’s something that connects us humans through this beautiful world we live on together – so why wouldn’t it be one of our greatest passions? I know many people who don’t care for any kind of music at all, but I think there’s nothing better than listening to some good tunes with your best friends while you work in the yard or go for an afternoon drive.
I like this quote from Neil Young: “GGML_ASSERT: ggml-cuda.cu:7759: ggml_is_contiguous(src0)

[slaren]

It happens while shifting due to the rope in llm_build_k_shift, the cache_k view is not contiguous, possibly due to the value of n_rot. This doesn't seem right, the layout in memory of the KV cache should be the same for all models, as far as I know. Are the params of llm_build_k_shift correct?

[Galunid]

They appear to be correct.

llama.cpp/llama.cpp

Lines 3454 to 3464 in 8e672ef

	static void llm_build_k_shift(
	struct ggml_context * ctx,
	const llama_hparams & hparams,
	const llama_cparams & cparams,
	const llama_kv_cache & kv,
	struct ggml_cgraph * graph,
	llm_rope_type type,
	int64_t n_ctx,
	int64_t n_rot,
	float freq_base,
	float freq_scale,

And we call it with

llama.cpp/llama.cpp

Line 4801 in 8e672ef

llm_build_k_shift(ctx0, hparams, cparams, kv_self, gf, LLM_ROPE_NEOX, n_ctx, hparams.n_rot, freq_base, freq_scale, cb);

The only suspicious thing is hparams.n_rot. It matches the arguments function takes, but other models use n_embd_head here.

[slaren]

The dimensions of the K tensor in the rope in the shift don't match the rope during evaluation, so there is definitely something wrong there. Maybe llm_build_k_shift needs to be modified to take different parameters for n_dims and n_rot.

[slaren]

This should work, but I am not sure that the naming of the parameters of llm_build_k_shift make a lot of sense now, n_embd_head is passed as n_rot, and hparams.n_rot as n_dims. @ggerganov please review.

[ggerganov]

Sorry for the delay. I think with the latest commit, llm_build_k_shift args now make sense.

@slaren If you agree, we can merge

[slaren]

Yes, this looks like a better solution than adding another parameter.