Support starcoder family architectures (1B/3B/7B/13B)

[wsxiaoys]

Related Issues:

#1901
#1441
#1326

Previously, it wasn't recommended to incorporate non-llama architectures into llama.cpp. However, in light of the recent addition of the Falcon architecture (see Pull Request #2717), it might be worth reconsidering this stance.

One distinguishing feature of Starcoder is its ability to provide a complete series of models ranging from 1B to 13B. This capability can prove highly beneficial for speculative decoding and making coding models available for edge devices (e.g., M1/M2 Macs).

I can contribute the PR if it matches llama.cpp's roadmap.

[Azeirah]

I was also looking for a small coding model. Someone on reddit recommended me to use stablecode 3b, it's based on neox architecture. I only just noticed that the model card says it's not supported by llama.cpp, but I did see that there's a convert script in this repo for gpt-neox so it might still be possible.

1b would of course be amazing too!

[ggerganov]

Yes, we can add more architectures - the main requirements are:

• concise implementations
• if the tokenizer is too complicated, just provide basic token-text mapping
• don't break LLaMA

The ggml repo already provides sample Starcoder implementation:

https://github.com/ggerganov/ggml/tree/master/examples

So it is a good starting point to bring it here

[wsxiaoys]

Great - will start working on it

[wsxiaoys]

done in #3187

[LaniakeaS]

run python convert.py models/starcoder/ got following output. is that mean llama.cpp doesn't support starcoder 15B?

Traceback (most recent call last):
  File "/home/guest/**/llama.cpp/convert.py", line 1295, in <module>
    main()
  File "/home/guest/**/llama.cpp/convert.py", line 1223, in main
    model_plus = load_some_model(args.model)
  File "/home/guest/**/llama.cpp/convert.py", line 1144, in load_some_model
    model_plus = merge_multifile_models(models_plus)
  File "/home/guest/**/llama.cpp/convert.py", line 637, in merge_multifile_models
    model = merge_sharded([mp.model for mp in models_plus])
  File "/home/guest/**/llama.cpp/convert.py", line 616, in merge_sharded
    return {name: convert(name) for name in names}
  File "/home/guest/**/llama.cpp/convert.py", line 616, in <dictcomp>
    return {name: convert(name) for name in names}
  File "/home/guest/**/llama.cpp/convert.py", line 591, in convert
    lazy_tensors: list[LazyTensor] = [model[name] for model in models]
  File "/home/guest/**/llama.cpp/convert.py", line 591, in <listcomp>
    lazy_tensors: list[LazyTensor] = [model[name] for model in models]
KeyError: 'transformer.wte.weight'

[LaniakeaS]

probably not relevant about what model it is since found same problem in here #4530.