Issue: Is Falcon 40B in GGML format form TheBloke usable?

[dlippold]

Issue you'd like to raise.

Is the instruct or chat version of the model Falcon 40B in GGML format form TheBloke, i.e. that from https://huggingface.co/TheBloke/falcon-40b-instruct-GGML or from https://huggingface.co/TheBloke/falcon-40b-sft-mix-1226-GGML or from https://huggingface.co/TheBloke/falcon-40b-sft-top1-560-GGML or from https://huggingface.co/TheBloke/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2-GGML, usable? If several are usable, which is the prefered one?

In principle Falcon 40B should be usable as it is specified in #775 and #849. But there is a link to https://huggingface.co/tiiuae/falcon-40b-instruct which is not the GGML format from TheBloke.

Suggestion:

No response

[cebtenzzre]

Despite the titles of the HF repos, these are GGCC files, which are only really supported by ggllm.cpp. But this project is based on llama.cpp. You need GGML files, like these: https://huggingface.co/TheBloke/falcon-40b-instruct-GGML/tree/ef68241787499747cb21a6c8bd48384d0864003a

The GGUF version of Falcon will be supported in the next release, which is a newer format that should be easier to find.