GGUF compatibility

[adam-clarey]

I've used the runpod/worker-vllm:0.3.0-cuda11.8.0 container for several different LLMs and it has worked fine so far.

I've just been given a requirement to test GGUF model (specifically https://huggingface.co/impactframes/llama3_if_ai_sdpromptmkr_q4km) and it keeps generating errors:

Entry Not Found for url: https://huggingface.co/impactframes/llama3_if_ai_sdpromptmkr_q4km/resolve/main/config.json.

Is this an issue with the model, or the worker? Is there a known workaround?

Thanks

[ashleykleynhans]

vllm itself doesn't support GGUF, therefore the worker cannot support it either:
vllm-project/vllm#1002

[alpayariyak]

@ashleykleynhans’s answer is correct

[thusinh1969]

vLLM support gguf now, would RunPod support it by single command line as usual?

Steve