how to use the model's checkpoint in local fold?

[zk19971101]

System Info

ghcr.io/huggingface/text-generation-inference 2.0.4
platform windows10
Docker version 27.0.3
llm model:lllyasviel/omost-llama-3-8b-4bits
cuda 12.3
gpu nvidia rtx A6000

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

C:\Users\Administrator>docker run --gpus all -p 8080:80 -v ./data:/data ghcr.io/huggingface/text-generation-inference:2.0.4 --model-id "F:\Omost-main\checkpoints\models--lllyasviel--omost-llama-3-8b-4bits" --max-total-tokens 9216 --cuda-memory-fraction 0.8

Expected behavior

eventhought i set the model-id =, docker raise a error.

[danieldk]

Did you try to remove the double dashes in the model name models--lllyasviel--omost-llama-3-8b-4bits as suggested in the error?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.