Bugged interference on Vicuna 7B GPTQ 16k

[pweglik]

I was playing simple example:

from vllm.entrypoints.llm import LLM
from vllm.sampling_params import SamplingParams

MODEL_NAME = "TheBloke/vicuna-7B-v1.5-16k-GPTQ"

llm = LLM(model=MODEL_NAME, quantization="gptq", gpu_memory_utilization=0.5)

sampling_params = SamplingParams(
    max_tokens=600,
    top_p=0.95,
    temperature=0.7,
    presence_penalty=0.5,
    frequency_penalty=0.5,
)

outputs = llm.generate(
    [
        """A chat between a curious user and an artificial intelligence assistant.
        The assistant gives helpful, detailed, and polite answers to the user's questions.\n\n
        USER: Write Python script converting RGB image to grayscale.\n
        ASSISTANT: """,
    ],
    sampling_params=sampling_params,
)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r},\nGenerated text: {generated_text!r}")
    print()

and I received random output. Either random character or few words looped endlessly.
When I switched backed to 4k model, everything was fine. I'm not sure if its related to GPTQ or 16k vicuna in general.
I'm not able to check unquantized or AWQ quantized model as it doesn't fit on my GPU during inference.
Relevant to GPTQ: #916
What inspired me to check model with shorter context (the last answer): #590

[Quang-elec44]

hi @pweglik May I ask which version of vllm you are using ? I installed version 0.1.7 but there is no quantization argument in LLM class

[pweglik]

Hi, I build vllm from source using this branch: #916 as I needed GPTQ feature which is not merged yet

[chu-tianxiang]

I guess it may be because rope scaling is only supported very recently. I merged the main branch and have no problem running the code in the post.

[hmellor]

Closing this issue as stale as there has been no discussion in the past 3 months.

If you are still experiencing the issue you describe, feel free to re-open this issue.