Llama2 answers is noise

I tried to use Llama2, code is executing but the results are random character, what is wrong?

llm = LLM(model="/home/gpt/.cache/huggingface/hub/models--meta-llama--Llama-2-13b-hf/snapshots/bff3f0e93147eb2c18aa8a888e73f010d8935d17")
prompts = [
    "I am so fast that I can",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

and got

Prompt: 'I am so fast that I can', Generated text: 'hd canciónbólści Zum framідnexProgram nov напskieWrapperabi go totale'
Prompt: 'The capital of France is', Generated text: 'Business Stock vieerves beskrevsiska}$. програкли laugh ","readerattednexшње савезној'
Prompt: 'The future of AI is', Generated text: "pleasure Mohрю Democratic Pel.»)}{ Foi Referencias maven Yeah beyond encryption river Catalunya '"

[admangan]

Is this perhaps a lora/peft model issue? I was looking at #182 and #289 to see if it works with the finetuned model.

[nootums]

Try with the chat version. llama-2-13b-chat-hf.

Try with the chat version. llama-2-13b-chat-hf.

Same result. i tried before, i was think that version without chat would be more suitable for continue of text

[admangan]

I had this same issue with a finetuned 7b llama2 not using lora or quantization...

Are you using the following tokenizer: 'hf-internal-testing/llama-tokenizer'

In tutorial examples the was no place where you should put specific tokenizer, maybe you can helps and show how to config it

[Sanster]

Maybe it's related to safetensors. After converting a baichuan-13b model into safetensor, the output results are completely incorrect. The llama2 chat models are also saved as safetensors.

[ktrapeznikov]

same issue here

[gesanqiu]

Tokenizer is not the issue. Make sure you use the huggingface weights, and not use safetensors format, in my test, only safetensors format weights will output gibberish.
Fine-tuned model is supported, since my SFT llama based model works fine with vLLM backend, and it's greedy search outputs are same of the transformers'.

[ktrapeznikov]

yep. not using safetensors fixed this. thanks @gesanqiu

I first download the model from huggingface by

model = LlamaForCausalLM.from_pretrained(model_name, cache_dir = CACHE_DIR,use_safetensors=False)
tokenizer = LlamaTokenizerFast.from_pretrained(model_name,cache_dir = CACHE_DIR,use_safetensors=False)

and then run it inside vllm

[ktrapeznikov]

you just need to download the model files first. I do it using the huggingface API with use_safetensors = False. You can also just clone the model repo from huggingface and delete safetensor files. For VLLM, you just need specify path to this model directory.

…

On Thu, Aug 3, 2023 at 12:43 AM Zac Pullar-Strecker < ***@***.***> wrote: @ktrapeznikov <https://github.com/ktrapeznikov> How did you do this? It doesn't seem like there's an obvious way to configure this in vllm? — Reply to this email directly, view it on GitHub <#615 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA65IIVJFMRR5UHL5DNQPHDXTMUA5ANCNFSM6AAAAAA24AVRLU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Sanster]

VLLM only looks for bin files to load model weights. If you want to be able to load safetensors (model loads much faster), you can modify the code at

vllm/vllm/model_executor/weight_utils.py

Line 20 in d2b2eed

def hf_model_weights_iterator(

The modified code, which will prioritize loading safetensors files, is as follows.

def _hf_model_weights_iterator(hf_folder):
    hf_safetensors_files = [
        x for x in glob.glob(os.path.join(hf_folder, "*.safetensors"))
    ]
    if len(hf_safetensors_files) != 0:
        from safetensors import safe_open

        hf_safetensors_files = [
            x for x in glob.glob(os.path.join(hf_folder, "*.safetensors"))
        ]
        for safetensor_file in hf_safetensors_files:
            with safe_open(safetensor_file, framework="pt", device="cpu") as f:
                for name in f.keys():
                    yield name, f.get_tensor(name)
    else:
        hf_bin_files = [
            x
            for x in glob.glob(os.path.join(hf_folder, "*.bin"))
            if not x.endswith("training_args.bin")
        ]
        for bin_file in hf_bin_files:
            state = torch.load(bin_file, map_location="cpu")
            for name, param in state.items():
                yield name, param
            del state
    torch.cuda.empty_cache()


def hf_model_weights_iterator(
    model_name_or_path: str,
    cache_dir: Optional[str] = None,
    use_np_cache: bool = False,
) -> Iterator[Tuple[str, torch.Tensor]]:
    # Prepare file lock directory to prevent multiple processes from
    # downloading the same model weights at the same time.
    lock_dir = cache_dir if cache_dir is not None else "/tmp"
    lock_file_name = model_name_or_path.replace("/", "-") + ".lock"
    lock = filelock.FileLock(os.path.join(lock_dir, lock_file_name))

    # Download model weights from huggingface.
    is_local = os.path.isdir(model_name_or_path)
    if not is_local:
        with lock:
            hf_folder = snapshot_download(
                model_name_or_path,
                allow_patterns="*.bin",
                cache_dir=cache_dir,
                tqdm_class=Disabledtqdm,
            )
    else:
        hf_folder = model_name_or_path

    if use_np_cache:
        # Convert the model weights from torch tensors to numpy arrays for
        # faster loading.
        np_folder = os.path.join(hf_folder, "np")
        os.makedirs(np_folder, exist_ok=True)
        weight_names_file = os.path.join(np_folder, "weight_names.json")
        with lock:
            if not os.path.exists(weight_names_file):
                weight_names = []
                for name, param in _hf_model_weights_iterator(hf_folder):
                    param_path = os.path.join(np_folder, name)
                    with open(param_path, "wb") as f:
                        np.save(f, param.cpu().detach().numpy())
                    weight_names.append(name)
                with open(weight_names_file, "w") as f:
                    json.dump(weight_names, f)

        with open(weight_names_file, "r") as f:
            weight_names = json.load(f)

        for name in weight_names:
            param_path = os.path.join(np_folder, name)
            with open(param_path, "rb") as f:
                param = np.load(f)
            yield name, torch.from_numpy(param)
    else:
        for name, param in _hf_model_weights_iterator(hf_folder):
            yield name, param

[LopezGG]

I've tested llama 2 models with 20 datasets with vllm and compared it with meta inference code. I have parity when I use

• weights downloaded from Huggingface
• use bin files