Unable to run Gemma models (2B, 8bit quantized)

[snowclipsed]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Model runs and answers the question to the program:

from llama_cpp import Llama
llm = Llama(
      model_path="models/gemma-7b-it.Q4_K_M.gguf",
      n_gpu_layers=-1, # Uncomment to use GPU acceleration
      # seed=1337, # Uncomment to set a specific seed
      # n_ctx=2048, # Uncomment to increase the context window
)
output = llm(
      "Q: Name the planets in the solar system? A: ", # Prompt) # Generate a completion, can also call create_completion
)
print(output)

Current Behavior

Assertion error

llama_model_loader: loaded meta data with 24 key-value pairs and 254 tensors from models/gemma-7b-it.Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = .
llama_model_loader: - kv 2: llama.context_length u32 = 8192
llama_model_loader: - kv 3: llama.embedding_length u32 = 3072
llama_model_loader: - kv 4: llama.block_count u32 = 28
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 24576
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 192
llama_model_loader: - kv 7: llama.attention.head_count u32 = 16
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 16
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 11: general.file_type u32 = 15
llama_model_loader: - kv 12: tokenizer.ggml.model str = llama
llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,256000] = ["", "", "", "", ...
llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,256000] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,256000] = [3, 3, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 2
llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 1
llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 3
llama_model_loader: - kv 19: tokenizer.ggml.padding_token_id u32 = 0
llama_model_loader: - kv 20: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 21: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 22: tokenizer.chat_template str = {% if messages[0]['role'] == 'system'...
llama_model_loader: - kv 23: general.quantization_version u32 = 2
llama_model_loader: - type f32: 57 tensors
llama_model_loader: - type q4_K: 169 tensors
llama_model_loader: - type q6_K: 28 tensors
llm_load_vocab: mismatch in special tokens definition ( 416/256000 vs 260/256000 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 256000
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 8192
llm_load_print_meta: n_embd = 3072
llm_load_print_meta: n_head = 16
llm_load_print_meta: n_head_kv = 16
llm_load_print_meta: n_layer = 28
llm_load_print_meta: n_rot = 192
llm_load_print_meta: n_embd_head_k = 192
llm_load_print_meta: n_embd_head_v = 192
llm_load_print_meta: n_gqa = 1
llm_load_print_meta: n_embd_k_gqa = 3072
llm_load_print_meta: n_embd_v_gqa = 3072
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff = 24576
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 8192
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: model type = ?B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 8.54 B
llm_load_print_meta: model size = 4.77 GiB (4.80 BPW)
llm_load_print_meta: general.name = .
llm_load_print_meta: BOS token = 2 ''
llm_load_print_meta: EOS token = 1 ''
llm_load_print_meta: UNK token = 3 ''
llm_load_print_meta: PAD token = 0 ''
llm_load_print_meta: LF token = 227 '<0x0A>'
llm_load_tensors: ggml ctx size = 0.10 MiB
llama_model_load: error loading model: create_tensor: tensor 'output.weight' not found
llama_load_model_from_file: failed to load model
Traceback (most recent call last):
File "/home/snow/a2n/src/test.py", line 2, in
llm = Llama(
^^^^^^
File "/home/snow/anaconda3/envs/a2n/lib/python3.11/site-packages/llama_cpp/llama.py", line 326, in init
self._ctx = _LlamaContext(
^^^^^^^^^^^^^^
File "/home/snow/anaconda3/envs/a2n/lib/python3.11/site-packages/llama_cpp/_internals.py", line 255, in init
assert self.model.model is not None
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Please provide a detailed written description of what llama-cpp-python did, instead.

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

• Physical (or virtual) hardware you are using, e.g. for Linux:
  Architecture: x86_64
  CPU op-mode(s): 32-bit, 64-bit
  Address sizes: 39 bits physical, 48 bits virtual
  Byte Order: Little Endian
  CPU(s): 16
  On-line CPU(s) list: 0-15
  Vendor ID: GenuineIntel
  Model name: 13th Gen Intel(R) Core(TM) i7-13620H
  CPU family: 6
  Model: 186
  Thread(s) per core: 2
  Core(s) per socket: 10
  Socket(s): 1
  Stepping: 2
  CPU max MHz: 4900.0000
  CPU min MHz: 400.0000
  BogoMIPS: 5836.80
  Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clf
  lush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm c
  onstant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc c
  puid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx
  est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc
  _deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_f
  ault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpi
  d ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx sm
  ap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_loc
  k_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp
  _epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid
  movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capa
  bilities
  Virtualization features:
  Virtualization: VT-x
  Caches (sum of all):
  L1d: 416 KiB (10 instances)
  L1i: 448 KiB (10 instances)
  L2: 9.5 MiB (7 instances)
  L3: 24 MiB (1 instance)
  NUMA:
  NUMA node(s): 1
  NUMA node0 CPU(s): 0-15
  Vulnerabilities:
  Gather data sampling: Not affected
  Itlb multihit: Not affected
  L1tf: Not affected
  Mds: Not affected
  Meltdown: Not affected
  Mmio stale data: Not affected
  Retbleed: Not affected
  Spec rstack overflow: Not affected
  Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB
  -eIBRS SW sequence
  Srbds: Not affected
  Tsx async abort: Not affected

• Operating System, e.g. for Linux:
  Ubuntu 22.04

• SDK version, e.g. for Linux:

python = 3.11.7
GNU Make 4.3
g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. step 1
2. step 2
3. step 3
4. etc.

Note: Many issues seem to be regarding functional or performance issues / differences with llama.cpp. In these cases we need to confirm that you're comparing against the version of llama.cpp that was built with your python package, and which parameters you're passing to the context.

Try the following:

1. git clone https://github.com/abetlen/llama-cpp-python
2. cd llama-cpp-python
3. rm -rf _skbuild/ # delete any old builds
4. python -m pip install .
5. cd ./vendor/llama.cpp
6. Follow llama.cpp's instructions to cmake llama.cpp
7. Run llama.cpp's ./main with the same arguments you previously passed to llama-cpp-python and see if you can reproduce the issue. If you can, log an issue with llama.cpp

Failure Logs

Please include any relevant log snippets or files. If it works under one configuration but not under another, please provide logs for both configurations and their corresponding outputs so it is easy to see where behavior changes.

Also, please try to avoid using screenshots if at all possible. Instead, copy/paste the console output and use Github's markdown to cleanly format your logs for easy readability.

Example environment info:

llama-cpp-python$ git log | head -1
commit 47b0aa6e957b93dbe2c29d53af16fbae2dd628f2

llama-cpp-python$ python3 --version
Python 3.10.10

llama-cpp-python$ pip list | egrep "uvicorn|fastapi|sse-starlette|numpy"
fastapi                  0.95.0
numpy                    1.24.3
sse-starlette            1.3.3
uvicorn                  0.21.1

llama-cpp-python/vendor/llama.cpp$ git log | head -3
commit 66874d4fbcc7866377246efbcee938e8cc9c7d76
Author: Kerfuffle <44031344+KerfuffleV2@users.noreply.github.com>
Date:   Thu May 25 20:18:01 2023 -0600

[ndavidson19]

Same issue as above. Gemma is supported in llama.cpp as of this morning ggml-org/llama.cpp@580111d

[obriensystems]

Thanks guys, 7B (32G model needs 64G on a CPU or a RTX-A6000/RTX-5000 Ada) and 2B (on a macbook M1Max:32G unified ram - working perfectly
https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf
https://blog.google/technology/developers/gemma-open-models/
https://obrienlabs.medium.com/google-gemma-7b-and-2b-llm-models-are-now-available-to-developers-as-oss-on-hugging-face-737f65688f0d

obrien@mbp7 llama.cpp % ./main -m models/gemma-2b.gguf -p "Describe how gold is made in collapsing stars" -t 24 -n 1000 -e --color 
Log start
main: build = 2234 (973053d8)
main: built with Apple clang version 15.0.0 (clang-1500.1.0.2.5) for arm64-apple-darwin23.3.0
main: seed  = 1708573311
llama_model_loader: loaded meta data with 19 key-value pairs and 164 tensors from models/gemma-2b.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
...
ggml_backend_metal_buffer_from_ptr: allocated buffer, size =  9561.31 MiB, ( 9561.38 / 21845.34)
llm_load_tensors: offloading 18 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 19/19 layers to GPU
llm_load_tensors:      Metal buffer size =  9561.30 MiB
llm_load_tensors:        CPU buffer size =  2001.00 MiB
.............................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Max
ggml_metal_init: picking default device: Apple M1 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil


llama_print_timings:        load time =   10956.18 ms
llama_print_timings:      sample time =     650.20 ms /  1000 runs   (    0.65 ms per token,  1537.98 tokens per second)
llama_print_timings: prompt eval time =      55.43 ms /     9 tokens (    6.16 ms per token,   162.36 tokens per second)
llama_print_timings:        eval time =   32141.38 ms /   999 runs   (   32.17 ms per token,    31.08 tokens per second)
llama_print_timings:       total time =   33773.63 ms /  1008 tokens
ggml_metal_free: deallocating

[snowclipsed]

@obriensystems can you try running a quantized model?

https://huggingface.co/sayhan/gemma-2b-it-GGUF-quantized

[abetlen]

@snowclipsed I think it's just those quantizations and something was fixed in the last few hours. LMStudio's quantized models work for me.

import llama_cpp

repo_id = "lmstudio-ai/gemma-2b-it-GGUF"

llama = llama_cpp.Llama.from_pretrained(
    repo_id=repo_id,
    filename="gemma-2b-it-q4_k_m.gguf",
    n_gpu_layers=-1,
    verbose=False
)

response = llama.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": "What is the capital of France?"
        }
    ],
    response_format={
        "type": "json_object",
        "schema": {
            "type": "object",
            "properties": {
                "country": {"type": "string"},
                "capital": {"type": "string"}
            },
            "required": ["country", "capital"],
        }
    },
    stream=True
)

for chunk in response:
    delta = chunk["choices"][0]["delta"]
    if "content" not in delta:
        continue
    print(delta["content"], end="", flush=True)
print()

Above should work with the most recent version.

[snowclipsed]

Got it working with the latest, thanks!

[SleepyYui]

@snowclipsed I think it's just those quantizations and something was fixed in the last few hours. LMStudio's quantized models work for me.

import llama_cpp

repo_id = "lmstudio-ai/gemma-2b-it-GGUF"

llama = llama_cpp.Llama.from_pretrained(
    repo_id=repo_id,
    filename="gemma-2b-it-q4_k_m.gguf",
    n_gpu_layers=-1,
    verbose=False
)

response = llama.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": "What is the capital of France?"
        }
    ],
    response_format={
        "type": "json_object",
        "schema": {
            "type": "object",
            "properties": {
                "country": {"type": "string"},
                "capital": {"type": "string"}
            },
            "required": ["country", "capital"],
        }
    },
    stream=True
)

for chunk in response:
    delta = chunk["choices"][0]["delta"]
    if "content" not in delta:
        continue
    print(delta["content"], end="", flush=True)
print()

Above should work with the most recent version.
I had a similar error, (only when having the schema object inside response_format set) and tried running that script on my system.

got the following output:

root@system:~/llama# ./venv/bin/python test.py 
gemma-2b-it-q4_k_m.gguf: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.50G/1.50G [00:41<00:00, 35.6MB/s]
terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check: __n (which is 9766) >= this->size() (which is 0)
Aborted

when running again, I get nothing but a simple Segmentation fault (verbose: https://hastebin.com/share/gutizutizo )

Is there anything I'm missing?