Google Gemma 7B 2B OSS models are available on Hugging Face as of 20240221

[obriensystems]

see #27
https://ai.google.dev/gemma/docs?hl=en
https://www.kaggle.com/models/google/gemma

Gemma on Vertex AI Model garden
https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/335?_ga=2.34476193.-1036776313.1707424880&hl=en

https://obrienlabs.medium.com/google-gemma-7b-and-2b-llm-models-are-now-available-to-developers-as-oss-on-hugging-face-737f65688f0d

https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf
https://blog.google/technology/developers/gemma-open-models/

https://huggingface.co/google/gemma-7b
https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/l4/PB-11316-001_v01.pdf

pull and remake the latest llama.cpp (see previous article running llama 70b - #7

abetlen/llama-cpp-python#1207
ggml-org/llama.cpp@580111d

7B (32G model needs 64G on a CPU or a RTX-A6000/RTX-5000 Ada) and 2B (on a macbook M1Max:32G unified ram - working perfectly

obrien@mbp7 llama.cpp % ./main -m models/gemma-2b.gguf -p "Describe how gold is made in collapsing stars" -t 24 -n 1000 -e --color 
Log start
main: build = 2234 (973053d8)
main: built with Apple clang version 15.0.0 (clang-1500.1.0.2.5) for arm64-apple-darwin23.3.0
main: seed  = 1708573311
llama_model_loader: loaded meta data with 19 key-value pairs and 164 tensors from models/gemma-2b.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
...
ggml_backend_metal_buffer_from_ptr: allocated buffer, size =  9561.31 MiB, ( 9561.38 / 21845.34)
llm_load_tensors: offloading 18 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 19/19 layers to GPU
llm_load_tensors:      Metal buffer size =  9561.30 MiB
llm_load_tensors:        CPU buffer size =  2001.00 MiB
.............................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Max
ggml_metal_init: picking default device: Apple M1 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil


llama_print_timings:        load time =   10956.18 ms
llama_print_timings:      sample time =     650.20 ms /  1000 runs   (    0.65 ms per token,  1537.98 tokens per second)
llama_print_timings: prompt eval time =      55.43 ms /     9 tokens (    6.16 ms per token,   162.36 tokens per second)
llama_print_timings:        eval time =   32141.38 ms /   999 runs   (   32.17 ms per token,    31.08 tokens per second)
llama_print_timings:       total time =   33773.63 ms /  1008 tokens
ggml_metal_free: deallocating

https://cloud.google.com/blog/products/ai-machine-learning/performance-deepdive-of-gemma-on-google-cloud

[obriensystems]

i9-13900KS running dual RTX-A4500 (20+20G) Ampere and i9-14900K running dual RTX-4090 (24+24G) Ada

CPU first

C:/wse_github/llama.cpp $  ./main.exe -m g:/models/gemma-7b.gguf  -p "what partion of gold is made in exploding stars" -n 2000 -e --color -t 24

Log start
main: build = 2234 (973053d8)
main: built with cc (GCC) 13.2.0 for x86_64-w64-mingw32
main: seed  = 1708573388
llama_model_loader: loaded meta data with 19 key-value pairs and 254 tensors from g:/models/gemma-7b.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma
llama_model_loader: - kv   1:                               general.name str              = gemma-7b
llama_model_loader: - kv   2:                       gemma.context_length u32              = 8192
llama_model_loader: - kv   3:                          gemma.block_count u32              = 28
llama_model_loader: - kv   4:                     gemma.embedding_length u32              = 3072
llama_model_loader: - kv   5:                  gemma.feed_forward_length u32              = 24576
llama_model_loader: - kv   6:                 gemma.attention.head_count u32              = 16
llama_model_loader: - kv   7:              gemma.attention.head_count_kv u32              = 16
llama_model_loader: - kv   8:                 gemma.attention.key_length u32              = 256
llama_model_loader: - kv   9:               gemma.attention.value_length u32              = 256
llama_model_loader: - kv  10:     gemma.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  12:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  13:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  14:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  15:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,256128]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  17:                      tokenizer.ggml.scores arr[f32,256128]  = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  18:                  tokenizer.ggml.token_type arr[i32,256128]  = [3, 3, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - type  f32:  254 tensors
llm_load_vocab: mismatch in special tokens definition ( 544/256128 vs 388/256128 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = gemma
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 256128
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 3072
llm_load_print_meta: n_head           = 16
llm_load_print_meta: n_head_kv        = 16
llm_load_print_meta: n_layer          = 28
llm_load_print_meta: n_rot            = 192
llm_load_print_meta: n_embd_head_k    = 256
llm_load_print_meta: n_embd_head_v    = 256
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 4096
llm_load_print_meta: n_embd_v_gqa     = 4096
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 24576
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = all F32 (guessed)
llm_load_print_meta: model params     = 8.54 B
llm_load_print_meta: model size       = 31.81 GiB (32.00 BPW)
llm_load_print_meta: general.name     = gemma-7b
llm_load_print_meta: BOS token        = 2 '<bos>'
llm_load_print_meta: EOS token        = 1 '<eos>'
llm_load_print_meta: UNK token        = 3 '<unk>'
llm_load_print_meta: PAD token        = 0 '<pad>'
llm_load_print_meta: LF token         = 227 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.10 MiB
llm_load_tensors:        CPU buffer size = 32570.17 MiB
......................................................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:        CPU KV buffer size =   224.00 MiB
llama_new_context_with_model: KV self size  =  224.00 MiB, K (f16):  112.00 MiB, V (f16):  112.00 MiB
llama_new_context_with_model:        CPU input buffer size   =     8.01 MiB
llama_new_context_with_model:        CPU compute buffer size =   506.25 MiB
llama_new_context_with_model: graph splits (measure): 1

system_info: n_threads = 24 / 32 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
sampling:
        repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000
        top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 512, n_batch = 512, n_predict = 2000, n_keep = 1


 what partion of gold is made in exploding stars.

We have seen that there are very strong indications, both from meteoritic data and terrestrial rocks (see Chapter 8), that the Earth has experienced a bombardment by large bodies (planetesimals) during the first few million years after its formation $460 \times$ $\left(1-\delta_{\mathrm{E}}\right)$ ago. These planetensimals were largely made of silicates and iron, but they contained less metal than is found in meteorites today: their meteoritic equivalent was the enstatite chondrite with a low amount ( $9 \%$ by mass) metallic FeNi alloy; this type has been identified from both lunar rock fragments and martian samples. Thus these projectiles were largely made of silicates, but they contained 15\% metal compared to $\sim 30-42$ vol.\% for the present terrestrial core: therefore their impact on a young Earth must have produced an important amount (at least $~ \frac{8}{6}$ ) metallic FeNi alloy.

The total mass of these projectiles, which is estimated from both lunar rocks and martian samples to be $\sim 05$ M $_{E}$, implies that the fraction ejected into space by their impact was between a half $(1 /:)$ or two thirds (2/3) depending on the initial amount in FeNi alloy. Thus our Moon represents $7 \%$ of these projectiles, while Mars may have formed as much as one third $(\sim 4 \%)$. This last figure is to be compared with that found for planetary formation by dynamical processes which indicates a mass ratio between Earth and its moon at most equal

<h1>CHAPTER $\mathrm{X</h1>
to unity. The fraction ejected into space in FeNi alloy was therefore of the order $02(1 / 3)$, i e., it represents about half $(\sim \delta)$ the total amount of metallic iron, or an average abundance for meteoritic chondrites (see Table X-5). It is interesting to note that this fraction is very similar both to what we have estimated in Chapter VIII from present terrestrial core data and also from meteorites. Thus it appears more probable that a large part of the Earth's metallic FeNi alloy was formed by impact, rather than due solely or mainly because of differentiation during planetary formation (see Fig 10-2).

The other half $(\sim \delta)$ is probably made in stellar interiors where we have seen above the Moon. at as: to to to to to to to once more than $45 on both by by over Earth's surface,,,,,,,,,,,,,,,,, but but but when it came be expected from during its formation is is may indicate that purely theoretical reason















 once. discretely and stimated to to to to to to to to in in in in in in in in in in in once more than 554 on on over Earth's a fine dust, mass the size of once they are small at any amount about $234 and as formed from above one- purely theoretical estimate is forこの日 twice its possible that hereto form $\quad$ formation. strictly speaking which was first very much larger in under an extremely large estimated fraction

 on over Earth's slightly greater than a mass may have been the average once or at at any more of just barely bigger $0 within 1 to to amongst it seems highly about one another small amount formed during its possible origin for each and (a great deal less is not yet further our very much larger. Thus far above all $\quad$ strictly speaking as we still greater than most probably in the average while a mes quite big. once a mere theory ofRDONLY more... an an example, thereunder weight half that hereto form 1mipmap or on favourably huge new super- once andANNES ( over its structure size thatpelier one extremely heavy duty earth large beyond where it is almost double how very much larger than life itself the outside world above all but at (()one forividuated several times more slightly heavier.この日, thereunder $\left(((( ( ( ( ( ( ( ( ( ( ( ( ( once again highly probably a just half- purely because ofcreateServer size one and hereto form earth large beyond which had it is mesquite the almost double duty above all mass that within its structure weight very much larger than life itself outside world over twice more heavy slightly heavier. thereunder, (()one was formed somewhat smaller at $0 on once again highly probably greater in forming just about a half- once more heavily inside out whose outer size one greatly beyond which might be the almost double duty earth quite large above all mass formation weight within its structure very much larger than life itself outside world would have (somehow it is slightly heavier. The amount of discretely under(and somewhat formed on had twice more heavy that was in from $\quad$ some amount greater below average (that it is just occurred halfway through and once but not, meselfromore the release . [end of text]
llama_print_timings:        load time =    4247.23 ms
llama_print_timings:      sample time =     859.49 ms /  1030 runs   (    0.83 ms per token,  1198.38 tokens per second)
llama_print_timings: prompt eval time =     693.44 ms /    11 tokens (   63.04 ms per token,    15.86 tokens per second)
llama_print_timings:        eval time =  412311.19 ms /  1029 runs   (  400.69 ms per token,     2.50 tokens per second)
llama_print_timings:       total time =  415226.57 ms /  1040 tokens
Log end

```

[obriensystems]

[obriensystems]

[obriensystems]

Team, thank you for integrating Gemma support into llama.cpp yesterday - this was an extremely fast and efficient alignment with a model that just came out a couple hours before.
I personally am very grateful to your efforts.
A wider community thank you is in order
thank you for
ggml-org/llama.cpp#5631

[fmichaelobrien]

review https://github.com/google/gemma.cpp

[obriensystems]

investigate TensorFlow 2 / keras support
https://developers.googleblog.com/2024/02/gemma-models-in-keras.html
https://ai.google.dev/gemma/docs/distributed_tuning

pip install -U torch
pip install -U transformers

[obriensystems]

gemma-7b model on dual RTX-4090 suprim liquid with 800W max and 2 x 24G = 48G vram

The model runs at 20% TDP or 100+100W because I am sharing the model across the PCIe bus at 8x which saturates it up to 75% at 8 x 2GBps or = 12GBps as opposed to NVlink on ampere cards at 112GBps

checking context length
Using the model-agnostic default max_length (=20) to control the generation length

outputs = model.generate(**input_ids, max_new_tokens=1000)

from transformers import AutoTokenizer, AutoModelForCausalLM

access_token='hf_cfT...QqH'

tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b", token=access_token)
model = AutoModelForCausalLM.from_pretrained("google/gemma-7b", device_map="auto", token=access_token)

input_text = "how is gold made in collapsing neutron stars - specifically what is the ratio created during the beta and r process."
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=3000)
print(tokenizer.decode(outputs[0]))

python pip summary

332  cd machine-learning/
  335  mkdir gemma
  337  vi gemma-cpu.py
  339  pip install -U transformers
  352  pip install -U torch
  353  python gemma-cpu.py
  355  nvcc --version
  364  pip install accelerate
  366  pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
  368  python gemma-gpu.py

run

michael@13900b MINGW64 /c/wse_github/obrienlabsdev/machine-learning/gemma (main)
$ python gemma-gpu.py
Loading checkpoint shards: 100%|████████████████████████████████████| 4/4 [00:06<00:00,  1.72s/it]
C:\Users\michael\AppData\Roaming\Python\Python311\site-packages\transformers\models\gemma\modeling_gemma.py:555: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.)
  attn_output = torch.nn.functional.scaled_dot_product_attention(
<bos>how is gold made in collapsing neutron stars - specifically what is the ratio created during the beta and r process.

Answer:

Step 1/2
First, when a neutron star collapses, it undergoes a process called gravitational collapse, which causes the star to rapidly lose mass and density. This process releases a tremendous amount of energy, which can cause the star to explode in a supernova. During the supernova, the star's core undergoes a process called the r-process, which is responsible for creating heavy elements like gold. The r-process occurs when neutrons are added to atomic nuclei, causing them to become unstable and undergo beta decay. This process continues until the nucleus reaches a stable state, which is usually a heavy element like gold.

Step 2/2
The ratio of gold created during the r-process is not well understood, as it depends on a variety of factors, including the mass and density of the star, the amount of energy released during the supernova, and the specific conditions of the r-process. However, it is believed that the r-process is responsible for creating most of the heavy elements in the universe, including gold.<eos>

[obriensystems]