GGML_ASSERT: /Users/pp/fiddle/llama.cpp/ggml-alloc.c:116: tensor->data == NULL when trying to train from scratch

[pliablepixels]

Hi there, while trying to train based on the example, I keep getting GGML_ASSERT: /Users/pp/fiddle/llama.cpp/ggml-alloc.c:116: tensor->data == NULL - any thoughts on how I can triage? The models look ok to me. I tried with falcon/llama

• same result. (M2 pro, 96GB memory)

(ml) [pp@pps-2023-MBP:~/fiddle/llama.cpp/build]$ ./bin/train-text-from-scratch         --vocab-model ../models/ggml-vocab-falcon.gguf         --ctx 64 --embd 256 --head 8 --layer 16         --checkpoint-in  chk-shakespeare-256x16-LATEST.gguf         --checkpoint-out chk-shakespeare-256x16-ITERATION.gguf         --model-out ggml-shakespeare-256x16-f32-ITERATION.gguf         --train-data "shakespeare.txt"         -t 6 -b 16 --seed 1 --adam-iter 256         --no-checkpointing
main: seed: 1
llama_model_loader: loaded meta data with 17 key-value pairs and 0 tensors from ../models/ggml-vocab-falcon.gguf (version GGUF V2 (latest))
llama_model_loader: - kv   0:                       general.architecture str
llama_model_loader: - kv   1:                               general.name str
llama_model_loader: - kv   2:                      falcon.context_length u32
llama_model_loader: - kv   3:                  falcon.tensor_data_layout str
llama_model_loader: - kv   4:                    falcon.embedding_length u32
llama_model_loader: - kv   5:                 falcon.feed_forward_length u32
llama_model_loader: - kv   6:                         falcon.block_count u32
llama_model_loader: - kv   7:                falcon.attention.head_count u32
llama_model_loader: - kv   8:             falcon.attention.head_count_kv u32
llama_model_loader: - kv   9:        falcon.attention.layer_norm_epsilon f32
llama_model_loader: - kv  10:                          general.file_type u32
llama_model_loader: - kv  11:                       tokenizer.ggml.model str
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr
llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr
llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32
llm_load_print_meta: format           = GGUF V2 (latest)
llm_load_print_meta: arch             = falcon
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 65024
llm_load_print_meta: n_merges         = 64784
llm_load_print_meta: n_ctx_train      = 2048
llm_load_print_meta: n_embd           = 4544
llm_load_print_meta: n_head           = 71
llm_load_print_meta: n_head_kv        = 1
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_gqa            = 71
llm_load_print_meta: f_norm_eps       = 1.0e-05
llm_load_print_meta: f_norm_rms_eps   = 0.0e+00
llm_load_print_meta: n_ff             = 18176
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = mostly F16
llm_load_print_meta: model params     = 0.00 B
llm_load_print_meta: model size       = 0.00 MiB (nan BPW)
llm_load_print_meta: general.name   = Falcon
llm_load_print_meta: BOS token = 11 '<|endoftext|>'
llm_load_print_meta: EOS token = 11 '<|endoftext|>'
llm_load_print_meta: LF token  = 138 'Ä'
llama_model_load: vocab only - skipping tensors
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
main: init model
GGML_ASSERT: /Users/pp/fiddle/llama.cpp/ggml-alloc.c:116: tensor->data == NULL
Abort trap: 6

Models:

(ml) [pp@pps-2023-MBP:~/fiddle/llama.cpp/build]$ ls -l ../models/
total 101338696
-rw-r--r--@ 1 pp  staff  23237177504 Oct 12 07:35 codellama-34b-instruct.Q5_K_S.gguf
-rw-r--r--@ 1 pp  staff  23838797984 Oct 12 07:32 codellama-34b.Q5_K_M.gguf
-rw-r--r--@ 1 pp  staff   4783256256 Oct 12 06:16 codellama-7b.Q5_K_M.gguf
-rw-r--r--  1 pp  staff      4825676 Oct  9 17:28 ggml-vocab-aquila.gguf
-rw-r--r--  1 pp  staff      2547782 Oct  9 17:28 ggml-vocab-falcon.gguf
-rw-r--r--  1 pp  staff       595423 Oct  9 17:28 ggml-vocab-llama.gguf

[ggerganov]

Fixed via #3618