Smoldocling support #14597

ryan-mangeno · 2025-07-09T14:17:23Z

added end of token check for smoldocling - <end_of_utterance>, also added tensor names after dumping tensors with llama-eval-callback to tensor_mappings.py, added regex for pre tokenizer defined in the hugging face config file. Smoldocling text model architecture is based on llama, so there wasn't a need to impliment its own architecture. I compared both the hugging face tensors and the gguf tensors and both checked out to be the same

gabe-l-hart

Thank you for tackling this one @ryan-mangeno! A couple of whitespace NITs, and one request since you're coming out of IBM with me: Our OSS policy requires that we sign our commits when committing to open source. To do this for your branch, you should be able to simply do:

git fetch origin && git rebase -i --signoff --rebase-merges origin/master

gguf-py/gguf/tensor_mapping.py

gabe-l-hart · 2025-07-09T14:42:31Z

cc @PeterStaar-IBM

Co-authored-by: Gabe Goodhart <gabe.l.hart@gmail.com>

Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

* v1 * push more fixes * another fix * fix * more fixes * minor fix * more cleaning on python code * python fixes * changed precision for multipliers float 32->64 * fixes * another fix * fix * pre-norm -> norm * fix * Revert "fix" This reverts commit 243e4d1. * fix * small fix ffn_norm * try * mix instead of max * fix vocab size * conflict solve * fixed multipliers * falcon-h1 specefic vocab resolved * read arch from gguf.MODEL_ARCH * mamba_d_ssm added to d_inner find_hparam * remove unused functions from gguf_writer.py * override modify_tensors instead of get_tensors * fix conversion and d_inner * added some cb functions for debugging puposes * inp_out_ids moved outside of layers loop * mup_vec create as float64 * fix rope_theta * injected mup * clean ups * rm extra space * rm unused MAMBA_CHUNK_SIZE * rm unused key * add bos False * changed ROPE_TYPE * cleaning debugging stuff * cleaning debug quant * fix comment * some cleanups * some cleanups * Update src/llama-model-loader.cpp * more cleanups * moe cleanuips * d_ssm -> d_inner; * cleaning unused hparams * cleanup * more cleanups * more cleanups on python conversion; * minor cleanups * Apply suggestions from code review Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * remove todo * added falcon-h1 * tensor not required * clean * remove unneeded attributes * more cleanups and fixed conversion * remove final_norm * flake8 fixes * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * flake8 fixes * Update src/llama-hparams.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-arch.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * added hashes * Update src/llama-arch.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update src/llama-vocab.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * update the update file * Revert "update the update file" This reverts commit 082ab4a. * fix: address suggestions * fix: update convert_hf_to_gguf.py * Update gguf-py/gguf/constants.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model-loader.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * d_inner fixed * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * reshaping ssm_norm for 34B * removing generate_mup * remove duplicates metadata keys * rm comment * final comment * fix unused args * fix constants * fix bad merge * Update src/llama-model.cpp Co-authored-by: compilade <git@compilade.net> * falcon-h1: remove unused ssm_in_b and bad merge * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * falcon-h1: fix last comment * Update convert_hf_to_gguf.py Co-authored-by: compilade <git@compilade.net> * falcon-h1: revert add_add_bos(False) * falcon-h1: fix tied weights * falcon-h1: remove whitespace * falcon-h1: fix wrong size param * falcon-h1: fix whitespace issues --------- Co-authored-by: younesbelkada <younes.belkada@tii.ae> Co-authored-by: Younes B <49240599+younesbelkada@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: compilade <git@compilade.net> Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

…-org#14595) Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

convert_hf_to_gguf_update.py

gguf-py/gguf/tensor_mapping.py

src/llama-model.h

src/llama-vocab.cpp

ngxson

Unless I missed something, SmolDocling-256M-preview is fine tuned from smolVLM, so it should work out of the box. Are you 100% sure that we need to modify the conversion script?

gguf-py/gguf/tensor_mapping.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

ryan-mangeno · 2025-07-09T23:47:04Z

I changed the tensormappings for the safetensors file on hugging face, also all the tensors are the same for smolvlm and smoldocling but noticed that there werent tensor mappings for the text model for smolvlm ... https://huggingface.co/ds4sd/SmolDocling-256M-preview?show_file_info=model.safetensors. I noticed when I didn't have || t.first == "<end_of_utterance>" // smoldocling that it would just keep generating

src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

…pp into smoldocling-support

convert_hf_to_gguf.py

gguf-py/gguf/tensor_mapping.py

convert_hf_to_gguf.py

convert_hf_to_gguf_update.py

include/llama.h

src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

CISC · 2025-07-10T14:35:00Z

Please re-check that conversion and inference is ok.

ryan-mangeno · 2025-07-10T14:37:35Z

Please re-check that conversion and inference is ok.

will do right now 👍

CISC · 2025-07-10T14:39:33Z

Please re-check that conversion and inference is ok.

will do right now 👍

Actually, according to this, you should not need to add most of the tensor mappings either:

llama.cpp/convert_hf_to_gguf.py

Lines 1963 to 1964 in 4a5686d

    
           elif name.startswith("model.text_model"): 
        
               name = name.replace("text_model.", "") # for SmolVLM

ryan-mangeno · 2025-07-10T14:41:24Z

Please re-check that conversion and inference is ok.

will do right now 👍

Actually, according to this, you should not need to add most of the tensor mappings either:

llama.cpp/convert_hf_to_gguf.py

Lines 1963 to 1964 in 4a5686d

elif name.startswith("model.text_model"):

name = name.replace("text_model.", "") # for SmolVLM

ahh ok that makes sense, the tensor names were the same for the text model so those can be removed I am pretty sure

gguf-py/gguf/tensor_mapping.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

ryan-mangeno · 2025-07-10T14:53:26Z

I reran conversion and inferencing and looks the same

CISC · 2025-07-10T14:59:31Z

Thank you, hope you enjoyed the ride, will merge when CI goes green. :)

ryan-mangeno · 2025-07-10T15:03:47Z

Thank you too! I had a lot of fun making my first contribution to llama cpp and hope to make more :)

* origin/master: Smoldocling support (ggml-org#14597) Docs: script to auto-generate ggml operations docs (ggml-org#14598)

ryan-mangeno and others added 3 commits July 8, 2025 16:35

support for smoldocling

61cfad0

fixed merge conflicts

8c18417

Merge branch 'master' into smoldocling-support

2b209cd

gabe-l-hart suggested changes Jul 9, 2025

View reviewed changes

gguf-py/gguf/tensor_mapping.py Outdated Show resolved Hide resolved

gguf-py/gguf/tensor_mapping.py Outdated Show resolved Hide resolved

gguf-py/gguf/tensor_mapping.py Outdated Show resolved Hide resolved

github-actions bot added the python python script changes label Jul 9, 2025

ryan-mangeno and others added 15 commits July 9, 2025 12:10

Update gguf-py/gguf/tensor_mapping.py

c36703a

Co-authored-by: Gabe Goodhart <gabe.l.hart@gmail.com>

Update gguf-py/gguf/tensor_mapping.py

6ea3b0b

Co-authored-by: Gabe Goodhart <gabe.l.hart@gmail.com>

merge conflicts

7e16cb8

merge conflicts

f4a6fa8

pre tokenizer merge fix

5050f27

convert : fix smollm3 jinja template (ggml-org#14586)

fbfcdae

Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

llama : remove unintended whitespace (ggml-org#14592)

00d2fc2

Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

model : add skt/A.X-4.0 model vocabulary (ggml-org#14589)

e1b0b9e

Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

support for smoldocling

ea265fe

Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

ggml : prevent integer overflow in gguf tensor size calculation (ggml…

529552d

…-org#14595) Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

fixed merge conflicts

996c578

Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

rebase and signing

6a3ed2e

Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

rebase and signing

bd137ff

Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

Merge branch 'master' into smoldocling-support

dd6d36b

CISC requested changes Jul 9, 2025

View reviewed changes

ngxson reviewed Jul 9, 2025

View reviewed changes

gguf-py/gguf/tensor_mapping.py Outdated Show resolved Hide resolved

ryan-mangeno and others added 5 commits July 9, 2025 19:14

Update src/llama-vocab.cpp

c971ea8

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Update gguf-py/gguf/tensor_mapping.py

ca3cc83

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Update gguf-py/gguf/tensor_mapping.py

b9b532c

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Update src/llama-model.h

376ea47

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

safetensors tensor mapping

5856f8c

Signed-off-by: ryan-mangeno <ryanmangeno@gmail.com>

compilade reviewed Jul 10, 2025

View reviewed changes

src/llama-vocab.cpp Outdated Show resolved Hide resolved

added back accidental removal of clean spaces for hunyuan

5c3d948

CISC reviewed Jul 10, 2025

View reviewed changes

src/llama-vocab.cpp Outdated Show resolved Hide resolved

ryan-mangeno and others added 3 commits July 10, 2025 09:28

Update src/llama-vocab.cpp

69fa672

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

updated hash and reordererd model list

f5e4d56

Merge branch 'smoldocling-support' of github.com:ryan-mangeno/llama.c…

e47dc34

…pp into smoldocling-support

CISC reviewed Jul 10, 2025

View reviewed changes

convert_hf_to_gguf.py Outdated Show resolved Hide resolved

CISC requested changes Jul 10, 2025

View reviewed changes

ryan-mangeno and others added 7 commits July 10, 2025 10:27

Update gguf-py/gguf/tensor_mapping.py

037dbbc

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Update src/llama-vocab.cpp

4331939

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Update include/llama.h

661abf5

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Update convert_hf_to_gguf.py

a3be53c

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Update convert_hf_to_gguf_update.py

859005d

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Update src/llama-vocab.cpp

db54ea5

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

removed old tensor name

f021d19

removed tensor mappings -> handled by smolvlm

3f4c532

CISC requested changes Jul 10, 2025

View reviewed changes

gguf-py/gguf/tensor_mapping.py Outdated Show resolved Hide resolved

gguf-py/gguf/tensor_mapping.py Outdated Show resolved Hide resolved

gguf-py/gguf/tensor_mapping.py Outdated Show resolved Hide resolved

ryan-mangeno and others added 3 commits July 10, 2025 10:51

Update gguf-py/gguf/tensor_mapping.py

310eb46

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Update gguf-py/gguf/tensor_mapping.py

bcf4ad3

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Update gguf-py/gguf/tensor_mapping.py

a4f662d

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

CISC approved these changes Jul 10, 2025

View reviewed changes

gabe-l-hart approved these changes Jul 10, 2025

View reviewed changes

CISC merged commit 4bb625b into ggml-org:master Jul 10, 2025
89 of 90 checks passed

gabe-l-hart added a commit to gabe-l-hart/llama.cpp that referenced this pull request Jul 10, 2025

Merge remote-tracking branch 'origin/master' into GraniteFour

d7d5b01

* origin/master: Smoldocling support (ggml-org#14597) Docs: script to auto-generate ggml operations docs (ggml-org#14598)

Smoldocling support #14597

Smoldocling support #14597

Conversation

ryan-mangeno commented Jul 9, 2025

Uh oh!

gabe-l-hart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gabe-l-hart commented Jul 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ryan-mangeno commented Jul 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CISC commented Jul 10, 2025

Uh oh!

ryan-mangeno commented Jul 10, 2025

Uh oh!

CISC commented Jul 10, 2025

Uh oh!

ryan-mangeno commented Jul 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryan-mangeno commented Jul 10, 2025

Uh oh!

CISC commented Jul 10, 2025

Uh oh!

ryan-mangeno commented Jul 10, 2025

Uh oh!

Uh oh!

Uh oh!