Skip to content

clip : remove boi/eoi embeddings for GLM-edge model (⚠️ breaking change) #13081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 24, 2025

Conversation

ngxson
Copy link
Collaborator

@ngxson ngxson commented Apr 23, 2025

GLM-edge model adds <|begin_of_image|> and <|end_of_image|> as part of the image embedding, this was a hack because at the time of #10573 , special tokens are not supported by llama-llava-cli

This removes the mentioned hack, making the support of special tokens become more standard

Important

BREAKING CHANGE: If you are still using libllava, you will need to add these tokens in your chat template.
If you are using libmtmd, no actions are required.

Test result:

OK:   llama-mtmd-cli ggml-org/SmolVLM-500M-Instruct-GGUF:Q8_0
OK:   llama-mtmd-cli ggml-org/SmolVLM2-2.2B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/SmolVLM2-500M-Video-Instruct-GGUF:Q8_0
OK:   llama-mtmd-cli ggml-org/gemma-3-4b-it-GGUF:Q4_K_M
OK:   llama-mtmd-cli guinmoon/MobileVLM-3B-GGUF:Q4_K_M
OK:   llama-mtmd-cli THUDM/glm-edge-v-5b-gguf:Q4_K_M
OK:   llama-mtmd-cli second-state/Llava-v1.5-7B-GGUF:Q2_K
OK:   llama-mtmd-cli cjpais/llava-1.6-mistral-7b-gguf:Q3_K
OK:   llama-mtmd-cli ibm-research/granite-vision-3.2-2b-GGUF:Q4_K_M
OK:   llama-mtmd-cli second-state/MiniCPM-Llama3-V-2_5-GGUF:Q2_K
OK:   llama-mtmd-cli openbmb/MiniCPM-V-2_6-gguf:Q2_K
OK:   llama-mtmd-cli openbmb/MiniCPM-o-2_6-gguf:Q4_0
OK:   llama-qwen2vl-cli bartowski/Qwen2-VL-2B-Instruct-GGUF:Q4_K_M

@ngxson ngxson requested a review from ggerganov April 23, 2025 21:07
@ngxson ngxson changed the title clip : remove boi/eoi embeddings for GLM-edge model clip : remove boi/eoi embeddings for GLM-edge model (⚠️ breaking change) Apr 23, 2025
@ngxson ngxson added the breaking change Changes that break ABIs, APIs, file formats, or other forms of backwards compatibility. label Apr 23, 2025
@ngxson ngxson mentioned this pull request Apr 23, 2025
18 tasks
@ngxson ngxson merged commit 13be08d into ggml-org:master Apr 24, 2025
48 checks passed
pockers21 pushed a commit to pockers21/llama.cpp that referenced this pull request Apr 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking change Changes that break ABIs, APIs, file formats, or other forms of backwards compatibility. examples
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants