Skip to content

Conversation

@ngxson
Copy link
Collaborator

@ngxson ngxson commented Dec 1, 2025

Supersede #17638

If user wants to warmup with a different image size, set warmup = false, and send one image with the desired size right after mtmd initialized for manual warmup

warmup = false can be set via the existing --no-warmup CLI argument

@ngxson ngxson requested a review from ggerganov as a code owner December 1, 2025 11:40
Copy link
Member

@ggerganov ggerganov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably more like a "reserve worst case" rather than a "warmup" if I understand correctly.

Does the warmup in CLIP do any actual ggml graph computation?

@ngxson
Copy link
Collaborator Author

ngxson commented Dec 1, 2025

Does the warmup in CLIP do any actual ggml graph computation?

It doesn't run the computation, but it allocate enough memory for cgraph.

For context: Ideally, the memory will be allocated for the worst case (max image size), but this doesn't work very well as some models support maximum image size that worth up several GB of memory for cgraph. Therefore, we have to set an arbitrary "ideal case" for this, hence set_warmup_n_tokens function.

This was done mostly to improve UX as we don't either want to limit the max image size to an arbitrary number.But we should improve this in the future.

Also, during the warmup, we also scan the cgraph for ops that are not supported by GPU.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants