[Model] tool calling support for ibm-granite/granite-20b-functioncalling

[wseaton]

Add tool calling parser support for ibm-granite/granite-20b-functioncalling.

Also adds an example chat template that is based off of the granite fc paper.

[maxdebayser]

According to the paper, the system prompt used for the end-to-end scenario where the model has to explain the tool output is

You are a helpful assistant with access to the following function calls. Your task is to understand the given conversation with function calls and responses and generate natural language response as the ASSISTANT to continue the conversation. You may use the following function calls to understand how to respond to the user query.

In my experiments with the model it worked well, so I perhaps it would be a nice default in this example file.

[wseaton]

Good callout! In my evals, I have mostly been doing zero-shot single turn calls.

Let me update the example template with the more general conversational one since that is probably what most people want.

[njhill]

@wseaton one of the tests is failing and looks most likely related: https://buildkite.com/vllm/fastcheck/builds/3985#0191dd9c-724a-4fbe-99d3-a1e1c94a2106

[wseaton]

@maxdebayser are you still interested in contributing the streaming JSON parser for granite support? I have rebased off of main and might have some bandwidth to work on it, just let me know :)

[maxdebayser]

Hi @wseaton , yes, I've been working on this. There is only one unit test that isn't passing. But I know what the cause is.

[maxdebayser]

@wseaton, now all tests are passing. I've pushed a squashed merge of everything to this draft PR so that you can take a look and perhaps merge with yours: #8915

[wseaton]

@maxdebayser I've cherry picked the commits from your PR, please review!

[maxdebayser]

Thanks a lot @wseaton . There is an extra entry on the unit test model dict, but other than that it looks good to me.

cc @njhill

[wseaton]

@maxdebayser working on fixing merge conflicts w/ upstream now

[wseaton]

@njhill Happy to rebuild my fork of vLLM and re-verify the PR manually on an A100, disabled the test for now

[maxdebayser]

@wseaton I've verified a couple of hours ago. The tool_use tests run successfully on an A100. I have a quantized version of the model that passes the tests and I'm testing it if it can fit in the CI environment.

[njhill]

The test could also be run on a larger GPU, although that's probably a bit wasteful

@maxdebayser yes that's what I thought too, since none of the other model sub-tests require it

[njhill]

Thanks @wseaton! Could you try merging in the latest main branch again? That should hopefully help with the failing CI tests.

[K-Mistele]

@wseaton just a minor point of clarification, this PR is for Granite 20B functioncalling -- is this the same function calling format as the granite 3.0 models? I saw that these were released recently:

• https://huggingface.co/collections/ibm-granite/granite-30-models-66fdb59bbb54785c3512114f
• https://huggingface.co/ibm-granite/granite-3.0-8b-instruct

I'm not as familiar with IBM/granite as with some other families of models. if they are compatible, then that's great and we should indicate that in the docs, as well as name the tool parser granite most likely. If not (and this is what I suspect is the case, based on a brief reading of their docs & github code for agentic stuff), then it might be worth adding a call-out in the docs that this is NOT compatible with granite 3.0

[maxdebayser]

@K-Mistele , they are not compatible. We have another PR for those models but we're waiting for this one to be merged first because there are some code dependencies.

[maxdebayser]

@njhill , @wseaton my tests with a quantized version of the model are now passing. With quantization the model size was reduced to ~20GB (mbayser/granite-20b-functioncalling-FP8-KV), but I still had to add a bunch of flags to reduce memory usage:
"--max_num_seqs", "1", "--max-model-len", "1024", "--enforce-eager", "--cpu-offload-gb", "20"

[wseaton]

Because of the nature of this branch and how long it's been running (with many merge commits from main), I don't feel comfortable doing a rebase to fix the DCO issue. Can it be bypassed to get this merged?

[K-Mistele]

@K-Mistele , they are not compatible. We have another PR for those models but we're waiting for this one to be merged first because there are some code dependencies.

sounds good! Let me know if you'd like for me to take a look at it whenever it's ready :)

[njhill]

Thanks @wseaton @maxdebayser for all of the work on this and thanks @K-Mistele for also reviewing.

[njhill]

@wseaton I'll merge this to unblock the other granite PR, could consider re-enabling the test with @maxdebayser's quantized model as a follow-on update...