ChatLlamaCpp doesn't seem to return logprobs.

[chaoyupeng]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from typing import TypedDict, Optional
from langchain_core.messages import BaseMessage
from langchain_community.chat_models import ChatLlamaCpp
from langgraph.graph import StateGraph, START, END
from langchain_core.messages import HumanMessage
from langchain_core.callbacks import CallbackManager, StreamingStdOutCallbackHandler
from langchain_core.prompts import PromptTemplate
from langchain.callbacks.base import BaseCallbackHandler
from typing import Dict, List, Any
from langchain_community.llms import LlamaCpp

class TokenProbabilityCallback(BaseCallbackHandler):
    def __init__(self):
        self.token_probs = []
        
    def on_llm_new_token(
        self,
        token: str,
        **kwargs: Any
    ) -> None:
        print(kwargs)
        # print("\n")
        # Store token and its metadata
        self.token_probs.append({
            'token': token,
            # 'log_probs': kwargs['token_logprobs'],
        })
    
    def get_token_probs(self):
        return self.token_probs



token_callback = TokenProbabilityCallback()
streaming_callback = StreamingStdOutCallbackHandler()
callback_manager = CallbackManager([streaming_callback, token_callback])

n_gpu_layers = -1
n_batch = 512

llm1 = ChatLlamaCpp(
    model_path = "./Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf",
    n_gpu_layers=n_gpu_layers,
    n_batch=n_batch,
    streaming=True,
    verbose=True,
    max_tokens = 30,
    n_ctx=2048,
    callbacks=[token_callback],  # Register callback
    logits_all=True,
    logprobs = 5,
)

def format_prompt(human_input: str) -> str:
    """Format the prompt with human and assistant context."""
    return f"""<|im_start|>human
{human_input}<|im_end|>
<|im_start|>assistant
"""

human_message = "who are you"
formatted_prompt = format_prompt(human_message)
response = llm1.invoke(formatted_prompt)

Description

Hi LangChain Community,
I've been trying to use the LlamaCpp and the ChatLlamaCpp class, the idea is to get the log probs for the generated tokens, which is needed for my follow up work.

But I found the following issue:

1. The ChatLlamaCpp class doesn't seem to provide the logprobs even if I set the logits_all to True and the logprobs to 5, I double check the doc and think ChatLlamaCpp does have these two parameters, any idea why is it not in the kwargs?
2. When I tried to use the LlamaCpp class, it does give me the log probs for the tokens, but it doesn't seem to be able to use tools, I tried binding LlamaCpp to a tool, and it doesn't seem to support tool using.

Any idea I could solve this?
The idea is to have the ability to use the logprobs of the predicted tokens, as well as being able to have tool using functionality, do I need to change frameworks or model engines to achieve this?

Thx

System Info

Ubuntu 24.04
Latest LangChain and LangGrpah.