ChatVertexAI doesn't work with tuned model

[bart-automation]

Tuned model works with VertexAI but not ChatVertexAI:

This works:

from langchain_google_vertexai import VertexAI
llm = VertexAI(model_name="gemini-1.0-pro-002",
    tuned_model_name="projects/XXXXXXXXXXXX/locations/us-west1/endpoints/XXXXXXXXXXXXXXXXXXX",
    max_output_tokens=1024)
result = llm.invoke("test")

Using vertexai directly works:

from vertexai.generative_models import GenerativeModel
model = GenerativeModel(
                "projects/XXXXXXXXXXXX/locations/us-west1/endpoints/XXXXXXXXXXXXXXXXXXX",
                system_instruction=["You are a helpful assistant that translates English to French. Translate the user sentence."]
            )
chat = model.start_chat()
result = chat.send_message(
	["""I love programming."""]
)

Using "ChatVertexAI" results in 404 error:

from langchain_google_vertexai import ChatVertexAI
llm = ChatVertexAI(model_name="gemini-1.0-pro-002",
                    tuned_model_name="projects/XXXXXXXXXXXX/locations/us-west1/endpoints/XXXXXXXXXXXXXXXXXXX",
                    max_output_tokens=1024)
messages = [
(
   "system",
    "You are a helpful assistant that translates English to French. Translate the user sentence.",
),
    ("human", "I love programming."),
]
result = llm.invoke(messages)
result

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
E status = StatusCode.NOT_FOUND
E details = "Endpoint projects/XXXXXXXXXXXX/locations/us-west1/endpoints/XXXXXXXXXXXXXXXXXXX not found."

langchain-google-vertexai: 1.0.8
langchain-core: 0.2.32

[lkuligin]

yes, it's not supported yet, a good feature to be added.

[lkuligin]

@bart-automation could you try the following, please: pass both model_name arg as the original model (the one you tuned), and your tuned endpoint as tuned_model_name?

[bart-automation]

Hi @lkuligin , that is what I did, see my sample code above

[lkuligin]

just tested it, works fine on my side:

from langchain_google_vertexai import ChatVertexAI
llm = ChatVertexAI(model_name="gemini-1.0-pro-002",
                    tuned_model_name=sft_tuning_job.tuned_model_endpoint_name,
                    max_output_tokens=1024)
messages = [
(
   "system",
    "You are a helpful assistant that translates English to French. Translate the user sentence.",
),
    ("human", "I love programming."),
]
result = llm.invoke(messages)
result

I'm sorry for asking, but are you sure you have the latest version of libraries installed and no typos in the endpoint's name?

[bart-automation]

Thanks for testing it, yes I'm using the latest version of langchain-google-vertexai and langchain-core. I also double checked the endpoint name.

Just to confirm, what is the format you are using for the endpoint name?

[lkuligin]

Could you compare two requests, please?

model = GenerativeModel(
                "projects/X/locations/us-central1/endpoints/X",
                system_instruction=["You are a helpful assistant that translates English to French. Translate the user sentence."])
llm = ChatVertexAI(model_name="gemini-1.0-pro-002",
                    tuned_model_name="projects/X/locations/us-central1/endpoints/X")

r = model._prepare_request("test", generation_config={"candidate_count": 1}) 
r1 = llm._prepare_request_gemini(
    messages=[
        SystemMessage(content="You are a helpful assistant that translates English to French. Translate the user sentence."),
        HumanMessage(content="test")])

assert r == r1

[bart-automation]

Yes, both requests are the same