docs/integrations/chat/google_vertex_ai_palm/

[giscus[bot]]

docs/integrations/chat/google_vertex_ai_palm/

This page provides a quick overview for getting started with VertexAI chat models. For detailed documentation of all ChatVertexAI features and configurations head to the API reference.

https://python.langchain.com/docs/integrations/chat/google_vertex_ai_palm/

[maryshgh]

What format should I use to invoke for multimodal use cases? I used the following but did not work:
llm = ChatVertexAI(
model_name='gemini-1.5-pro-001',
temperature=0,
max_output_tokens=8192,
top_p=1.0,
top_k=1,
safety_settings={
HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_LOW_AND_ABOVE
},
convert_system_message_to_human=True
)

model = primary_assistant_prompt | llm.bind_tools(tools)

message = HumanMessage(
content=[
{"type": "text", "text": "what is the fifth step in the third page of this document"},
{"type": "image_url", "image_url": {"url": 'sample_files/slide_sample.pdf'}},
],
)
response = model.invoke([message])

[codeananda]

Go to langchain_google_vertexai/chat_models.py and the _parse_chat_history_gemini method. That is the bit that parses the message and you'll see the options.

type can be one of

1. 'text'
2. 'tool_use'
3. 'image_url'
4. 'media'

Looks like you can pass anything as long as it's bytes. So to pass a pdf document for example, do

from langchain_core.messages import HumanMessage
from langchain_google_vertexai import ChatVertexAI

my_pdf = 'path/to/pdf.pdf'
with open(my_pdf, "rb") as pdf_file:
        pdf_bytes = pdf_file.read()

message = HumanMessage(
    content=[
        {
            "type": "media",
            "mime_type": "application/pdf",
            "data": pdf_bytes
        },
        {
            "type": "text",
            "text": f"Please summarise this document"
        },
    ],
)

llm = ChatVertexAI(model='gemini-2.0-flash')
response = llm.invoke([message])
print(response.content)

[codeananda]

@maryshgh

[codeananda]

For images

If it's an image url like this you can use "type": "image_url"

If you want to process a local image do

message = HumanMessage(
    content=[
        {
            "type": "media",
            "mime_type": "image/png",  # or image/jpeg
            "data": image_as_bytes
        },
        {
            "type": "text",
            "text": f"Please summarise this document"
        },
    ],
)