Skip to content

Count image-bearing messages as regular messages in the Gemini adapter - #5910

Open
BlueX888 wants to merge 1 commit into
pipecat-ai:mainfrom
BlueX888:fix/prep-gemini-system-instruction-injected-as-trailing-user-message
Open

BlueX888 wants to merge 1 commit into
pipecat-ai:mainfrom
BlueX888:fix/prep-gemini-system-instruction-injected-as-trailing-user-message

Conversation

@BlueX888

Copy link
Copy Markdown

Please describe the changes in your PR. If it is addressing an issue, please reference that as well.

Summary

The Gemini adapter appended the system instruction as an extra trailing user message when the context held only image-bearing user messages, while the same text was also sent through the system_instruction parameter.

Why

GeminiLLMAdapter._from_universal_context_messages() decides whether to re-append the system prompt as a user message using a check that only counted a message as "regular" when it had exactly one part carrying text (src/pipecat/adapters/services/gemini_adapter.py:400 on main):

has_regular_messages = any(
    msg.parts is not None
    and len(msg.parts) == 1
    and getattr(msg.parts[0], "text", None)
    and not getattr(msg.parts[0], "function_call", None)
    and not getattr(msg.parts[0], "function_response", None)
    for msg in messages
)

A user turn created by LLMContext.create_image_message() (src/pipecat/processors/aggregators/llm_context.py:137) converts to a Content with two parts (text + image), and an image-only turn converts to a single part with inline_data and no text. Neither matched that single-text-part shape, so a context holding only such turns was treated like a function-messages-only context and the system instruction was injected a second time as the conversation's final user message (gemini_adapter.py:410 on main).

Changes

  • A message now counts as regular when any of its parts is neither a function call nor a function response (src/pipecat/adapters/services/gemini_adapter.py:400). The system instruction is still appended as a user message for contexts holding nothing but function calls and responses; text, image, audio, and file parts all count as regular content now.
  • Changelog fragment changelog/5910.fixed.md.

Testing

  • uv run pytest "tests/test_get_llm_invocation_params.py" -k "system_instruction_not_appended_for_image or system_instruction_appended_when_only_function" — 3 passed. On main the two image tests fail with AssertionError: 2 != 1 (the system instruction appended as a second message); the third test pins the preserved function-messages-only behavior.
  • uv run pytest tests/test_get_llm_invocation_params.py — 179 passed, 18 subtests passed.
  • uv run pytest tests/test_run_inference.py tests/test_function_calling_adapters.py tests/test_google_thinking_defaults.py tests/test_flows_context_strategies.py tests/test_google_stream_timeout.py tests/test_realtime_service_tool_sync.py — 152 passed, 2 subtests passed.
  • uv run pytest tests/test_gemini_live_interaction_status.py tests/test_gemini_live_tracing.py tests/test_gemini_stt.py — 56 passed, 5 skipped.
  • uv run pytest (full suite) — 4756 passed, 33 skipped, 99 subtests passed. The only 3 failures are the RNNoise tests, which fail identically on main in this environment (pyrnnoise raises TypeError: Graph.__init__() got an unexpected keyword argument 'rate'), so they are pre-existing and unrelated.
  • uv run ruff check — clean. uv run ruff format --check — clean. uv run pyright src/pipecat/adapters/services/gemini_adapter.py tests/test_get_llm_invocation_params.py — 0 errors.

A user turn created by LLMContext.create_image_message() carries its text
and image in one Content (two parts), and an image-only turn carries a
single part with no text. Neither matched the single-text-part shape the
adapter's regular-message check required, so a context holding only such
turns got the system instruction appended as an extra trailing user
message, while it was also sent as the system_instruction parameter.

A message now counts as regular when any of its parts is neither a
function call nor a function response, so the injection still fires only
for contexts holding nothing but function calls and responses.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant