feat: refactor message content structure to support multi-modal content blocks #74
Conversation
|
Warning Rate limit exceeded@Pratham-Mishra04 has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 0 minutes and 1 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (12)
Summary by CodeRabbit
Summary by CodeRabbit
WalkthroughThis update introduces a new, structured approach to message content handling across the codebase and documentation. Messages now use a Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant API
participant Provider
Client->>API: Send chat request with content blocks (text/image)
API->>Provider: Prepare messages using MessageContent
Provider->>Provider: Serialize/deserialize content blocks
Provider-->>API: Return response with structured content
API-->>Client: Respond with content blocks (text/image)
Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches
🧪 Generate Unit Tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
This stack of pull requests is managed by Graphite. Learn more about stacking. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🔭 Outside diff range comments (2)
core/providers/anthropic.go (2)
329-361:⚠️ Potential issueFix potential null pointer dereference.
The function could panic if
urlTypeInfo.MediaTypeis nil.Apply this fix to handle nil MediaType:
formattedImgContent := AnthropicImageContent{ Type: urlTypeInfo.Type, - MediaType: *urlTypeInfo.MediaType, + MediaType: coalesceString(urlTypeInfo.MediaType), }
368-410:⚠️ Potential issueFix missing pointer dereference in tool message handling.
Line 407 attempts to use
block.Textdirectly but it's a pointer that needs to be dereferenced.Apply this fix:
toolCallResultContent = append(toolCallResultContent, map[string]interface{}{ "type": "text", - "text": block.Text, + "text": *block.Text, })
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (13)
README.md(1 hunks)core/providers/anthropic.go(7 hunks)core/providers/azure.go(1 hunks)core/providers/bedrock.go(9 hunks)core/providers/cohere.go(5 hunks)core/providers/openai.go(1 hunks)core/providers/utils.go(2 hunks)core/schemas/bifrost.go(3 hunks)core/tests/e2e_tool_test.go(3 hunks)core/tests/openai_test.go(1 hunks)core/tests/tests.go(10 hunks)docs/http-transport-api.md(6 hunks)docs/openapi.json(3 hunks)
🧰 Additional context used
🧠 Learnings (6)
core/tests/e2e_tool_test.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
core/tests/tests.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/cohere.go (2)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/bedrock.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/anthropic.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/schemas/bifrost.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
🧬 Code Graph Analysis (5)
core/tests/e2e_tool_test.go (2)
core/schemas/bifrost.go (3)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ModelChatMessageRoleTool(32-32)core/utils.go (1)
Ptr(3-5)
core/providers/azure.go (1)
core/schemas/bifrost.go (2)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)
core/tests/tests.go (3)
core/schemas/provider.go (1)
Provider(142-149)core/schemas/bifrost.go (7)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ContentBlock(208-212)ContentBlockTypeText(204-204)ContentBlockTypeImage(205-205)ImageURLStruct(228-231)BifrostResponse(236-246)core/utils.go (1)
Ptr(3-5)
core/providers/bedrock.go (3)
core/schemas/bifrost.go (3)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)ContentBlock(208-212)core/providers/utils.go (2)
SanitizeImageURL(339-385)ExtractURLTypeInfo(390-395)core/providers/anthropic.go (1)
AnthropicImageContent(72-76)
core/providers/anthropic.go (3)
core/providers/utils.go (4)
ImageContentType(34-34)SanitizeImageURL(339-385)ExtractURLTypeInfo(390-395)ImageContentTypeURL(38-38)core/schemas/bifrost.go (4)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)ImageURLStruct(228-231)ContentBlock(208-212)core/providers/bedrock.go (1)
BedrockAnthropicSystemMessage(69-71)
🪛 golangci-lint (1.64.8)
core/providers/cohere.go
460-460: func processImageContent is unused
(unused)
🔇 Additional comments (27)
core/providers/azure.go (1)
272-276: Good adaptation toMessageContentThe assistant reply now wraps the text in
MessageContent{ContentStr: &textCopy}– this avoids dangling pointers once the pooled Azure response is released. Looks correct and keeps allocations minimal. 👍core/providers/openai.go (1)
226-237:⚠️ Potential issueSanitised URL never written back – loop uses copy, losing the mutation
for _, block := range contentBlocks { … block.ImageURL.URL = sanitizedURL }iterates by value, so the slice elements are copied; the modifiedblockisn’t stored, leaving the original URL unsanitised.- for _, block := range contentBlocks { - if block.Type == schemas.ContentBlockTypeImage && block.ImageURL != nil { - sanitizedURL, _ := SanitizeImageURL(block.ImageURL.URL) - block.ImageURL.URL = sanitizedURL - } - } + for i := range contentBlocks { + if contentBlocks[i].Type == schemas.ContentBlockTypeImage && contentBlocks[i].ImageURL != nil { + sanitizedURL, _ := SanitizeImageURL(contentBlocks[i].ImageURL.URL) + contentBlocks[i].ImageURL.URL = sanitizedURL + } + }Without this fix, OpenAI receives unsanitised URLs, defeating the security measure.
Likely an incorrect or invalid review comment.
core/tests/e2e_tool_test.go (1)
94-98: 👍 Tool-result message created only when content presentMatches the learning from PR #55 – tool result messages are emitted only when
ContentStris non-nil. Implementation aligns with guideline.core/tests/tests.go (1)
403-415: Good addition of thegetResultContenthelper function.This function properly centralizes the logic for extracting text content from either
ContentStrorContentBlocks, improving code maintainability and consistency.docs/openapi.json (1)
112-141: LGTM! OpenAPI spec correctly reflects the new message content structure.The schema changes properly support both simple string content (backward compatibility) and structured content blocks with mixed text and images. The new example clearly demonstrates the usage pattern.
Also applies to: 447-461, 497-534
docs/http-transport-api.md (1)
81-110: Excellent documentation updates for the new message content structure.The documentation comprehensively covers the new structured content format with clear examples across all integration languages. The schema definitions are well-documented, and the backward compatibility with simple string content is preserved.
Also applies to: 245-273, 486-505, 598-609, 634-649, 671-684
core/providers/bedrock.go (2)
409-426: Good addition of flexible content parsing for tool calls.The
parseBedrockAnthropicMessageToolCallContentfunction properly handles both JSON and plain text content, providing flexibility for different tool response formats.
538-547: Verify media type format handling for Bedrock's image format field.The code removes the "image/" prefix from the media type, but this might not work correctly for all cases. For example, if
MediaTypeis "image/png", the result would be "png", but if it's already "png" (without prefix), it would remain "png".Consider using a more robust approach:
-Format: func() string { - if formattedImgContent.MediaType != "" { - mediaType := formattedImgContent.MediaType - // Remove "image/" prefix if present, since normalizeMediaType ensures full format - mediaType = strings.TrimPrefix(mediaType, "image/") - return mediaType - } - return "" -}(), +Format: func() string { + if formattedImgContent.MediaType != "" { + // Extract format from media type (e.g., "image/png" -> "png", "png" -> "png") + parts := strings.Split(formattedImgContent.MediaType, "/") + if len(parts) == 2 { + return parts[1] + } + return formattedImgContent.MediaType + } + return "" +}(),core/providers/cohere.go (1)
231-231: Message content handling correctly updated to use the new structure.The changes properly handle both
ContentStrandContentBlocksthroughout the provider, maintaining compatibility while supporting the new structured content format.Also applies to: 238-291, 427-430, 512-514
core/schemas/bifrost.go (7)
4-8: LGTM!The import additions are necessary for the custom JSON marshalling implementation.
149-151: Well-designed content structure change.The change from
*stringtoMessageContentenables support for multi-modal content while maintaining backward compatibility through custom JSON marshalling.
163-178: Robust JSON marshalling implementation.The custom marshalling correctly handles both string and content block scenarios, with proper nil checks and empty object fallback.
183-199: Flexible JSON unmarshalling with good error handling.The implementation correctly attempts to unmarshal as both direct string and array formats, providing clear error messages when neither format matches.
201-212: Well-structured content block definition.The ContentBlock design with type discriminator and optional fields for text/image content aligns well with the multi-modal content objectives.
216-217: Appropriate simplification of ToolMessage.Removing ImageContent from ToolMessage aligns with the learning that only UserMessage should contain images, not tool messages.
228-231: Clean ImageURLStruct design.The simplified structure with URL and optional detail field provides a cleaner interface for image content.
core/providers/utils.go (6)
31-31: LGTM!The regex pattern correctly matches data URI format with optional base64 encoding.
33-46: Well-defined type system for image content.The ImageContentType enum and URLTypeInfo struct provide clear type definitions for image URL handling.
339-385: Comprehensive URL sanitization with excellent error handling.The function properly handles:
- Empty URL validation
- Data URL format validation
- Raw base64 detection and conversion
- HTTP/HTTPS scheme validation
- Host validation
The automatic detection and conversion of raw base64 data to proper data URLs is particularly useful.
390-464: Well-structured URL metadata extraction.The functions properly extract media types from both data URLs and regular URLs, with good support for common image formats.
467-495: Thorough image type detection from base64 signatures.The detection logic covers all major image formats by checking their base64-encoded file signatures. The fallback to JPEG for unknown formats is reasonable.
498-505: Clean base64 validation logic.The function properly validates base64 format after removing whitespace.
core/providers/anthropic.go (5)
72-76: LGTM!The struct properly represents Anthropic's image content format.
264-268: Correct usage of MessageContent structure.The response properly constructs a MessageContent with ContentStr for text completion responses.
415-439: Excellent handling of multi-modal content blocks.The code properly processes both text and image content blocks, correctly using the sanitized image URLs through buildAnthropicImageSourceMap.
597-607: Clean content block construction in response parsing.The response parser correctly creates ContentBlock structures for text content.
648-651: Proper MessageContent structure in response.The response correctly uses ContentBlocks for the assistant message content.
Pratham-Mishra04
force-pushed
the
06-11-refactor_changed_content_schema_to_follow_openai
branch
from
5c964ce to
d28c2c1
Compare
There was a problem hiding this comment.
Actionable comments posted: 8
🔭 Outside diff range comments (2)
core/providers/azure.go (1)
267-276:⚠️ Potential issuePointer avoidance is incomplete – the string still aliases pooled bytes
textCopyis only a shallow copy of the string header; its underlying byte slice still points toresponse.Body(), which will be reused once the pooledAzureTextResponseis returned.
Dereferencing this after the function returns will access mutated data and can corrupt the caller’s view.- // Create a copy of the text to avoid dangling pointer to pooled object - textCopy := response.Choices[0].Text + // Deep-copy the bytes so the slice no longer aliases the pooled buffer + textCopy := string([]byte(response.Choices[0].Text))If you prefer Go ≥1.20,
strings.Cloneis clearer:import "strings" // … textCopy := strings.Clone(response.Choices[0].Text)Either approach guarantees the returned pointer owns its own backing data.
core/providers/anthropic.go (1)
329-361:⚠️ Potential issueFix potential nil pointer dereference.
On line 339,
*urlTypeInfo.MediaTypeis dereferenced without checking if it's nil, but later on line 355 there's a check for empty string. This could cause a panic if MediaType is nil.Apply this fix:
formattedImgContent := AnthropicImageContent{ Type: urlTypeInfo.Type, - MediaType: *urlTypeInfo.MediaType, + MediaType: "", } + if urlTypeInfo.MediaType != nil { + formattedImgContent.MediaType = *urlTypeInfo.MediaType + }
♻️ Duplicate comments (1)
core/providers/cohere.go (1)
460-480: Unused processImageContent function maintained for future use.The function is correctly updated to use the new
ImageURLStructtype and image URL utilities. As discussed in previous reviews, this is intentionally kept for future Cohere v2 support.🧰 Tools
🪛 golangci-lint (1.64.8)
460-460: func
processImageContentis unused(unused)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (12)
README.md(1 hunks)core/providers/anthropic.go(7 hunks)core/providers/azure.go(1 hunks)core/providers/bedrock.go(9 hunks)core/providers/cohere.go(5 hunks)core/providers/openai.go(1 hunks)core/providers/utils.go(2 hunks)core/schemas/bifrost.go(3 hunks)core/tests/e2e_tool_test.go(3 hunks)core/tests/tests.go(10 hunks)docs/http-transport-api.md(6 hunks)docs/openapi.json(3 hunks)
🧰 Additional context used
🧠 Learnings (6)
core/tests/e2e_tool_test.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
core/tests/tests.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/cohere.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/schemas/bifrost.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/bedrock.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/anthropic.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
🧬 Code Graph Analysis (4)
core/providers/openai.go (2)
core/schemas/bifrost.go (2)
AssistantMessage(220-225)ContentBlockTypeImage(205-205)core/providers/utils.go (1)
SanitizeImageURL(339-385)
core/providers/azure.go (1)
core/schemas/bifrost.go (2)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)
core/tests/tests.go (3)
core/schemas/provider.go (1)
Provider(142-149)core/schemas/bifrost.go (7)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ContentBlock(208-212)ContentBlockTypeText(204-204)ContentBlockTypeImage(205-205)ImageURLStruct(228-231)BifrostResponse(236-246)core/utils.go (1)
Ptr(3-5)
core/providers/bedrock.go (3)
core/schemas/bifrost.go (3)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)ContentBlock(208-212)core/providers/utils.go (2)
SanitizeImageURL(339-385)ExtractURLTypeInfo(390-395)core/providers/anthropic.go (1)
AnthropicImageContent(72-76)
🪛 golangci-lint (1.64.8)
core/providers/cohere.go
460-460: func processImageContent is unused
(unused)
🔇 Additional comments (21)
README.md (1)
152-157: Example aligns with newMessageContentAPI – looks goodThe quick-start snippet correctly uses
schemas.MessageContent{ContentStr: bifrost.Ptr(...)}and the updated role constant.core/tests/e2e_tool_test.go (1)
121-124: Helpful comment and helper usage – no issues
Switching togetResultContentremoves duplication and the accompanying comment is accurate.docs/openapi.json (3)
113-141: Well-structured example for multi-modal content.The "structured_content" example clearly demonstrates the new message format with both text and image content, including the optional "detail" parameter for image quality control.
447-461: Proper use of oneOf for flexible content types.The schema correctly uses OpenAPI's
oneOfto support both simple string content and structured content blocks, maintaining backward compatibility while enabling new functionality.
497-534: Well-designed schemas for content blocks and image URLs.The
ContentBlockandImageURLStructschemas are properly structured with:
- Clear type discrimination using the
typefield- Appropriate use of required/optional fields
- Good enum constraints for
typeanddetailfieldsdocs/http-transport-api.md (3)
81-110: Clear example of structured content request.The example effectively demonstrates how to send a message with both text and image content, matching the OpenAPI specification.
245-273: Comprehensive schema documentation for new content types.The tables clearly document the new
BifrostMessage.contentfield structure and the newContentBlockandImageURLStructtypes with appropriate field descriptions and requirements.
592-609: Well-structured client examples across multiple languages.The Python, Node.js, and Go examples are properly updated to demonstrate both simple text messages and structured content with images, providing clear guidance for API consumers.
Also applies to: 634-649, 671-684
core/providers/bedrock.go (3)
408-426: Well-implemented tool call content parser.The function properly handles both JSON and plain text content, with appropriate fallback behavior and array wrapping for JSON arrays.
454-466: Comprehensive system message content extraction.The code correctly handles both
ContentStrandContentBlocks, extracting text from all content blocks when present.
519-554: Proper integration with new image URL utilities.The image content handling correctly uses the new
SanitizeImageURLandExtractURLTypeInfoutilities, with appropriate media type extraction and formatting for the Bedrock API.core/providers/cohere.go (4)
231-231: Correct usage of ContentStr for tool results.The code properly uses
ContentStrto access the tool result content.
238-264: Proper handling of ContentBlocks in message history.The code correctly processes both
ContentStrandContentBlocks, extracting text content from each block appropriately.
281-291: Correct extraction of last message content.The code properly handles both content types, concatenating text from multiple content blocks when present.
427-430: Consistent use of MessageContent structure in responses.Both the main response and chat history conversion correctly construct
MessageContentwithContentStr.Also applies to: 512-514
core/schemas/bifrost.go (2)
201-212: LGTM!The ContentBlock structure is well-designed with proper type safety and JSON tags.
228-231: LGTM!The simplified
ImageURLStructprovides a cleaner interface for image URLs with optional detail level.core/providers/anthropic.go (3)
72-76: LGTM!The provider-specific image content structure appropriately handles Anthropic's API requirements.
264-267: LGTM!The response correctly uses the new MessageContent structure with ContentStr for text completions.
597-651: LGTM!The response parsing correctly maintains the structure of content blocks instead of losing information by concatenating text.
core/providers/utils.go (1)
31-46: LGTM!The updated regex correctly handles both base64 and non-base64 data URLs, and the new types provide clear structure for URL metadata.
Pratham-Mishra04
force-pushed
the
06-11-refactor_changed_content_schema_to_follow_openai
branch
from
d28c2c1 to
ea464dd
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (4)
core/tests/e2e_tool_test.go (2)
36-39: Pointer helper verbosity ‑ comment already made earlierThe repeated
bifrost.Ptr("…")calls add noise. Previous review suggested a local helper/alias; author preferred current style, so re-raising would be redundant.
95-97: Same observation as above – no further action.core/providers/openai.go (1)
226-238: Sanitisation errors still silently discarded (previously flagged)
SanitizeImageURLreturns an error that’s ignored:sanitizedURL, _ := SanitizeImageURL(block.ImageURL.URL)If the URL is malformed we still forward it to OpenAI, risking a 400 that’s hard to trace. Earlier review discussed bubbling or logging the error and the decision was to defer. Just noting the unchanged behaviour.
core/providers/cohere.go (1)
460-480: Function updated correctly for future use.The
processImageContentfunction has been properly updated to use the newImageURLStructtype, maintaining readiness for when Cohere adds image support.🧰 Tools
🪛 golangci-lint (1.64.8)
460-460: func
processImageContentis unused(unused)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (12)
README.md(1 hunks)core/providers/anthropic.go(7 hunks)core/providers/azure.go(1 hunks)core/providers/bedrock.go(9 hunks)core/providers/cohere.go(5 hunks)core/providers/openai.go(1 hunks)core/providers/utils.go(2 hunks)core/schemas/bifrost.go(3 hunks)core/tests/e2e_tool_test.go(3 hunks)core/tests/tests.go(10 hunks)docs/http-transport-api.md(6 hunks)docs/openapi.json(3 hunks)
🧰 Additional context used
🧠 Learnings (7)
core/tests/e2e_tool_test.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
core/providers/openai.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#63
File: transports/bifrost-http/integrations/openai/types.go:89-119
Timestamp: 2025-06-10T11:00:02.852Z
Learning: In the Bifrost OpenAI integration (transports/bifrost-http/integrations/openai/types.go), the convertOpenAIContent function currently only handles the last image URL when multiple images are present in a content array. The user Pratham-Mishra04 has acknowledged this limitation and indicated it's part of a larger architectural issue that will be addressed comprehensively later, rather than with piecemeal fixes.
core/tests/tests.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/bedrock.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/cohere.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/anthropic.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/schemas/bifrost.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
🧬 Code Graph Analysis (4)
core/providers/azure.go (1)
core/schemas/bifrost.go (2)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)
core/tests/e2e_tool_test.go (2)
core/schemas/bifrost.go (3)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ModelChatMessageRoleTool(32-32)core/utils.go (1)
Ptr(3-5)
core/providers/openai.go (2)
core/schemas/bifrost.go (2)
AssistantMessage(221-226)ContentBlockTypeImage(206-206)core/providers/utils.go (1)
SanitizeImageURL(343-389)
core/tests/tests.go (3)
core/schemas/provider.go (1)
Provider(142-149)core/schemas/bifrost.go (7)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ContentBlock(209-213)ContentBlockTypeText(205-205)ContentBlockTypeImage(206-206)ImageURLStruct(229-232)BifrostResponse(237-247)core/utils.go (1)
Ptr(3-5)
🪛 golangci-lint (1.64.8)
core/providers/cohere.go
460-460: func processImageContent is unused
(unused)
🔇 Additional comments (15)
core/providers/azure.go (1)
272-276: NewMessageContentintegration looks correctThe text completion branch now wraps the plain string inside the new
schemas.MessageContentstruct, which keeps the provider aligned with the global multimodal refactor. Copying the string intotextCopyavoids holding a pointer to the pooled response buffer – nice touch.README.md (1)
152-157: README snippet updated for multimodal refactor – LGTMThe example now uses the
schemas.MessageContentwrapper and the canonical role constant. Nothing blocking.core/tests/e2e_tool_test.go (1)
121-123: Good use ofgetResultContenthelperSwitching to the shared helper eliminates duplicated pointer-deref logic.
docs/http-transport-api.md (1)
81-684: Documentation updates are comprehensive and well-structured.The documentation clearly explains the new structured content format with:
- Clear examples showing how to combine text and image content
- Updated schema definitions for ContentBlock and ImageURLStruct
- Practical examples in multiple languages (curl, Python, Node.js, Go)
- Backward compatibility maintained with string content option
🧰 Tools
🪛 LanguageTool
[uncategorized] ~239-~239: Possible missing comma found.
Context: ... | *Eithermessagesortextis required depending on the endpoint. ### Bifrost...(AI_HYDRA_LEO_MISSING_COMMA)
docs/openapi.json (1)
112-534: OpenAPI specification correctly implements the new content structure.The changes properly define the polymorphic content field using
oneOfpattern and include:
- Well-structured ContentBlock schema with required fields
- ImageURLStruct with appropriate properties and enums for detail levels
- Comprehensive example demonstrating the new structured content format
🧰 Tools
🪛 Checkov (3.2.334)
[MEDIUM] 384-390: Ensure that arrays have a maximum number of items
(CKV_OPENAPI_21)
core/providers/bedrock.go (3)
408-427: Well-designed helper function for tool call content parsing.The
parseBedrockAnthropicMessageToolCallContentfunction correctly handles both JSON and plain text content, providing a flexible approach to parsing tool results.
454-492: Correct handling of system and tool messages with new content structure.The implementation properly extracts text content from both
ContentStrandContentBlocksfor system messages, and correctly aggregates tool call results using the parsing helper function.
513-554: Proper implementation of image content processing and multi-modal response handling.The changes correctly:
- Process image URLs with sanitization and metadata extraction
- Handle the Bedrock-specific image format requirements
- Preserve multi-modal content in responses by using ContentBlocks instead of concatenating text
Also applies to: 909-961
core/providers/cohere.go (1)
231-291: Consistent implementation of the new MessageContent structure.The Cohere provider correctly handles the new content structure:
- Tool results properly access
Content.ContentStr- Message content handling supports both
ContentStrandContentBlocks- Chat history conversion wraps strings in
MessageContentNote: The image content code is intentionally commented out as Cohere v1 doesn't support images.
Also applies to: 427-431, 511-515
core/schemas/bifrost.go (2)
158-179: Well-implemented polymorphic content structure with proper validation.The
MessageContentimplementation correctly addresses the previous marshaling issues and includes validation to ensure only one field is set at a time. The direct marshaling without wrapper objects is the correct approach.
202-213: Clean discriminated union pattern for multi-modal content blocks.The
ContentBlockstructure with its type discriminator is well-designed for handling different content types. The optional fields align properly with the content type.core/providers/anthropic.go (2)
328-361: Proper integration with new image URL utilities.The refactored
buildAnthropicImageSourceMapcorrectly utilizes the new sanitization and type extraction utilities, providing consistent image URL handling across the codebase.
597-651: Correct adaptation to structured content blocks.The response parsing now properly creates an array of
ContentBlockelements instead of concatenating text, preserving the structured nature of the response content.core/providers/utils.go (2)
340-389: Comprehensive URL sanitization with proper validation.The
SanitizeImageURLfunction provides thorough validation and sanitization for all supported URL types - data URLs, raw base64, and regular HTTP(S) URLs. The automatic detection and conversion of raw base64 to proper data URLs is a helpful feature.
470-508: Robust image type detection using header signatures and extensions.The image type detection logic efficiently identifies common image formats using base64 header signatures and file extensions. The fallback to JPEG for unknown formats is a reasonable default. The pre-compiled regex for base64 validation addresses the previous performance concern.
Pratham-Mishra04
force-pushed
the
06-11-refactor_changed_content_schema_to_follow_openai
branch
from
ea464dd to
ead45eb
Compare
There was a problem hiding this comment.
Actionable comments posted: 4
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (12)
README.md(1 hunks)core/providers/anthropic.go(7 hunks)core/providers/azure.go(1 hunks)core/providers/bedrock.go(9 hunks)core/providers/cohere.go(5 hunks)core/providers/openai.go(1 hunks)core/providers/utils.go(2 hunks)core/schemas/bifrost.go(3 hunks)core/tests/e2e_tool_test.go(3 hunks)core/tests/tests.go(10 hunks)docs/http-transport-api.md(6 hunks)docs/openapi.json(3 hunks)
🧰 Additional context used
🧠 Learnings (7)
core/tests/e2e_tool_test.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
core/tests/tests.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/bedrock.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/openai.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#63
File: transports/bifrost-http/integrations/openai/types.go:89-119
Timestamp: 2025-06-10T11:00:02.852Z
Learning: In the Bifrost OpenAI integration (transports/bifrost-http/integrations/openai/types.go), the convertOpenAIContent function currently only handles the last image URL when multiple images are present in a content array. The user Pratham-Mishra04 has acknowledged this limitation and indicated it's part of a larger architectural issue that will be addressed comprehensively later, rather than with piecemeal fixes.
core/providers/cohere.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/anthropic.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/schemas/bifrost.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
🧬 Code Graph Analysis (5)
core/providers/azure.go (1)
core/schemas/bifrost.go (2)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)
core/tests/e2e_tool_test.go (2)
core/schemas/bifrost.go (3)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ModelChatMessageRoleTool(32-32)core/utils.go (1)
Ptr(3-5)
core/tests/tests.go (3)
core/schemas/provider.go (1)
Provider(142-149)core/schemas/bifrost.go (7)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ContentBlock(209-213)ContentBlockTypeText(205-205)ContentBlockTypeImage(206-206)ImageURLStruct(229-232)BifrostResponse(237-247)core/utils.go (1)
Ptr(3-5)
core/providers/cohere.go (3)
core/schemas/bifrost.go (2)
MessageContent(158-161)ImageURLStruct(229-232)core/providers/utils.go (2)
SanitizeImageURL(343-389)ExtractURLTypeInfo(394-399)core/providers/anthropic.go (1)
AnthropicImageContent(72-76)
core/providers/anthropic.go (3)
core/providers/utils.go (4)
ImageContentType(38-38)SanitizeImageURL(343-389)ExtractURLTypeInfo(394-399)ImageContentTypeURL(42-42)core/schemas/bifrost.go (4)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)ImageURLStruct(229-232)ContentBlock(209-213)core/providers/bedrock.go (1)
BedrockAnthropicSystemMessage(69-71)
🪛 golangci-lint (1.64.8)
core/providers/cohere.go
460-460: func processImageContent is unused
(unused)
🔇 Additional comments (23)
README.md (1)
152-157: Example compiles & matches new APIThe quick-start snippet correctly demonstrates the
MessageContentwrapper and the helper pointer function. ✅core/tests/e2e_tool_test.go (3)
35-40: Message construction aligns with new schemaThe test now wraps the prompt in
MessageContent, preventing compilation mismatches with the refactor.
Looks good.
94-99: Tool result message correctly populated
Role: toolmessages now carry the JSON result insideContentStr; this matches the contract added in #55.
No issues spotted.
121-124: Helper usage improves readabilityUsing
getResultContentremoves duplication and centralises pointer-handling. Nice clean-up.core/tests/tests.go (4)
112-117: Log output streamlinedSingle call to
getResultContentkeeps the logging path consistent across tests.
147-152: Pointer helper keeps code terseGood use of
bifrost.Ptrfor test messages – reduces boilerplate.
190-205: Image-input test matches multimodal contractThe URL & text blocks are constructed per spec; test should exercise provider handling correctly.
393-409: Guard-clause makes helper safeThe early-return prevents panics on malformed responses – thanks for adding it.
docs/http-transport-api.md (1)
81-109: Documentation updates are comprehensive and well-structured.The additions clearly explain the new structured content format, provide helpful examples for text+image messages, and update all integration code samples consistently. The schema definitions for
ContentBlockandImageURLStructare properly documented with all required fields marked.Also applies to: 245-273, 486-505, 598-608, 634-649, 672-684
docs/openapi.json (1)
113-141: OpenAPI specification correctly reflects the new content structure.The
oneOfschema for message content properly supports both simple strings and structured content blocks. The example demonstrates the new format clearly, and the schema definitions are consistent with the API documentation.Also applies to: 447-461, 497-534
core/providers/bedrock.go (4)
408-426: Well-designed helper function for flexible content parsing.The function properly handles both JSON and plain text content, with appropriate fallback behavior. The special handling for arrays (wrapping in a content object) ensures consistent structure.
454-466: Content extraction properly handles all message formats.The implementation correctly:
- Extracts system message text from either single strings or multiple content blocks
- Aggregates tool call results using the new parsing helper
- Processes image content with proper URL sanitization and media type extraction
Also applies to: 480-490, 513-554
342-346: Text completion results correctly use the new MessageContent structure.The response content is properly wrapped in
MessageContentwithContentStr, maintaining consistency with the new content format.Also applies to: 382-386
909-917: Chat completion response correctly uses content blocks.The implementation properly collects text content as individual
ContentBlockentries and sets the final message content usingContentBlocksinstead of a single concatenated string.Also applies to: 957-960
core/providers/cohere.go (2)
231-232: Other content handling changes are implemented correctly.The code properly:
- Uses
ContentStrfor tool message outputs- Handles last message content extraction from both formats
- Wraps response messages in the new
MessageContentstructureAlso applies to: 281-291, 427-430, 511-514
245-257:⚠️ Potential issueFix logic error in content block processing.
Inside the
ContentBlocksiteration, the code incorrectly checksmsg.Content.ContentStrinstead of using the block's text content. This would add the message-level content string multiple times.Apply this fix to correctly process content blocks:
} else if msg.Content.ContentBlocks != nil { for _, block := range *msg.Content.ContentBlocks { if block.Text != nil { contentArray = append(contentArray, map[string]interface{}{ "type": "text", "text": *block.Text, }) } - // Add image content using our helper function - // NOTE: Cohere v1 does not support image content - // if processedImageContent := processImageContent(block.ImageContent); processedImageContent != nil { - // contentArray = append(contentArray, processedImageContent) - // } } }Also remove the redundant outer condition:
-if msg.Content.ContentStr != nil { - contentArray = append(contentArray, map[string]interface{}{ - "type": "text", - "text": *msg.Content.ContentStr, - }) -} else if msg.Content.ContentBlocks != nil { +if msg.Content.ContentBlocks != nil {Likely an incorrect or invalid review comment.
core/schemas/bifrost.go (4)
158-161: LGTM! Well-designed polymorphic content structure.The
MessageContentstruct provides a clean way to represent either simple text or structured content blocks, aligning well with the multi-modal content requirements.
165-179: Excellent implementation addressing previous concerns.The
MarshalJSONmethod correctly:
- Validates that only one field is set at a time
- Marshals content directly without wrapping
- Returns null for empty content
This addresses the previous review feedback perfectly.
183-200: Clean unmarshalling logic with proper type detection.The
UnmarshalJSONmethod correctly handles both string and array formats, providing clear error messages when the content doesn't match either expected type.
202-213: Well-structured content block definitions.The
ContentBlocktype and structure provide a flexible foundation for multi-modal content, with clear type constants and proper JSON serialization tags.core/providers/anthropic.go (2)
329-361: Excellent refactoring using centralized URL utilities.The updated
buildAnthropicImageSourceMapfunction now leverages the newSanitizeImageURLandExtractURLTypeInfoutilities, improving consistency across providers and centralizing URL validation logic.
597-651: Good adaptation to structured content blocks.The response parsing now properly creates an array of
ContentBlockitems instead of concatenating text, preserving the response structure and aligning with the new content model.core/providers/utils.go (1)
343-389: Comprehensive URL sanitization with good error handling.The
SanitizeImageURLfunction provides thorough validation for all URL types:
- Proper data URL format validation
- Raw base64 detection and conversion
- HTTP/HTTPS URL validation with scheme and host checks
The implementation handles edge cases well with clear error messages.
Pratham-Mishra04
force-pushed
the
06-11-refactor_changed_content_schema_to_follow_openai
branch
from
ead45eb to
fc217ab
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (2)
core/providers/azure.go (1)
272-275: Prior suggestion already discussed – no further actionThe manual
&textCopyaddress-taking is still present. This mirrors the earlier discussion on usingbifrost.Ptr(...); since you explicitly chose to keep the manual pointer last round, I’m not re-opening the topic.core/providers/openai.go (1)
226-234: Sanitisation-error handling unchanged (see prior review)The
SanitizeImageURLerror is still discarded and URL updates occur on a copy of the slice header—both points were covered in earlier feedback. No extra comments here.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (12)
README.md(1 hunks)core/providers/anthropic.go(7 hunks)core/providers/azure.go(1 hunks)core/providers/bedrock.go(9 hunks)core/providers/cohere.go(5 hunks)core/providers/openai.go(1 hunks)core/providers/utils.go(2 hunks)core/schemas/bifrost.go(3 hunks)core/tests/e2e_tool_test.go(3 hunks)core/tests/tests.go(10 hunks)docs/http-transport-api.md(6 hunks)docs/openapi.json(3 hunks)
🧰 Additional context used
🧠 Learnings (7)
core/tests/e2e_tool_test.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
core/providers/openai.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#63
File: transports/bifrost-http/integrations/openai/types.go:89-119
Timestamp: 2025-06-10T11:00:02.852Z
Learning: In the Bifrost OpenAI integration (transports/bifrost-http/integrations/openai/types.go), the convertOpenAIContent function currently only handles the last image URL when multiple images are present in a content array. The user Pratham-Mishra04 has acknowledged this limitation and indicated it's part of a larger architectural issue that will be addressed comprehensively later, rather than with piecemeal fixes.
core/tests/tests.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/bedrock.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/anthropic.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/schemas/bifrost.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/cohere.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
🧬 Code Graph Analysis (5)
core/tests/e2e_tool_test.go (2)
core/schemas/bifrost.go (3)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ModelChatMessageRoleTool(32-32)core/utils.go (1)
Ptr(3-5)
core/providers/azure.go (1)
core/schemas/bifrost.go (2)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)
core/providers/openai.go (2)
core/schemas/bifrost.go (2)
AssistantMessage(221-226)ContentBlockTypeImage(206-206)core/providers/utils.go (1)
SanitizeImageURL(355-401)
core/tests/tests.go (3)
core/schemas/provider.go (1)
Provider(142-149)core/schemas/bifrost.go (7)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ContentBlock(209-213)ContentBlockTypeText(205-205)ContentBlockTypeImage(206-206)ImageURLStruct(229-232)BifrostResponse(237-247)core/utils.go (1)
Ptr(3-5)
core/providers/anthropic.go (3)
core/providers/utils.go (4)
ImageContentType(50-50)SanitizeImageURL(355-401)ExtractURLTypeInfo(406-411)ImageContentTypeURL(54-54)core/schemas/bifrost.go (4)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)ImageURLStruct(229-232)ContentBlock(209-213)core/providers/bedrock.go (1)
BedrockAnthropicSystemMessage(69-71)
🪛 golangci-lint (1.64.8)
core/providers/cohere.go
460-460: func processImageContent is unused
(unused)
🔇 Additional comments (12)
README.md (1)
152-157: Example aligns with newMessageContentAPINice, the snippet now mirrors the struct-based content used in code. No issues spotted.
core/tests/e2e_tool_test.go (1)
121-126:getResultContentkeeps the assertion conciseSwitching to the helper removes pointer boiler-plate and reads well. Looks good.
docs/http-transport-api.md (1)
81-110: Documentation updates are comprehensive and well-structured.The documentation clearly explains the new structured content format with appropriate examples demonstrating how to send messages with mixed text and image content. The schema definitions for
ContentBlockandImageURLStructare properly documented with all required fields and descriptions.Also applies to: 245-273, 487-505
docs/openapi.json (1)
113-141: OpenAPI specification correctly reflects the new content structure.The schema changes properly define the polymorphic
contentfield usingoneOf, allowing both simple strings and structured content blocks. The newContentBlockandImageURLStructschemas are well-defined with appropriate required fields and enums.Also applies to: 447-462, 497-534
core/providers/bedrock.go (3)
409-426: Well-designed helper function for parsing tool call content.The
parseBedrockAnthropicMessageToolCallContentfunction elegantly handles both JSON and plain text content, providing a consistent interface for tool results.
454-466: Proper implementation of the new MessageContent structure.The code correctly handles both
ContentStrandContentBlocksthroughout the message preparation logic. The image URL sanitization and metadata extraction are properly integrated using the new utility functions.Also applies to: 479-490, 513-554
909-917: Response content aggregation correctly uses ContentBlocks.The change from string concatenation to collecting ContentBlock entries aligns with the new structured content format, providing better flexibility for multi-modal responses.
Also applies to: 957-960
core/schemas/bifrost.go (1)
166-179: Well-implemented validation in MarshalJSON!The validation logic correctly ensures that only one of
ContentStrorContentBlocksis set at a time, preventing undefined behavior. The direct marshaling without wrapper objects is also correct.core/providers/anthropic.go (2)
329-361: Excellent integration with the new image URL utilities!The updated
buildAnthropicImageSourceMapfunction correctly:
- Sanitizes the image URL
- Extracts URL type information
- Handles both regular URLs and data URLs without prefixes
- Properly maps the content based on the detected type
The implementation aligns perfectly with the new structured content model.
403-410:⚠️ Potential issuePotential nil pointer dereference in tool message content.
At line 407,
block.Textis used directly without checking if it's nil. While text blocks should have text content, consider adding a nil check for defensive programming:} else if msg.Content.ContentBlocks != nil { for _, block := range *msg.Content.ContentBlocks { - toolCallResultContent = append(toolCallResultContent, map[string]interface{}{ - "type": "text", - "text": block.Text, - }) + if block.Text != nil { + toolCallResultContent = append(toolCallResultContent, map[string]interface{}{ + "type": "text", + "text": *block.Text, + }) + } } }⛔ Skipped due to learnings
Learnt from: Pratham-Mishra04 PR: maximhq/bifrost#55 File: core/providers/bedrock.go:443-468 Timestamp: 2025-06-04T09:32:15.826Z Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.core/providers/utils.go (2)
31-47: Great improvements addressing previous feedback!The implementation successfully addresses all previous review comments:
- ✓ Pre-compiled regex patterns for efficiency
- ✓ File extension mapping extracted into a maintainable map
- ✓ More precise WebP detection
The code is well-structured and efficient.
489-491: WebP detection properly checks exact header positions.The WebP detection now correctly validates both the RIFF header at the start and the WEBP signature at the precise offset, ensuring accurate format detection.
Pratham-Mishra04
force-pushed
the
06-11-refactor_changed_content_schema_to_follow_openai
branch
from
fc217ab to
05c93c4
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🔭 Outside diff range comments (1)
core/providers/cohere.go (1)
453-473: 💡 Verification agent🧩 Analysis chain
Fix provider-specific type usage.
The
processImageContentfunction creates anAnthropicImageContentstruct within the Cohere provider. This cross-provider type usage is inconsistent and could cause confusion. Consider either:
- Creating a Cohere-specific image content type
- Using a generic/shared image content structure
- Removing the function entirely since it's unused (as indicated by static analysis)
🏁 Script executed:
#!/bin/bash # Check if AnthropicImageContent is used elsewhere in cohere.go or if there's a Cohere-specific type rg "AnthropicImageContent" core/providers/cohere.go rg "CohereImageContent" core/providers/Length of output: 138
Align image content type in the Cohere provider
- In core/providers/cohere.go (lines 453–473),
processImageContentis instantiatingAnthropicImageContentinside the Cohere provider—even though noCohereImageContentor sharedImageContenttype exists.- To resolve this inconsistency, choose one of the following:
- Define a
CohereImageContentstruct and use it here.- Extract a generic/shared
ImageContenttype for all providers.- Remove the unused
processImageContentfunction (static analysis shows it’s not referenced elsewhere).🧰 Tools
🪛 golangci-lint (1.64.8)
453-453: func
processImageContentis unused(unused)
♻️ Duplicate comments (1)
core/providers/cohere.go (1)
238-257: Logic correctly handles both content types.The implementation properly handles both
ContentStrandContentBlockscases without redundant checks. The commented image content code is appropriately preserved for future Cohere versions.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (12)
README.md(1 hunks)core/providers/anthropic.go(7 hunks)core/providers/azure.go(1 hunks)core/providers/bedrock.go(9 hunks)core/providers/cohere.go(5 hunks)core/providers/openai.go(1 hunks)core/providers/utils.go(2 hunks)core/schemas/bifrost.go(3 hunks)core/tests/e2e_tool_test.go(3 hunks)core/tests/tests.go(10 hunks)docs/http-transport-api.md(6 hunks)docs/openapi.json(3 hunks)
🧰 Additional context used
🧠 Learnings (7)
core/tests/e2e_tool_test.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
core/tests/tests.go (2)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#74
File: core/tests/tests.go:393-409
Timestamp: 2025-06-11T14:35:19.121Z
Learning: In core/tests/tests.go, the helper function getResultContent is intentionally designed to extract content only from the first element of result.Choices; this is expected behavior for the current test strategy.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/openai.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#63
File: transports/bifrost-http/integrations/openai/types.go:89-119
Timestamp: 2025-06-10T11:00:02.852Z
Learning: In the Bifrost OpenAI integration (transports/bifrost-http/integrations/openai/types.go), the convertOpenAIContent function currently only handles the last image URL when multiple images are present in a content array. The user Pratham-Mishra04 has acknowledged this limitation and indicated it's part of a larger architectural issue that will be addressed comprehensively later, rather than with piecemeal fixes.
core/providers/bedrock.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/cohere.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/schemas/bifrost.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/anthropic.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
🧬 Code Graph Analysis (6)
core/providers/azure.go (1)
core/schemas/bifrost.go (2)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)
core/tests/e2e_tool_test.go (2)
core/schemas/bifrost.go (3)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ModelChatMessageRoleTool(32-32)core/utils.go (1)
Ptr(3-5)
core/tests/tests.go (3)
core/schemas/provider.go (1)
Provider(142-149)core/schemas/bifrost.go (7)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ContentBlock(209-213)ContentBlockTypeText(205-205)ContentBlockTypeImage(206-206)ImageURLStruct(229-232)BifrostResponse(237-247)core/utils.go (1)
Ptr(3-5)
core/providers/openai.go (2)
core/schemas/bifrost.go (2)
AssistantMessage(221-226)ContentBlockTypeImage(206-206)core/providers/utils.go (1)
SanitizeImageURL(355-401)
core/providers/cohere.go (3)
core/schemas/bifrost.go (2)
MessageContent(158-161)ImageURLStruct(229-232)core/providers/utils.go (2)
SanitizeImageURL(355-401)ExtractURLTypeInfo(406-411)core/providers/anthropic.go (1)
AnthropicImageContent(72-76)
core/providers/anthropic.go (3)
core/providers/utils.go (4)
ImageContentType(50-50)SanitizeImageURL(355-401)ExtractURLTypeInfo(406-411)ImageContentTypeURL(54-54)core/schemas/bifrost.go (4)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)ImageURLStruct(229-232)ContentBlock(209-213)core/providers/bedrock.go (1)
BedrockAnthropicSystemMessage(69-71)
🪛 golangci-lint (1.64.8)
core/providers/cohere.go
453-453: func processImageContent is unused
(unused)
🔇 Additional comments (23)
core/providers/utils.go (3)
29-47: Well-structured constants and pre-compiled patterns!The pre-compiled regex patterns and the file extension map are well-implemented, addressing previous performance and maintainability concerns effectively.
352-401: Comprehensive URL sanitization implementation!The function provides thorough validation for all URL types (data URLs, base64, HTTP/HTTPS) with clear error messages for each validation failure case.
470-499: Accurate image type detection with improved WebP handling!The function correctly identifies image formats from base64 signatures, with precise WebP detection using exact header position and signature location as suggested in previous reviews.
core/providers/azure.go (1)
272-276: Correct implementation of the new MessageContent structure!The change properly wraps the text content in the MessageContent struct, aligning with the refactored message content structure across the codebase.
README.md (1)
152-157: Example accurately reflects the new message content structure!The updated example correctly demonstrates the new
BifrostMessagetype withMessageContentwrapper, providing users with an accurate reference for the current API.core/tests/e2e_tool_test.go (1)
36-40: Test correctly updated to use the new MessageContent structure!The test properly constructs messages with the
MessageContentwrapper and uses thegetResultContenthelper for safe content extraction, ensuring compatibility with the refactored message structure.Also applies to: 94-97, 121-123
core/tests/tests.go (2)
147-152: Consistent and correct message construction throughout tests!All test messages properly use the new
MessageContentstructure:
- Text-only messages correctly use
ContentStr- Multi-modal messages properly use
ContentBlockswith appropriate types- Image URLs (both HTTP and base64) are correctly wrapped in
ImageURLStructAlso applies to: 191-204, 233-246, 302-306
393-409: Well-implemented content extraction helper!The
getResultContentfunction properly handles both content formats (string and blocks) with appropriate nil checks, providing a clean abstraction for test result verification.core/providers/openai.go (2)
215-215: LGTM! Correct content access for assistant messages.The code properly uses
ContentStrfor assistant messages, which aligns with the schema design where assistant messages cannot contain images.
226-238: Well-implemented polymorphic content handling.The code correctly handles both simple string content and structured content blocks. The index-based iteration properly modifies the underlying slice elements, and the image URL sanitization is applied as expected.
Note: Sanitization errors are intentionally ignored to let OpenAI handle invalid URLs, as per the design decision.
docs/openapi.json (1)
113-141: Excellent OpenAPI documentation updates!The schema changes accurately reflect the new polymorphic content structure:
- The
oneOfschema properly documents that content can be either a string or an array of ContentBlock objects- The new
structured_contentexample clearly demonstrates the usage- ContentBlock and ImageURLStruct schemas are well-defined with appropriate descriptions
Also applies to: 447-462, 497-534
docs/http-transport-api.md (1)
81-109: Outstanding documentation coverage for structured content!The documentation comprehensively covers the new feature with:
- Clear request/response examples showing structured content usage
- Well-organized schema documentation for BifrostMessage, ContentBlock, and ImageURLStruct
- Practical integration examples in multiple programming languages
- Helpful curl examples demonstrating the API usage
Also applies to: 245-273, 486-505, 592-609, 634-649, 672-684
core/providers/bedrock.go (5)
408-426: Well-designed tool call content parser!The function robustly handles both JSON and text content with:
- Proper error handling and fallback to text
- Special handling for JSON arrays (wrapped in a content object)
- Clear return structure with either "json" or "text" key
454-466: Flexible system message content extraction.The code properly handles both content formats:
- Direct access via ContentStr for simple text
- Iteration through ContentBlocks to extract text content
- Graceful handling of nil values
480-492: Consistent tool call content handling.Good use of the
parseBedrockAnthropicMessageToolCallContentfunction to handle both content formats uniformly. The aggregation of multiple content blocks is implemented correctly.
519-554: Comprehensive image content processing!Excellent implementation with:
- Proper URL sanitization using the shared utility
- Metadata extraction for format detection
- Correct adaptation to Bedrock's expected image structure
- Media type prefix removal for Bedrock compatibility
909-917: Correct transition to structured content blocks.The response handling properly:
- Builds an array of ContentBlock objects instead of concatenating strings
- Wraps each text content in the appropriate ContentBlock structure
- Sets the ContentBlocks field in the response message
Also applies to: 957-960
core/providers/cohere.go (3)
231-231: LGTM!The change correctly accesses the
ContentStrfield of the newMessageContentstructure for tool result outputs.
420-424: LGTM!Correctly constructs
BifrostMessagewith the newMessageContentstructure.
504-507: LGTM!Correctly converts chat history messages to use the new
MessageContentstructure.core/schemas/bifrost.go (3)
158-200: Well-implemented polymorphic content structure.The
MessageContentimplementation correctly:
- Validates that only one field is set during marshaling
- Handles both string and array formats during unmarshaling
- Marshals content directly without unnecessary wrapper objects
202-213: Clean content block design.The
ContentBlockstructure provides a well-designed, extensible format for supporting different content types (text and images).
217-217: Good simplification of message structures.The removal of
ImageContentfromToolMessageand the introduction ofImageURLStructsimplifies the schema while maintaining functionality through the newContentBlocksystem.Also applies to: 229-232
Pratham-Mishra04
force-pushed
the
06-11-refactor_changed_content_schema_to_follow_openai
branch
from
05c93c4 to
72a4006
Compare
There was a problem hiding this comment.
Actionable comments posted: 10
🔭 Outside diff range comments (2)
docs/openapi.json (1)
440-462:⚠️ Potential issue
contentshould be declared as a required field inBifrostMessageThe markdown docs (table at lines 245-254 of
http-transport-api.md) state thatcontentis required, and the server code relies on its presence.
However, the OpenAPI schema still lists only"role"in therequiredarray, leavingcontentoptional. This divergence will break client-side validation / code-gen.- "required": ["role"], + "required": ["role", "content"],Update the
requiredlist (and regenerate client stubs) so that SDKs do not allow messages without content.core/providers/cohere.go (1)
453-472: 🧹 Nitpick (assertive)Consider documenting the unused function's purpose.
The
processImageContentfunction is correctly updated but currently unused. Consider adding a comment explaining it's reserved for future Cohere image support.Add documentation above the function:
+// processImageContent is reserved for future use when Cohere adds image support. +// Currently unused as Cohere v1 does not support image content. // processImageContent processes image content for Cohere API format.🧰 Tools
🪛 golangci-lint (1.64.8)
453-453: func
processImageContentis unused(unused)
♻️ Duplicate comments (7)
core/providers/azure.go (1)
272-275: Preferbifrost.Ptrover taking the address manuallyThe code base conventionally uses the helper
bifrost.Ptr(...)to create pointers to stack variables. Re-using the helper keeps style consistent and avoids accidental mistakes when the helper’s semantics evolve.core/tests/tests.go (6)
148-152: Pointer helper usage repetition noted earlier – leaving as-is per author preference.
165-168: Same logging consideration as above; applies throughout similarlog.Printlnsites.
222-225: Logging nit identical to the first comment – same refactor advice applies.
264-267: Logging nit identical to the first comment – same refactor advice applies.
302-305: Pointer helper repetition – acknowledged and intentionally kept as per earlier discussion.
318-318: Logging nit identical to the first comment – same refactor advice applies.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (12)
README.md(1 hunks)core/providers/anthropic.go(7 hunks)core/providers/azure.go(1 hunks)core/providers/bedrock.go(9 hunks)core/providers/cohere.go(5 hunks)core/providers/openai.go(1 hunks)core/providers/utils.go(2 hunks)core/schemas/bifrost.go(3 hunks)core/tests/e2e_tool_test.go(3 hunks)core/tests/tests.go(10 hunks)docs/http-transport-api.md(6 hunks)docs/openapi.json(3 hunks)
🧰 Additional context used
🧠 Learnings (7)
core/tests/e2e_tool_test.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
core/providers/openai.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#63
File: transports/bifrost-http/integrations/openai/types.go:89-119
Timestamp: 2025-06-10T11:00:02.852Z
Learning: In the Bifrost OpenAI integration (transports/bifrost-http/integrations/openai/types.go), the convertOpenAIContent function currently only handles the last image URL when multiple images are present in a content array. The user Pratham-Mishra04 has acknowledged this limitation and indicated it's part of a larger architectural issue that will be addressed comprehensively later, rather than with piecemeal fixes.
core/tests/tests.go (2)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#74
File: core/tests/tests.go:393-409
Timestamp: 2025-06-11T14:35:19.121Z
Learning: In core/tests/tests.go, the helper function getResultContent is intentionally designed to extract content only from the first element of result.Choices; this is expected behavior for the current test strategy.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/bedrock.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/cohere.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/providers/anthropic.go (3)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/schemas/bifrost.go (1)
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
🧬 Code Graph Analysis (5)
core/providers/azure.go (1)
core/schemas/bifrost.go (2)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)
core/tests/e2e_tool_test.go (2)
core/schemas/bifrost.go (3)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ModelChatMessageRoleTool(32-32)core/utils.go (1)
Ptr(3-5)
core/tests/tests.go (3)
core/schemas/provider.go (1)
Provider(142-149)core/schemas/bifrost.go (7)
ModelChatMessageRoleUser(29-29)MessageContent(158-161)ContentBlock(209-213)ContentBlockTypeText(205-205)ContentBlockTypeImage(206-206)ImageURLStruct(229-232)BifrostResponse(237-247)core/utils.go (1)
Ptr(3-5)
core/providers/cohere.go (3)
core/schemas/bifrost.go (2)
MessageContent(158-161)ImageURLStruct(229-232)core/providers/utils.go (2)
SanitizeImageURL(355-401)ExtractURLTypeInfo(406-411)core/providers/anthropic.go (1)
AnthropicImageContent(72-76)
core/providers/anthropic.go (3)
core/providers/utils.go (4)
ImageContentType(50-50)SanitizeImageURL(355-401)ExtractURLTypeInfo(406-411)ImageContentTypeURL(54-54)core/schemas/bifrost.go (4)
ModelChatMessageRoleAssistant(28-28)MessageContent(158-161)ImageURLStruct(229-232)ContentBlock(209-213)core/providers/bedrock.go (1)
BedrockAnthropicSystemMessage(69-71)
🪛 golangci-lint (1.64.8)
core/providers/cohere.go
453-453: func processImageContent is unused
(unused)
🔇 Additional comments (30)
core/providers/openai.go (1)
231-234: 👍 Slice-index assignment fixes the earlier copy-mutation bugUsing
contentBlocks[i]instead of the loop variable ensures the sanitised URL is written back to the underlying slice. Nice catch.docs/http-transport-api.md (1)
81-109: Great addition of a concrete structured-content exampleThe JSON sample clearly shows mixed
textandimage_urlblocks and will be invaluable for integrators.core/providers/cohere.go (5)
231-231: LGTM!Correctly updated to access the string content from the new
MessageContentstructure.
238-261: Content handling correctly updated for the new structure.The code properly handles both
ContentStrandContentBlocks, extracting text content appropriately for Cohere's API format.
273-284: Last message handling is correct.Appropriately processes both content types, concatenating text blocks when needed for Cohere's single message string requirement.
420-423: Response construction updated correctly.Properly wraps the response content in the new
MessageContentstructure.
504-507: Chat history conversion updated correctly.Properly wraps message content in the new structure.
core/schemas/bifrost.go (7)
4-7: Necessary imports added.The
encoding/jsonandfmtpackages are required for the custom JSON marshaling implementation.
150-150: Breaking change correctly implemented.The
Contentfield now supports the polymorphicMessageContenttype, enabling multi-modal content as intended.
158-161: Well-designed polymorphic content structure.The
MessageContentstruct appropriately supports both simple strings and structured content blocks.
163-179: Excellent JSON marshaling implementation.The method correctly validates mutual exclusivity, marshals content directly without wrapping, and handles the nil case appropriately.
181-200: Robust JSON unmarshaling logic.The method correctly attempts both content types in order and provides clear error messaging.
202-213: Content block types properly defined.The enum and struct correctly support text and image URL content blocks for multi-modal messages.
217-217: Schema simplifications are appropriate.The
ToolMessagesimplification andImageURLStructrename align well with the new content model.Also applies to: 229-232
core/providers/anthropic.go (5)
72-76: Provider-specific image content struct added.The
AnthropicImageContentstruct correctly models Anthropic's image content format requirements.
264-267: Text completion response correctly updated.Properly wraps the completion text in the new
MessageContentstructure.
368-441: Comprehensive content handling implementation.The code correctly handles all message types and content formats:
- System messages extract text from both ContentStr and ContentBlocks
- Tool messages properly handle content with nil checks
- User/Assistant messages support both text and image content blocks
The pointer handling is correct throughout.
329-361: Image source mapping correctly updated.The function properly utilizes the new image URL sanitization utilities and correctly builds source maps for both base64 and URL image types.
599-653: Response parsing preserves content structure.The implementation correctly builds an array of
ContentBlockitems, preserving the structured nature of the response content.core/providers/utils.go (7)
31-62: Well-structured utility foundations.The pre-compiled regex patterns, file extension mappings, and type definitions provide a solid foundation for the image URL processing utilities.
355-401: Comprehensive URL sanitization implementation.The function correctly handles all URL types: data URLs, raw base64 data, and HTTP/HTTPS URLs. The validation logic is thorough and the base64 detection/conversion is well-implemented.
406-411: Clean extraction logic.Appropriately delegates to specialized functions based on URL type.
414-442: Data URL parsing is correct.The function properly extracts media type, encoding, and data components from data URLs.
445-468: File extension inference implemented well.The function correctly uses the map-based approach for media type inference from file extensions.
471-499: Image type detection is comprehensive.The function correctly identifies image types from base64 prefixes, including the improved WebP detection that checks specific byte positions.
502-508: Efficient base64 validation.The function correctly uses the pre-compiled regex for performance as suggested.
README.md (1)
152-157: Example snippet correctly reflects the newMessageContentAPI – no issues spotted.
The updated example compiles fine against the refactored schemas and clearly demonstrates the newContentStrusage.core/tests/e2e_tool_test.go (3)
36-40: User-message construction aligns with the new schema.
Switching toMessageContent{ContentStr: bifrost.Ptr(...)}is exactly what we need after the refactor.
94-98: Tool-result message adapted correctly.
The test now wraps the JSON string insideContentStr; this will exercise the provider’s tool-result path as intended.
121-124: Assertion helper use looks solid.
getResultContentkeeps the test concise and prevents pointer-boilerplate in the assertion block.
Pratham-Mishra04
force-pushed
the
06-11-refactor_changed_content_schema_to_follow_openai
branch
from
72a4006 to
e437920
Compare
Merge activity
|
akshaydeo
deleted the
06-11-refactor_changed_content_schema_to_follow_openai
branch
Update Message Content Structure for Multi-Modal Support
This PR refactors the message content structure to better support multi-modal interactions (text and images) across all providers. The key changes include:
Content *stringfield with a structuredMessageContenttype that can contain either a string or an array of content blocksContentBlocktype to represent different content types (text or image)ImageContenttype in favor of the more flexibleImageURLStructThis change provides a more consistent way to handle multi-modal content across different providers and aligns better with OpenAI's content structure, making it easier to work with both text and image inputs in the same message.