Skip to content

refactor: standardize image block data structure #1082

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zerob13 merged 3 commits into from
Nov 6, 2025
Merged

refactor: standardize image block data structure #1082

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ca322c9
refactor: standardize image block data structure with backward compat…
Dw9 Nov 5, 2025
c9d8f46
fix: properly normalize image data URIs before persistence
Dw9 Nov 5, 2025
db3e969
fix: normalize legacy data URIs in renderer to prevent double-encoding
Dw9 Nov 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 32 additions & 2 deletions src/main/presenter/threadPresenter/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -575,12 +575,42 @@ export class ThreadPresenter implements IThreadPresenter {
}
}

if (image_data) {
if (image_data?.data) {
const rawData = image_data.data ?? ''
let normalizedData = rawData
let normalizedMimeType = image_data.mimeType?.trim() ?? ''

// Handle URLs (imgcache://, http://, https://)
if (
rawData.startsWith('imgcache://') ||
rawData.startsWith('http://') ||
rawData.startsWith('https://')
) {
normalizedMimeType = 'deepchat/image-url'
}
// Handle data URIs: extract base64 content and mime type
else if (rawData.startsWith('data:image/')) {
const match = rawData.match(/^data:([^;]+);base64,(.*)$/)
if (match?.[1] && match?.[2]) {
normalizedMimeType = match[1]
normalizedData = match[2]
}
}
// Fallback to image/png if no mime type is provided
else if (!normalizedMimeType) {
normalizedMimeType = 'image/png'
}

const normalizedImageData = {
data: normalizedData,
mimeType: normalizedMimeType
}
Comment on lines +583 to +607
Copy link
Contributor

@coderabbitai coderabbitai bot Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Fix data URI normalization edge cases.
The regex only matches data:mime;base64,... with no extra parameters. Providers often emit data:image/png;charset=utf-8;base64,..., so we fail to strip the header, fall back to image/png, and persist the entire data URI. The renderer then builds data:image/png;base64,data:image/png;charset=utf-8;base64,..., which ships a broken image. Please widen the match to tolerate additional ;param segments (and keep the real mime type) before falling back to the default.

-      else if (rawData.startsWith('data:image/')) {
-        const match = rawData.match(/^data:([^;]+);base64,(.*)$/)
+      else if (rawData.startsWith('data:image/')) {
+        const match = rawData.match(/^data:([^;,]+)(?:;[^,]+)*;base64,(.*)$/i)
         if (match?.[1] && match?.[2]) {
           normalizedMimeType = match[1]
           normalizedData = match[2]
         }
       }

That ensures any extra parameters (charset, name, etc.) are handled and the payload stays in the expected raw/base64 form.

🤖 Prompt for AI Agents 
In src/main/presenter/threadPresenter/index.ts around lines 583 to 607, the data
URI handling only matches headers of the exact form data:mime;base64,... which
breaks when extra parameters like ;charset=utf-8 appear; update the parsing to
accept optional semicolon-separated parameters before the final ;base64 token,
capture the actual mime type (first part after data:) and the base64 payload
(everything after the final comma), and then set normalizedMimeType to that
captured mime type and normalizedData to the base64 content so the header is
stripped correctly and fallback logic is only used when no mime type can be
determined.

const imageBlock: AssistantMessageBlock = {
type: 'image',
status: 'success',
timestamp: currentTime,
content: image_data
content: 'image',
image_data: normalizedImageData
}
state.message.content.push(imageBlock)
}
Expand Down
149 changes: 139 additions & 10 deletions src/renderer/src/components/message/MessageBlockImage.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@
<div class="flex flex-col space-y-2">
<!-- 图片加载区域 -->
<div class="flex justify-center">
<template v-if="block.image_data">
<template v-if="resolvedImageData">
<img
v-if="block.image_data.mimeType === 'deepchat/image-url'"
:src="`${block.image_data.data}`"
v-if="resolvedImageData.mimeType === 'deepchat/image-url'"
:src="`${resolvedImageData.data}`"
class="max-w-[400px] rounded-md cursor-pointer hover:shadow-md transition-shadow"
@click="openFullImage"
@error="handleImageError"
/>
<img
v-else
:src="`data:${block.image_data.mimeType};base64,${block.image_data.data}`"
:src="`data:${resolvedImageData.mimeType};base64,${resolvedImageData.data}`"
class="max-w-[400px] rounded-md cursor-pointer hover:shadow-md transition-shadow"
@click="openFullImage"
@error="handleImageError"
Expand All @@ -41,15 +41,15 @@
</DialogTitle>
</DialogHeader>
<div class="flex items-center justify-center">
<template v-if="block.image_data">
<template v-if="resolvedImageData">
<img
v-if="block.image_data.mimeType === 'deepchat/image-url'"
:src="block.image_data.data"
v-if="resolvedImageData.mimeType === 'deepchat/image-url'"
:src="resolvedImageData.data"
class="rounded-md max-h-[80vh] max-w-full object-contain"
/>
<img
v-else
:src="`data:${block.image_data.mimeType};base64,${block.image_data.data}`"
:src="`data:${resolvedImageData.mimeType};base64,${resolvedImageData.data}`"
class="rounded-md max-h-[80vh] max-w-full object-contain"
/>
</template>
Expand All @@ -60,7 +60,7 @@
</template>

<script setup lang="ts">
import { ref } from 'vue'
import { computed, ref } from 'vue'
import { Icon } from '@iconify/vue'
import { AssistantMessageBlock } from '@shared/chat'
import { useI18n } from 'vue-i18n'
Expand Down Expand Up @@ -90,15 +90,144 @@ const props = defineProps<{
threadId?: string
}>()

type LegacyImageBlockContent = {
data?: string
mimeType?: string
}

const imageError = ref(false)
const showFullImage = ref(false)

const inferMimeType = (data: string, mimeType?: string): string => {
if (mimeType && mimeType.trim().length > 0) {
return mimeType
}

if (data.startsWith('imgcache://') || data.startsWith('http://') || data.startsWith('https://')) {
return 'deepchat/image-url'
}

if (data.startsWith('data:image/')) {
const match = data.match(/^data:([^;]+);base64,(.*)$/)
if (match?.[1]) {
return match[1]
}
}

return 'image/png'
}

const resolvedImageData = computed(() => {
// Handle new format with image_data field
if (props.block.image_data?.data) {
const rawData = props.block.image_data.data

// Handle URLs
if (
rawData.startsWith('imgcache://') ||
rawData.startsWith('http://') ||
rawData.startsWith('https://')
) {
return {
data: rawData,
mimeType: 'deepchat/image-url'
}
}

let normalizedData = rawData
let normalizedMimeType = inferMimeType(rawData, props.block.image_data.mimeType)

// Handle legacy data URIs that may still exist in persisted data
if (rawData.startsWith('data:image/')) {
const match = rawData.match(/^data:([^;]+);base64,(.*)$/)
if (match?.[1] && match?.[2]) {
normalizedMimeType = match[1]
normalizedData = match[2]
}
}

return {
data: normalizedData,
mimeType: normalizedMimeType
}
}

// Handle legacy formats (for backward compatibility)
const content = props.block.content

if (content && typeof content === 'object' && 'data' in (content as LegacyImageBlockContent)) {
const legacyContent = content as LegacyImageBlockContent
if (legacyContent.data) {
const rawData = legacyContent.data

// Handle URLs
if (
rawData.startsWith('imgcache://') ||
rawData.startsWith('http://') ||
rawData.startsWith('https://')
) {
return {
data: rawData,
mimeType: 'deepchat/image-url'
}
}

let normalizedData = rawData
let normalizedMimeType = inferMimeType(rawData, legacyContent.mimeType)

// Handle data URIs
if (rawData.startsWith('data:image/')) {
const match = rawData.match(/^data:([^;]+);base64,(.*)$/)
if (match?.[1] && match?.[2]) {
normalizedMimeType = match[1]
normalizedData = match[2]
}
}

return {
data: normalizedData,
mimeType: normalizedMimeType
}
}
}

if (typeof content === 'string' && content.length > 0) {
if (content.startsWith('data:image/')) {
const match = content.match(/^data:([^;]+);base64,(.*)$/)
if (match?.[1] && match?.[2]) {
return {
data: match[2],
mimeType: match[1]
}
}
}

if (
content.startsWith('imgcache://') ||
content.startsWith('http://') ||
content.startsWith('https://')
) {
return {
data: content,
mimeType: 'deepchat/image-url'
}
}

return {
data: content,
mimeType: inferMimeType(content)
}
}

return null
Comment on lines +101 to +222
Copy link
Contributor

@coderabbitai coderabbitai bot Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Normalize legacy data URIs before rendering.
inferMimeType and the downstream matches still demand data:mime;base64,... with no extra parameters. When a legacy block contains data:image/png;charset=utf-8;base64,..., inferMimeType defaults to image/png and we return the untouched data URI. The template then renders data:image/png;base64,data:image/png;charset=utf-8;base64,..., so the image breaks. Please generalize both the mime-type extraction and the normalization regex so we always strip the header (while grabbing the real mime), even when optional parameters are present.

const inferMimeType = (data: string, mimeType?: string): string => {
  if (mimeType && mimeType.trim().length > 0) {
    return mimeType
  }

  if (data.startsWith('imgcache://') || data.startsWith('http://') || data.startsWith('https://')) {
    return 'deepchat/image-url'
  }

-  if (data.startsWith('data:image/')) {
-    const match = data.match(/^data:([^;]+);base64,(.*)$/)
-    if (match?.[1]) {
-      return match[1]
-    }
-  }
+  if (data.startsWith('data:image/')) {
+    const header = data.slice(5, data.indexOf(',')) ?? ''
+    const mime = header.split(';')[0]?.trim()
+    if (mime) {
+      return mime
+    }
+  }

  return 'image/png'
}
…
-    if (rawData.startsWith('data:image/')) {
-      const match = rawData.match(/^data:([^;]+);base64,(.*)$/)
+    if (rawData.startsWith('data:image/')) {
+      const match = rawData.match(/^data:([^;,]+)(?:;[^,]+)*;base64,(.*)$/i)
       if (match?.[1] && match?.[2]) {
         normalizedMimeType = match[1]
         normalizedData = match[2]
       }
     }

Apply the same regex tweak in the legacy branches below so every data URI path uses the sanitized payload.

Committable suggestion skipped: line range outside the PR's diff.

🤖 Prompt for AI Agents 
In src/renderer/src/components/message/MessageBlockImage.vue around lines 101 to
222, the data URI regex only matches headers without optional parameters,
causing duplicated headers when params like ;charset=utf-8 are present; update
the regex used in inferMimeType and in all data-URI normalization branches to
match optional parameters before ;base64 and capture the real mime and payload
(for example use a pattern that captures the mime-type as the first group and
the base64 payload as the second by allowing optional semicolon-prefixed params
before ;base64), then return the captured mime as the mimeType and replace the
data with the captured payload (ensure you apply this same regex change in
inferMimeType, the new image_data branch, the legacy content branch, and the
string content branch so every path strips the full header and keeps only the
base64 payload).

})

const handleImageError = () => {
imageError.value = true
}

const openFullImage = () => {
if (props.block.image_data) {
if (resolvedImageData.value) {
showFullImage.value = true
}
}
Expand Down
5 changes: 3 additions & 2 deletions test/renderer/message/__snapshots__/messageBlockSnapshot.test.ts.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,10 @@ exports[`Message Block Data Structure Snapshot Tests > Block Structure Validatio

exports[`Message Block Data Structure Snapshot Tests > Block Structure Validation > should create consistent image block structure 1`] = `
{
"content": "image",
"image_data": {
"data": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==",
"mimeType": "image/png",
"data": "imgcache://test-image.png",
"mimeType": "deepchat/image-url",
},
"status": "success",
"timestamp": 1704067200000,
Expand Down
5 changes: 3 additions & 2 deletions test/renderer/message/messageBlockSnapshot.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,10 @@ describe('Message Block Data Structure Snapshot Tests', () => {
type: 'image',
status: 'success',
timestamp: 1704067200000,
content: 'image',
image_data: {
data: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==',
mimeType: 'image/png'
data: 'imgcache://test-image.png',
mimeType: 'deepchat/image-url'
}
}

Expand Down