Streaming TTS: The front-end receives the StreamData repeatedly

[ganxiaozhe]

Description

I am using SvelteKit 2.0 and vercel/ai 3.1.22.

On the backend, I use StreamingTextResponse to merge AIStream and StreamData, where the former transmits a text stream and the latter transmits an audio stream (base64 buffer). The audio stream takes a longer time to transmit.

The issue is that when the AIStream completes, the data in StreamData is repeatedly received by the frontend.

The solution I am currently using is to pass two keys, voice_buffer_base64 and voice_buffer_index, in data.append to filter out duplicate content on the frontend.

Thank you for your attention to this matter.

Code example

import { createOpenAI } from '@ai-sdk/openai';
import { createOllama } from 'ollama-ai-provider';
import { StreamData, StreamingTextResponse, streamText } from 'ai';
import { env } from '$env/dynamic/private';
import type { RequestHandler } from './$types.js';
import { StreamPlayer } from '$lib/core/tts-stream.server.js';

const openai = createOpenAI({
  baseURL: env.OPENAI_BASE_URL ?? undefined,
  apiKey: env.OPENAI_API_KEY ?? ''
});

const gxz_prompt = `你是一个私人助手，你的名字叫「甘小小蔗」。
你的任务是尽可能帮助用户解决其提出的问题。
你的回答将被合成为语音，所以**使用口语化风格，并禁止使用 Markdown 或 Latex 语法**。
你的回答可以是中文或英文，这取决于用户使用的语言。`;

export const POST = (async ({ request }) => {
  const { messages } = await request.json();

  const result = await streamText({
    model: openai('gpt-4o'),
    messages,
    system: gxz_prompt
  });

  const data = new StreamData();
  let bufferCount = 0;
  const streamPlayer = new StreamPlayer({
    onBuffer:(buffer)=>{
      data.append({
        voice_buffer_base64: buffer.toString('base64'),
        voice_buffer_index: bufferCount
      });
      bufferCount++;
    },
    onEnd:()=>{
      console.log(`[StreamPlayer] Completion, buffer counts: ${bufferCount}`);
      data.close();
    }
  });
  await streamPlayer.init();

  const stream = result.toAIStream({
    onText(text){
      streamPlayer.send(text);
    },
    onFinal(_){
      streamPlayer.stop();
    },
    onCompletion(_){
      console.log(`[AIStream] [${Date.now()}] Completion`);
    }
  });

  return new StreamingTextResponse(stream, {}, data);
}) satisfies RequestHandler;

Additional context

No response