Skip to content

Conversation

@Moshiii
Copy link

@Moshiii Moshiii commented Nov 22, 2025

I notice that the title-generation is block the response flow and become the bottle neck of response speed:
Screenshot 2025-11-23 at 3 40 46 AM

Problem
The chatbot API endpoint was experiencing severe performance issues with response times exceeding 84 seconds for new chat creation. The root cause was blocking title generation - the system waited for an LLM API call to generate a chat title before returning the stream response.

Solution
Made title generation non-blocking using a two-phase approach:
Generate a temporary title immediately from the first 60 characters of the user's message
Generate the LLM-based title asynchronously in the background and update when ready
This allows the chat stream to start immediately while the polished title is generated asynchronously.

Changes
lib/db/queries.ts
Added updateChatTitleById function to update chat titles after creation
app/(chat)/api/chat/route.ts
Replaced blocking await generateTitleFromUserMessage() with immediate temporary title generation
Added asynchronous real title generation that updates the chat when ready
Added imports for updateChatTitleById and getTextFromMessage

Performance Impact
Before: 19+ seconds (blocking on LLM title generation)
After: <1 second (immediate response with temporary title, real title updates in background)

@vercel
Copy link

vercel bot commented Nov 22, 2025

@Moshiii is attempting to deploy a commit to the Templates Test vtest314 Team on Vercel.

A member of the Team first needs to authorize it.

@Moshiii
Copy link
Author

Moshiii commented Nov 22, 2025

@shaper @rauchg @tomocchino @marcusschiesser Please review this response optimization-related PR

@Moshiii
Copy link
Author

Moshiii commented Nov 23, 2025

@MrAllgoodWilson what should I do to get it approved? Please advice.

@MrAllgoodWilson
Copy link

I approve

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants