optimize title generation speed #1325
Open
+39
−5
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I notice that the title-generation is block the response flow and become the bottle neck of response speed:

Problem
The chatbot API endpoint was experiencing severe performance issues with response times exceeding 84 seconds for new chat creation. The root cause was blocking title generation - the system waited for an LLM API call to generate a chat title before returning the stream response.
Solution
Made title generation non-blocking using a two-phase approach:
Generate a temporary title immediately from the first 60 characters of the user's message
Generate the LLM-based title asynchronously in the background and update when ready
This allows the chat stream to start immediately while the polished title is generated asynchronously.
Changes
lib/db/queries.ts
Added updateChatTitleById function to update chat titles after creation
app/(chat)/api/chat/route.ts
Replaced blocking await generateTitleFromUserMessage() with immediate temporary title generation
Added asynchronous real title generation that updates the chat when ready
Added imports for updateChatTitleById and getTextFromMessage
Performance Impact
Before: 19+ seconds (blocking on LLM title generation)
After: <1 second (immediate response with temporary title, real title updates in background)