Skip to content

fix(spring-ai): emit final response on stream completion instead of punctuation heuristic - #1620

Open
caps-xia wants to merge 1 commit into
google:mainfrom
caps-xia:fix/1608-stream-completion-final
Open

caps-xia wants to merge 1 commit into
google:mainfrom
caps-xia:fix/1608-stream-completion-final

Conversation

@caps-xia

@caps-xia caps-xia commented Oct 6, 2026

Copy link
Copy Markdown

Link to Issue or Description of Change

Closes: #1608

Problem:

MessageConverter#toLlmResponse classifies each streaming chunk as partial/final via isPartialResponse, an ASCII terminal-punctuation heuristic (., !, ?, \n). Two failure modes:

  • A turn ending with CJK terminal punctuation (。!?) is never classified final → the response is never persisted to the session, and the next turn has no memory of what the model said.
  • Conversely, a mid-stream chunk that happens to end with . fires a premature final inside a longer answer.

There is also a related defect: since Runner streams partial events but never persists them (by design — see the comment in Runner), and the punctuation-"final" chunk only carries its own delta text, sessions today persist only the last punctuation-ending chunk fragment instead of the full answer.

Solution:

Stop classifying chunks mid-stream. Streaming chunks are always partial; StreamingResponseAggregator (already present but never wired) produces the single aggregated final response on stream completion:

  • SpringAI streaming path: feed each converted chunk to a per-request aggregator and forward it; on the Flux completion callback, emit aggregator.getFinalResponse() — exactly one final event carrying the full text and accumulated tool calls — before onComplete().
  • MessageConverter: streaming chunks are always partial; the old punctuation heuristic is retained (unused) for rollback/reference.
  • The aggregated final carries the stream's last non-empty usageMetadata, which would otherwise be lost with the unpersisted partial events.

Why the completion signal: it is the only end-of-turn signal guaranteed across providers — finishReason spelling/availability varies (stop/end_turn/STOP; tool-call turns use tool_calls; some gateways omit it entirely), and punctuation is locale-dependent. This also matches the framework's intended design: Runner already expects "the final aggregated event" to carry the full content, and the Gemini bridge (Gemini.java) uses the same partials + aggregated-final pattern via its own StreamingResponseAggregator.

Behavioral notes:

  • Persistence timing in the correct case is unchanged: the final chunk and stream completion are the same moment.
  • Tool-call turns: accumulated function calls ride on the aggregated final, so execution timing is unchanged (tool calls arrive with the last chunks of a turn).
  • Mid-stream errors now produce no final, so truncated turns are no longer persisted as if complete.

Testing Plan

Unit Tests:

  • testGenerateContentStreamingCjkTerminalPunctuationPersistsFinal — a CJK-terminated turn produces exactly one final (persisted) event with the full text.
  • testToLlmResponseStreamingAlwaysMarksChunksPartial — streaming chunks are always partial regardless of punctuation.
  • testGenerateContentStreamingFinalCarriesUsageMetadata — usage metadata from the last chunk survives onto the aggregated final.
  • Updated testGenerateContentStreaming and testGenerateContentStreamingBackpressure for the final event (3 chunks + 1 final; 1000 + 1).
  • Full module suite passes locally: mvn -pl contrib/spring-ai -am test → 217 tests, 0 failures, BUILD SUCCESS.

Manual End-to-End (E2E) Tests:

Reproduced with a real OpenAI-compatible gateway and a Chinese-language workflow:

  • Before: responses ending with 。!? never persisted to the session (next turn lost context); non-CJK turns persisted only the last punctuation-ending fragment.
  • After: every turn persists exactly once with the full text; live partial rendering is unaffected.

Checklist

  • I have read the CONTRIBUTING.md document.
  • My pull request contains a single commit.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules. — N/A, no dependent changes.

Additional context

Implements option 2 as discussed on the issue (root fix — stream-completion-based final detection; @hirematha confirmed the reproduction there). isPartialResponse is kept unused as rollback/reference documentation of the previous behavior. Happy to also provide the interim CJK-punctuation patch as a separate PR if maintainers want a short-term mitigation.

…unctuation heuristic

MessageConverter.isPartialResponse classified streaming chunks by ASCII
terminal punctuation (. ! ? \n). Turns ending with CJK terminal punctuation
(。!?) were never marked final, so the response was silently not persisted
to the session; conversely a '.' inside a longer answer fired a premature
final mid-stream.

Stream chunks are now always partial; StreamingResponseAggregator
(already present, previously unused) produces the single final response
on stream completion. The completion signal is authoritative regardless
of locale, provider vocabulary (stop/end_turn/STOP/tool_calls) and
gateways that drop finishReason. Persistence timing is unchanged: the
final event was already the last chunk in the correct case. Mid-stream
errors now produce no final, instead of persisting a truncated response.

isPartialResponse is deprecated and no longer called.

Fixes google#1608
@hirematha hirematha self-assigned this Oct 8, 2026
@hirematha

Copy link
Copy Markdown

Hi @caps-xia, thank you for your contribution and quick fix.It is currently under review by our team. We will update you if any additional details are needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[spring-ai] Streaming responses ending with CJK punctuation (。!?) are misclassified as partial and never persisted to the session

2 participants