The currentSize accumulation in TokenCountBatchingStrategy is inaccurate

[libra19911018]

When I was examining the source code of TokenCountBatchingStrategy, I identified an issue with the batching and saving process in the batch() method.

When creating a new instance of currentBatch, the currentSize is reset to 0, but the Document documents traversed during the current iteration will be added to the new instance of currentBatch

This results in a situation where, during the loop, each time the currentSize is reset, one token length of the current iteration through the Document is not accumulated

coding node：
for (Map.Entry<Document, Integer> entry : documentTokens.entrySet()) {
Document document = entry.getKey();
currentSize += entry.getValue();
if (currentSize > this.maxInputTokenCount) {
batches.add(currentBatch);
currentBatch = new ArrayList<>();
currentSize = 0;
}
currentBatch.add(document);
}

I apologize for my poor English. This is the bug description I generated through machine translation

[OiPunk]

Submitted a fix in #5528.

Root cause: when currentSize > maxInputTokenCount triggers a new batch, currentSize resets to 0 but the triggering document still gets added to the new batch — so its token count is silently dropped from the running total.

Fix: reset currentSize = entry.getValue() instead of 0.

AI-assisted testing, AI-assisted review