[Bug]

[390620652]

Context / Scenario

In versions such as 0.98 and 0.97, import Chinese files like docx/txt into the vector database.

await memory.ImportDocumentAsync(new Document(docId).AddFile( filePath), index);

What happened?

Some Chinese characters in the vector database may appear duplicated, but it was completely normal below v0.96.

Importance

I cannot use Kernel Memory

Platform, Language, Versions

.net8/9 , v0.98.* and v0.97.*

Relevant log output

[HeadHorn]

I have the same issue. And the partition becomes normal (no duplicated characters) if I set the OverlappingTokens of TextPartitioning to 0.