[MEDI] SemanticSimilarityChunker raising OllamaSharp.Models.Exceptions.OllamaException: the input length exceeds the context length

[quicoli]

Description

Hi.

Probably this is caused by some misunderstanding mine. If it is please explain it to me.
I'm using the AI chat template as it is. The only changes: I'm setting the MaxTokensPerChunk and I'm using ElBruno.OllamaSharp.Extensions to extend the timeout.

IChatClient chatClient = new OllamaApiClient(new Uri("http://localhost:11434"),
    "llama3.2").WithLongTimeout();
IEmbeddingGenerator<string, Embedding<float>> embeddingGenerator = new OllamaApiClient( new Uri("http://localhost:11434"),
    "all-minilm").WithLongTimeout();

VectorStore: SqliteVectorStore

Note: I tried to use a PostgreSQLVectorStore but results are worst, even bringing MaxTokensPerChunk to 100 or less raises the exception.

In DataIngestor.IngestDataAsync I have the following:

 using var pipeline = new IngestionPipeline<string>(
     reader: new DocumentReader(directory),
     chunker: new SemanticSimilarityChunker(embeddingGenerator,
         new IngestionChunkerOptions(TiktokenTokenizer.CreateForModel("gpt-4o")){
             MaxTokensPerChunk = 200    }
         ),
     writer: writer,
     loggerFactory: loggerFactory);

 await foreach (var result in pipeline.ProcessAsync(directory, searchPattern))
 {
     logger.LogInformation("Completed processing '{id}'. Succeeded: '{succeeded}'.", result.DocumentId, result.Succeeded);
 }

I have 3 pdf files to be ingested. 2 of them are perfectly ingested. 1 is raising the exception listed below.

If I change to another checker like DocumentTokenChunker, all 3 are ingested.

Is SemanticSimilarityChunker ignoring the MaxTokensPerChunk?
Am I doing anything wrong?

Microsoft.Extensions.DataIngestion.IngestionPipeline[6]
      An error occurred while ingesting document 'my.pdf'.
      OllamaSharp.Models.Exceptions.OllamaException: the input length exceeds the context length
         at OllamaSharp.OllamaApiClient.EnsureSuccessStatusCodeAsync(HttpResponseMessage response)
         at OllamaSharp.OllamaApiClient.SendToOllamaAsync(HttpRequestMessage requestMessage, OllamaRequest ollamaRequest, HttpCompletionOption completionOption, CancellationToken cancellationToken)
         at OllamaSharp.OllamaApiClient.PostAsync[TRequest,TResponse](String endpoint, TRequest ollamaRequest, CancellationToken cancellationToken)
         at OllamaSharp.OllamaApiClient.Microsoft.Extensions.AI.IEmbeddingGenerator<System.String,Microsoft.Extensions.AI.Embedding<System.Single>>.GenerateAsync(IEnumerable`1 values, EmbeddingGenerationOptions options, CancellationToken cancellationToken)
         at Microsoft.Extensions.DataIngestion.Chunkers.SemanticSimilarityChunker.CalculateDistancesAsync(IngestionDocument documents, CancellationToken cancellationToken)
         at Microsoft.Extensions.DataIngestion.Chunkers.SemanticSimilarityChunker.ProcessAsync(IngestionDocument document, CancellationToken cancellationToken)+MoveNext()
         at Microsoft.Extensions.DataIngestion.Chunkers.SemanticSimilarityChunker.ProcessAsync(IngestionDocument document, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Microsoft.Extensions.DataIngestion.VectorStoreWriter`1.WriteAsync(IAsyncEnumerable`1 chunks, CancellationToken cancellationToken)
         at Microsoft.Extensions.DataIngestion.VectorStoreWriter`1.WriteAsync(IAsyncEnumerable`1 chunks, CancellationToken cancellationToken)
         at Microsoft.Extensions.DataIngestion.IngestionPipeline`1.IngestAsync(IngestionDocument document, Activity parentActivity, CancellationToken cancellationToken)
         at Microsoft.Extensions.DataIngestion.IngestionPipeline`1.ProcessAsync(IEnumerable`1 files, Activity rootActivity, CancellationToken cancellationToken)+MoveNext()

Reproduction Steps

Use a relative sized PDF file
use SemanticSimilarityChunker
use local ollama from the AI Chat template wizard
use Sqlite Vector Store

Expected behavior

All 'big' PDFs should be ingested.

Actual behavior

The biggest among those is not ingested

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response