.Net: Fixes Qdrant 1.17 issue with returning empty vectors

[TsengSR]

Motivation and Context

With Qdrant server 1.17 Embeddings stopped being returned.

Fixes #13587

Description

The VectorOutput.Data property became deprecated in Qdrant.Client 1.17. The alternative is to use VectorOutput.Dense.Data instead or VectorOutput.GetDenseVector() (from Qdrant.Client 1.16.0 or later).

Since the .Dense already exyists in Qdrant.Client 1.15.1 that is used by the current connector, I used these. The .GetDenseVector() also returns nullable DenseVector, so I was unsure how to handle in context of Semantic Kernel, as as far as I see it Semantic Kernel only supports Dense Vectors (the other two are .Sparse for sparse vectors and .MultiDense).

So this may need special handling in future to throw an exception, when the server returns one of the other two vector types and not dense vectors.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

I'm currently unable (or rather I don't know how) to run the local integration/unit tests against the qdrant database. But I validated the change in my personal project mentioned in #13587 by directly referencing the connector in my project and it correctly returned the vectors on the sample code where it failed previously

    VectorStoreCollection<ulong, VectorModels.Project> projects = await GetProjectsCollection();
    VectorModels.Project? project = await projects.GetAsync((ulong)projectId, new()
    {
        IncludeVectors = true
    });

    // project at this poind hat 0 sized Embedding property
    if (project is null)
    {
        return;
    }
    await projects.UpsertAsync(project);

[westey-m]

Thanks @TsengSR, seems like a straightforward enough change. The integration tests automatically spin up a qdrant container in docker to test against locally. So running the tests should do all the work, as long as you have docker installed.
They also run during the PR merge process, so we can check then if nothing has regressed.

Looks like a few of the unit tests need updating though to match the latest code.

[TsengSR]

Okay, seems the test can run (net472 fails, seems tls validation issue with the qdrant store), but there seems to be issues.

The containers spinning up during the tests are qdrant 1.13.4 (shows under the qdrant logo, when they spin up) which probably still only suports the old format. I had 1.16.3 previously and upgraded to 1.17 when it happened.

But all I could find is

semantic-kernel/dotnet/samples/Concepts/Memory/VectorStoreFixtures/VectorStoreInfra.cs

Lines 65 to 72 in 4d67592

	await client.Images.CreateImageAsync(
	new ImagesCreateParameters
	{
	FromImage = "qdrant/qdrant",
	Tag = "latest",
	},
	null,
	new Progress<JSONMessage>());

which should be 1.17.0, but its not what runs in the container. Any ideas/pointers?

[westey-m]

@TsengSR, check out https://github.com/microsoft/semantic-kernel/blob/main/dotnet/test/VectorData/Qdrant.ConformanceTests/Support/QdrantTestStore.cs#L21.

We use https://testcontainers.com/modules/qdrant/?language=dotnet. I see we don't pass a version into QdrantBuilder, so I would have expected it to use the latest version, but I guess not. It's possible to pass a specific version though:

var qdrantContainer = new QdrantBuilder("qdrant/qdrant:v1.13.4")
  .Build();

The code you shared is actually samples, but the tests are all under https://github.com/microsoft/semantic-kernel/blob/main/dotnet/test/VectorData

[TsengSR]

@TsengSR, check out https://github.com/microsoft/semantic-kernel/blob/main/dotnet/test/VectorData/Qdrant.ConformanceTests/Support/QdrantTestStore.cs#L21.

We use https://testcontainers.com/modules/qdrant/?language=dotnet. I see we don't pass a version into QdrantBuilder, so I would have expected it to use the latest version, but I guess not. It's possible to pass a specific version though:

var qdrantContainer = new QdrantBuilder("qdrant/qdrant:v1.13.4")
  .Build();

The code you shared is actually samples, but the tests are all under https://github.com/microsoft/semantic-kernel/blob/main/dotnet/test/VectorData

Thanks, that helped a lot. The current Testcontainers.Qdrant didn't had any constructor, updated to 4.10.0 locally which contain these (and deprecates the parameterless one). Its still odd ":latest" tag gave v1.16.3, but ":v1.17.0" pulled the correct image.

Had to change the setup slightly for Weaviate Tests, as UntilPortIsAvailable was removed, see comment.

Fixed the test issues by doing pattern matching whether .Dense is null or not and otherwise fall back to .Data. .Data will eventually be removed at some point in future as its deprecated from Qdrant.Client: 1.16.0 onwards, but for l left the Qdrant.Client at 1.15.1

update:
One thing I noticed is that QdrantUnitTests don't run against the database and something internally sets the VectorOutput.Data, so these tests were failing when not having the { Data: not null} => fall back. Didn't bother yet to look where that happens, but its something to consider when VectorOutput.Data gets removed eventually.

[roji]

LGTM, but see the global.json change.

[roji]

LGTM, thanks!

@westey-m waiting for your approval now.

[roji]

@TsengSR see build error in CI.

[TsengSR]

@TsengSR see build error in CI.

Fixed that. There are 4 tests failing on pinecone, but this also fail when I run them against main branch. Not sure its related to my machine setup. For qdrant net472 most fail too, but net10 they succeed. Hope it will work though.