Kernel Memory is broken with latest nugets

[JohnGalt1717]

Using the 0.8 release of LlamaSharp and Kernal-Memory with the samples there is an error because the LlamaSharpTextEmbeddingGeneration doesn't implement the Attributes property.

I took the source and created my own and added this:

public IReadOnlyDictionary<string, string> Attributes => new Dictionary<string, string>();

So it wouldn't error.

But no matter what model I use I get "INFO NOT FOUND." (I've tried kai-7b-instruct.Q5_K_M.gguf, llama-2-7b-32k-instruct.Q6_K.gguf, llama-2-7b-chat.Q6_K.gguf and a few others)

I've tried loading just text, an html file, and a web page to no avail.

[AsakusaRinne]

Hi, which version of kernel-memory did you install?

[xbotter]

The new version of the semantic kernel has added some breaking change, and the latest package has been updated in the PR #306

[JohnGalt1717]

I also notice that there is no way to add grammar to this right now so AskAsync can't force good output of JSON as an example. Can this be added? Same with MainGPU and the other settings on the context. And there are a few optoins missing (TOP_K) for the executer as well that you can't get to with Microsoft's With(new TextGenerationOptions). Perhaps an extended one for LlamaSharp that has all of the rest of the settings that are available?

And I'm noticing, now that I have it hacked to work(ish) that a memory.AskAsync works, but if you do it again (i.e. memory.AskAsync) for a second question against the data, I just get a ton of \n characters and no valid response. I'm not sure if that's kernel-memory doing that, or if it's this that's causing the issue. (If it helps, I'm in GPU only mode with dual nvidia 8 gb cards.)

[xbotter]

I also notice that there is no way to add grammar to this right now so AskAsync can't force good output of JSON as an example. Can this be added? Same with MainGPU and the other settings on the context. And there are a few optoins missing (TOP_K) for the executer as well that you can't get to with Microsoft's With(new TextGenerationOptions). Perhaps an extended one for LlamaSharp that has all of the rest of the settings that are available?

Yes, the current version of Kernel Memory does not provide more customizable options.
However, there are currently two ways to format the output: one is by reformatting the Ask Result through LLM, and the other is by using the Search method to obtain search results from documents and then organizing prompts to generate desired results.

And I'm noticing, now that I have it hacked to work(ish) that a memory.AskAsync works, but if you do it again (i.e. memory.AskAsync) for a second question against the data, I just get a ton of \n characters and no valid response. I'm not sure if that's kernel-memory doing that, or if it's this that's causing the issue. (If it helps, I'm in GPU only mode with dual nvidia 8 gb cards.)

It seems to be related to the state management of the model. @AsakusaRinne any idea?

[JohnGalt1717]

On the first issue I was able to take your code, and manually add a grammar to it. Is there a way that we can just expose Grammar in the LLamaSharpConfig for right now that will get passed through? Same with MainGPU, and TOP_K?

[xbotter]

On the first issue I was able to take your code, and manually add a grammar to it. Is there a way that we can just expose Grammar in the LLamaSharpConfig for right now that will get passed through? Same with MainGPU, and TOP_K?

👍 Good idea, I seem to have a solution to the issue #289. 😃 Thank you for your suggestion.

[JohnGalt1717]

On the issue with getting the endless \n on the second AskAsync, it also does it with embeddings. If you import 2 documents in a row, the second one will fail. It appears that it fails because it's failing in the text generation the same way and it appears that it is something in the context, because if I regen the model and the executor every time, it doesn't fix the problem

I also noticed that EmbeddingMode is not being set on the embedder ModelParams which makes it much slower.

[JohnGalt1717]

Update: It isn't the context, and the native handle shows that it's still valid. I've tried to replace all of the shared versions of the context and model with new versions spun up per request, and there was no difference.

[AsakusaRinne]

And I'm noticing, now that I have it hacked to work(ish) that a memory.AskAsync works, but if you do it again (i.e. memory.AskAsync) for a second question against the data, I just get a ton of \n characters and no valid response. I'm not sure if that's kernel-memory doing that, or if it's this that's causing the issue. (If it helps, I'm in GPU only mode with dual nvidia 8 gb cards.)

It seems to be related to the state management of the model. @AsakusaRinne any idea?

It needs a deep dive. Did you use WithLLamaSharpDefaults of kernel-memory integration?

I also noticed that EmbeddingMode is not being set on the embedder ModelParams which makes it much slower.

It's absolutely an unexpected behaviour! Could you please share your code with us to re-preoduce it? (a minimal example is better)

[JohnGalt1717]

Yes, I used WithlllamaSharpDefaults.

I'm using a console app

var modelPath = Path.GetFullPath(Path.Combine(Assembly.GetEntryAssembly()!.Location, "..", "..", "..", "..", @"openhermes-2.5-mistral-7b-16k.Q5_0.gguf"));

var builder = new KernelMemoryBuilder()
        .WithQdrant(new QdrantConfig
        {
            APIKey = "",
            Endpoint = "http://localhost:6333",

        })
        .WithLLamaSharpDefaults(new LLamaSharpConfig(modelPath)
        {
            ContextSize = 16_384,
            Seed = 1337,
            GpuLayerCount = 20,
        })
        .With(new TextPartitioningOptions
        {
            MaxTokensPerLine = 100,
            MaxTokensPerParagraph = 300,
            OverlappingTokens = 30,
        })
        .With(new TextGenerationOptions
        {
            MaxTokens = -1,
            TopP = 0.95F,
            Temperature = 0.0,
            FrequencyPenalty = 1.1F,
            StopSequences = ["\n"],
            ResultsPerPrompt = 1
        });

    var memory = builder
                    .BuildServerlessClient();

await memory.ImportTextAsync("The quick brown fox jumps over the lazy dog.");

await memory.AskAsync("Who jumps over the lazy dog?");

await memory.AskAsync("What kind of dog is it?");

The second AskAsync will return a ton of \n\n no matter the model at least when using CUDA.

The second part about EmbeddingMode is just in the setup of the WithLLamaSharpDefaults it doesn't split out the parameters so they both use the same params for text creation and embedding instead of optimizing out the embedding params.

[AsakusaRinne]

@xbotter I think we should watch the prompt put into our model in the second run with context size reduced to 4096. Could you please take a look? I'm on duty this weekend.

[xbotter]

@xbotter I think we should watch the prompt put into our model in the second run with context size reduced to 4096. Could you please take a look? I'm on duty this weekend.

This can only be resolved from the kernel memory . I have already submitted an issue microsoft/kernel-memory#164 and waiting for further updates.

[JohnGalt1717]

If it matters I’m not changing the context length and it still happens.