.NET Allow ISKFunction to return IAsyncEnumerable

[wayne-o]

Important

Labeled High because it will not require a breaking change, but it's very important to complete by v1.0.0

Is there any way currently to allow ISKFunction to return an IAsyncEnumerable? If not is there any plan to?

[wayne-o]

Or to put it another way, is there any way to get the gpt client to use the streaming endpoint and emit events on each message?

[craigomatic]

We added support for streaming chat messages semi-recently, although I'm not sure this is what you are asking for: https://github.com/microsoft/semantic-kernel/blob/main/samples/dotnet/kernel-syntax-examples/Example33_StreamingChat.cs

Can you be a little more specific about the types of events you are looking to consume?

[wayne-o]

So for example in the copilot example, I would want the results as a stream. Even if that meant subscribing to a SignalR stream. That’s ok. Because it makes sense to have some of the data in a single payload. But I would want to see the chat response as soon as it started generating. As opposed to waiting for the whole thing to complete - does that make sense? Thank you :)

…

On Fri, 2 Jun 2023 at 19:37, Craig Presti ***@***.***> wrote: We added support for streaming chat messages semi-recently, although I'm not sure this is what you are asking for: https://github.com/microsoft/semantic-kernel/blob/main/samples/dotnet/kernel-syntax-examples/Example33_StreamingChat.cs Can you be a little more specific about the types of events you are looking to consume? — Reply to this email directly, view it on GitHub <#1298 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABLPDHX5UPJXNEDSRXHRFDXJIXFJANCNFSM6AAAAAAYXBZ4YU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[awharrison-28]

@RogerBarreto has some ideas in the works to make the requested behavior possible. TBD - when we get those changes in, let's make sure to tag this issue.

[wayne-o]

Happy to help out anyway I can Thank you :)

…

On Fri, 2 Jun 2023 at 23:13, Abby Harrison ***@***.***> wrote: @RogerBarreto <https://github.com/RogerBarreto> has some ideas in the works to make the requested behavior possible. TBD - when we get those changes in, let's make sure to tag this issue. — Reply to this email directly, view it on GitHub <#1298 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABLPDDV45RWWEFQJ27P4DLXJJQSFANCNFSM6AAAAAAYXBZ4YU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[MithrilMan]

Since you don't have this yet in Semantic Kernel, let me give my view on this:

I'm working on my own implementation of generative LLM integration here: https://github.com/MithrilMan/AIdentities
since a couple of months (I didn't knew about semantic kernel and when I found it it was already late and anyway wasn't adhering 100% to my vision) and I implemented what @wayne-o is asking in my design: I've a concept of Skills and when a skill is executed, this is the signature of the call
IAsyncEnumerable<Thought> ExecuteAsync(Prompt prompt, SkillExecutionContext context, CancellationToken cancellationToken);
The core of my skill/brain thing is here (readme isn't updated) https://github.com/MithrilMan/AIdentities/tree/25e60f875979a0d595561fb7d0f1f5e82e5ab7a1/src/AIdentities.Shared/Features/CognitiveEngine

It works, but I'm sick of a problem that arises around the use of IAsyncEnumerable: when the API fails in the middle of generating a streamed content, it fails badly because it fails when you start to consume the iterator and .Net doesn't have a nice way to handle such cases, so if you want to implement a retry pattern for example by using polly (and we've seen how this is important with OpenAI that lately had big problems serving properly their API), you can't just wrap the call to the stream method because the exception of course arise when you materialize the enumerator.
Unluckily .Net doesn't allow to call a yield return within a try/catch block, so you have to create all sort of workaround to have a proper retry method.

Also when you stream chunks of text, what would you do if something goes wrong in the API ?
In a simple chat application you can just ask the user to send again the message, but if it's part of a skill (or skfunction as you call it) It's very likely that You'd need to inform the consumer that you have to start over and that it should forget the partial generated text (= higher complexity)

I've yet not figured out how to properly and affectively handle such scenarios, in my case when it fails, fails badly, but this makes me wonder if it's worth to use streaming calls in skills at all.

Of course it's cool to see that my Skill is producing a text as a stream and the user can start reading as soon as it's generating, also what I called AIdentity can generate different kind of "thoughts" that can be streemed too (think of it like a kind of log in some scanrio) but at the same time if it breaks in the middle you'll have big troubles to rollback what you eventually did with the partial generated text, without mentioning minor problems it can give if you try to parse that text on a markdown viewer for example (that's why recently in my chat section I'm building the message on a normal div and then swith to markdown once completed), and I'm starting to think that maybe all this complexity isn't worth the effort.

I'm a bit torn as to whether or not implementing stream at the function level is something I'd want to deal with. Maybe just being able to signal to consumer events like "StartingTextGeneration" / "EndingTextGeneration" could be enough