- 
                Notifications
    You must be signed in to change notification settings 
- Fork 34
Open
Labels
[Feature]New feature to highlight in changelogs.New feature to highlight in changelogs.
Milestone
Description
See #8: A crucial piece of functionality still missing in our SDK is response streaming - a quite complex piece of functionality to get right, especially with the incomplete documentation of how different providers implement it. But we need it, that's for sure.
What we'll need to do is:
- Implement the primitives needed for streaming (likely some kind of stream response class, potentially a new stream method on HttpTransporter).
- Implement actual streaming for the three providers (which can fortunately be centralized in the OpenAI API compatible implementation for now).
Putting here some references for inspiration in how I implemented streaming (still with some limitations) in the AI Services plugin for WordPress:
- https://github.com/felixarntz/ai-services/blob/main/includes/Services/HTTP/Contracts/Stream_Request_Handler.php
- https://github.com/felixarntz/ai-services/blob/main/includes/Services/HTTP/Stream_Response.php
- https://github.com/felixarntz/ai-services/blob/main/includes/Services/Traits/Generative_AI_API_Client_Trait.php#L182
- https://github.com/felixarntz/ai-services/blob/main/includes/Services/Base/OpenAI_Compatible_AI_Text_Generation_Model.php#L118
There's definitely things to improve there which we should do here, but it's a decent starting point to review this and come up with effectively a v2 of that implementation.
jeffpaul
Metadata
Metadata
Assignees
Labels
[Feature]New feature to highlight in changelogs.New feature to highlight in changelogs.