iterative streaming #241
iterative streaming #241
Conversation
| for event in stream_messages(model, system_prompt, messages, tool_config): | ||
| if "callback" in event: | ||
| inputs = {**event["callback"], **(kwargs if "delta" in event["callback"] else {})} | ||
| callback_handler(**inputs) |
There was a problem hiding this comment.
@zastrowm had the following comment:
I presume that test_event_loop has enough UT coverage verifying the callbacks of events are as expected? At which point we have a lot of confidence that we maintained backwards compatibility.
This was a good call out. We didn't actually have the best coverage on callbacks. I added a test that actually caught an issue with passing kwargs which we need for backwards compatibility. I personally don't think passing kwargs is useful and so we should revisit when we formally type events (tracking issue).
| inputs = {**event["callback"], **(kwargs if "delta" in event["callback"] else {})} | ||
| callback_handler(**inputs) | ||
| else: | ||
| stop_reason, message, usage, metrics = event["stop"] |
There was a problem hiding this comment.
| if "toolUse" in delta_content: | ||
| if "input" not in state["current_tool_use"]: | ||
| state["current_tool_use"]["input"] = "" | ||
|
|
||
| state["current_tool_use"]["input"] += delta_content["toolUse"]["input"] | ||
| callback_handler(delta=delta_content, current_tool_use=state["current_tool_use"], **kwargs) | ||
| callback_event["callback"] = {"delta": delta_content, "current_tool_use": state["current_tool_use"]} |
There was a problem hiding this comment.
This is a temporary format to maintain backwards compatibility. We are planning on formalizing and strongly typing the construction of our agent events for consistency and ease of use.
| call(event={"contentBlockStop": {}}), | ||
| call(event={"contentBlockStart": {"start": {}}}), | ||
| call(event={"contentBlockDelta": {"delta": {"text": "value"}}}), | ||
| call( |
There was a problem hiding this comment.
There was a comment about duplicate events yielded while streaming. This is an example (813-814) and will be addressed as part of #242.
Description
We are currently working on support for an iterative async stream method on the agent class (#83). As part of this work, we need to yield underlying events of the model stream. This PR thus converts the
stream_messagesfunction to a generator.Related Issues
#83
Type of Change
Testing
hatch fmt --linterhatch fmt --formatterhatch test --allhatch run test-lintChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.