[FEATURE] Structured Outputs with Pydantic

[Shekswess]

Problem Statement

Currently cannot find option to utilize structured outputs with JSON schema or Pydantic Schema, even though most of the models on Bedrock (Nova family, Claude family, etc...) support structured outputs

Proposed Solution

No response

Use Case

This is really useful when working with parsing documents by using VLMs !

Alternatives Solutions

No response

Additional Context

No response

[Shekswess]

Hey, what is the status of this one ?
When we can expect it ?
@theagenticguy , @awsarron

[Unshure]

Hi @Shekswess , I have been having discussions with the team on this feature, and we have come to a consensus on what we want the solution to look like:

Building on the work that @theagenticguy has already done in this pr, we will add a new method to the Agent class called structured_output(output_model: Type[BaseModel], prompt: Optional[str]). This method will not perform an agentic loop, and instead take the current conversation history and system_prompt into account when prompting the model to generate structured output.

class Agent:
  ...
  def structured_output(self, output_model: Type[BaseModel], prompt: Optional[str]) -> BaseModel:
      """Get structured output from the Agent's current context.
      Args:
          output_model: The output model the agent will use when responding.
          prompt: An optional string to help prompt the model when generating the output.
      Returns: the loaded basemodel
      Raises ValidationException: The response format from the model does not match the output_model
      """

Under the hood, a new abstract method will be added to the ModelProvider class also called structured_output(messages: Messages, system_prompt: str, output_model: Type[BaseModel]). When this abstract method is implemented, it will call the underlying model_provider's Structured Output feature:

• Anthropic: https://github.com/anthropics/courses/blob/master/tool_use/03_structured_outputs.ipynb
• Bedrock:
  • Anthropic models: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_SpecificToolChoice.html
  • Other models: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_AnyToolChoice.html
• Ollama: https://ollama.com/blog/structured-outputs
• Llama: https://llama.developer.meta.com/docs/features/structured-output
• OpenAI: https://platform.openai.com/docs/guides/structured-outputs?api-mode=chat
• LiteLLM: https://docs.litellm.ai/docs/completion/json_mode

After getting a response from the model, this method will return the response serialized to the output_model's shape if the response is valid, or throw a ValidationException if the model's response does not match the expected output.

class ModelProvider:
  ...
  @abstractmethod
  def structured_output(self, output_model: Type[BaseModel], messages: Messages, system_prompt: Optional[str]) -> BaseModel:
      """Invokes the underlying model's structured_output feature.
      Args:
          output_model: The output model the agent will use when responding.
          messages: The messages to include when generating the structured output
          system_prompt: The system prompt to use when generating the structured output
      Returns: the loaded basemodel
      Raises ValidationException: The response format from the model does not match the output_model
      """

[Unshure]

@theagenticguy will be working on the implementation for this feature.

[Shekswess]

Awesome !
If you are overrunned with features, now knowing your plan I can help and contribute.
Feel free to ping me 🙌
@Unshure

[zastrowm]

Re-opening to centralize requests/updates on this.

This is being implemented in #60 (feat: add structured output support using Pydantic models), which is currently in review.