Improve structured outputs and tool calling experience

[nathanfallet]

As we can see on https://platform.openai.com/docs/guides/structured-outputs?context=without_parse (see How to use > Manual schema > Step 2), the payload looks something like this:

{
      "type": "json_schema",
      "json_schema": {
          "name": "math_response",
          "schema": {
              "type": "object",
              "properties": {
                  "steps": {
                      "type": "array",
                      "items": {
                          "type": "object",
                          "properties": {
                              "explanation": {"type": "string"},
                              "output": {"type": "string"}
                          },
                          "required": ["explanation", "output"],
                          "additionalProperties": false
                      }
                  },
                  "final_answer": {"type": "string"}
              },
              "required": ["steps", "final_answer"],
              "additionalProperties": false
          },
          "strict": true
      }

Currently, only additionalProperties is available at the schema level (no type, properties, items, required, ...)

Is this still WIP, or any plan about this? I'm also ok to open a PR to add it if it can help (and no one is already working on it)

[TomerAberbach]

Ah, I see the confusion here. So the idea is that you can use putAdditionalProperty on this object to set all the JSON schema properties.

The name additionalProperties is actually not supposed to correspond to the additionalProperties of JSON schema (the same naming is a coincidence haha). We have additionalProperties on every object for the purpose of, well, adding additional properties to it!

For your use-case you'd do the following:

Schema.builder()
  .putAdditionalProperty("type", JsonValue.from("object"))
  .putAdditionalProperty("properties", JsonValue.from(
    Map.of(
      "steps", Map.of("type", "array", "items", Map.of(
        "type", "object",
        "properties", Map.of(/* etc. */),
        "required", List.of("explanation", "output"),
        "additionalProperties", false
      )),
      "final_answer", Map.of("type", "string")
    )
  ))
  // Other properties...
  .build()

Aside: ResponseFormatJsonSchema.JsonSchema.Schema should probably just be replaced with a Map<String, JsonValue> or even just a JsonValue. That would be less confusing. I'll look into that :)

[nathanfallet]

@TomerAberbach Yes that's actually what I discovered and did until we got a better solution. But having a proper Schema builder would avoid any error with it. Let me know if you want me to work on something as a PR.

[TomerAberbach]

I'm not sure we want to have the entire JSON schema spec encoded in the builders, since it's pretty complicated/large. It might just be easier in the current state (as long as we document it)

@kwhinnery-openai do you have any thoughts? To be clear, the currently generated builders are the result of the OpenAPI schema having an "unknown/arbitrary" type for this field (vs a JSON schema spec describing the JSON schema spec lol)

[Kathy3552]

I'm not sure we want to have the entire JSON schema spec encoded in the builders, since it's pretty complicated/large. It might just be easier in the current state (as long as we document it)

[jjestrel]

Having first class support would be great.

I'm moving some existing code from TypeScript to Kotlin and miss the js sdk.

I would love to have either some more convenient helpers or support that interops with another library that we can use to parse the response like the other language SDKs (zod/pydantic for js/python respectively).

The important thing to remember about the current implementation is specifying the required and additionalProperties fields when type is "object". required is an array that must include the name of all properties in the object, additionalProperties must always be false. to be clear, additionalProperties here is not referring to the builder methods (see: #139 (comment))

I need to add/edit response schemas often so I ended up building a very simple builder based on @TomerAberbach's sample. It's definitely incomplete and could use some QoL but may save some time for anyone else who is waiting for this support.

data class SchemaObjectSpec(
    private val description: String? = null,
    private val properties: MutableMap<String, Any?> = mutableMapOf(),
    private val required: MutableList<String> = mutableListOf(),
) : SchemaSpec {
  fun str(name: String, description: String) {
    prop(name, SchemaStrSpec(description))
  }

  fun obj(name: String, spec: SchemaObjectSpec) {
    prop(name, spec)
  }

  fun arr(
      name: String,
      description: String? = null,
      childSpec: SchemaSpec,
  ) {
    prop(name, SchemaArraySpec(description = description, child = childSpec))
  }

  fun arr(name: String, spec: SchemaArraySpec) {
    prop(name, spec)
  }

  private fun prop(name: String, spec: SchemaSpec) {
    properties[name] = spec.build()
    required.add(name)
  }

  override fun build(): Map<String, Any?> {
    return mapOf(
        "type" to "object",
        "description" to description,
        "properties" to properties,
        "additionalProperties" to JsonValue.from(false),
        "required" to required,
    )
  }
}

interface SchemaSpec {
  fun build(): Map<String, Any?>
}

data class SchemaArraySpec(
    private val description: String? = null,
    private val child: SchemaSpec,
) : SchemaSpec {
  override fun build(): Map<String, Any?> {
    return mapOf(
        "type" to "array",
        "description" to description,
        "items" to child.build(),
    )
  }
}

data class SchemaStrSpec(
    private val description: String? = null,
) : SchemaSpec {
  override fun build(): Map<String, Any?> {
    return mapOf<String, Any?>(
        "type" to "string",
        "description" to description,
    )
  }
}

And you can use it like:

 val schema =
        SchemaObjectSpec()
            .apply {
              str(
                  "someVariable",
                  "a description of a list variable")
              arr(
                  "list_of_strs",
                  description =
                      "A description for an array",
                  childSpec = SchemaStrSpec())
              arr(
                  "list_of_objects",
                  description =
                      "A description for the list of objects",
                  SchemaObjectSpec().apply {
                    str("foo", "description of foo")
                    str("bar", "description of bar")
                  })
            }
            .build()
    val jsonSchema =
        JsonSchema.builder()
            .strict(true)
            .name("name_of_schema")
            .schema(JsonValue.from(schema))
            .build()

    val responseFormat =
        ChatCompletionCreateParams.ResponseFormat.ofJsonSchema(
            ResponseFormatJsonSchema.builder().jsonSchema(jsonSchema).build())

    val params =
        ChatCompletionCreateParams.builder()
            .addMessage(message)
            .model(ChatModel.GPT_4O)
            .responseFormat(responseFormat)
            .build()
    val response = openAi.chat().completions().create(params)

[TomerAberbach]

Yeah, so we ultimately want to provide a Python and TS like experience

It would probably look like passing a Java class, perhaps with some annotations, and then we'd automatically convert it to a JSON schema for you

No ETA yet, but we will definitely do something like this

[jjestrel]

@TomerAberbach Tool calling has the same problem as structured outputs where you cannot specify the schema in FunctionParameters.

Please let me know if you'd prefer I file a new ticket, or if the work would all be bundled together. I suspect the fix for both looks pretty similar.

[TomerAberbach]

Yeah, I would say tooling calling fits under this same issue

To clarify though, you can specify the schema in FunctionParameters, right? It's just not convenient?

[jjestrel]

Yeah, I would say tooling calling fits under this same issue

To clarify though, you can specify the schema in FunctionParameters, right? It's just not convenient?

That's correct. It needs the same additionalProperty workaround you specified here: #139 (comment)

Thanks for all your help triaging these!