Structured outputs: JSON schema doesn't respect nullable modifier on numbers

[davidjwiner]

Confirm this is a Node library issue and not an underlying OpenAI API issue

• This is an issue with the Node library

Describe the bug

The API doesn't respect the nullable modifier when appended to numbers.

To Reproduce

1. Try to extract a schema containing nullable numbers from a user's input query, for example:

const schema = z.object({
  lowerBound: z.number().nullable(),
  upperBound: z.number().nullable(),
});

2. Even if the query clearly doesn't indicate a lower or upper bound (or indicates that the lower / upper bound should be null), the model will still hallucinate a lower / upper bound.

Code snippets

import OpenAI from "openai";
import { z } from "zod";
import { zodResponseFormat } from "openai/helpers/zod";

const schema = z.object({
  lowerBound: z.number().nullable(),
  upperBound: z.number().nullable(),
});

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
  {
    role: "system",
    content: `You are an assistant that extracts headcount ranges. Follow the provided JSON schema precisely. You should take the user's
      query and extract the headcount range for relevant companies based on the user's query.`,
  },
  {
    role: "user",
    content: `
      I'm looking for companies with at least 1000 employees. There should be no upper bound on the number of employees.
      `,
  },
];

async function runRepro() {
  console.log("--- Schema Definition ---");
  console.log(
    JSON.stringify(
      zodResponseFormat(schema, "nestedData").json_schema,
      null,
      2,
    ),
  );
  console.log("\n--- Running OpenAI Call ---");

  try {
    const completion = await openai.beta.chat.completions.parse({
      model: "gpt-4o",
      messages: messages,
      response_format: zodResponseFormat(schema, "nestedData"),
      temperature: 0,
    });

    const result = completion.choices[0]?.message?.parsed;

    console.log("\n--- OpenAI Response (Parsed) ---");
    console.log(JSON.stringify(result, null, 2));

  } catch (error) {
    console.error("\n--- Error during OpenAI call ---");
    console.error(error);
  }
}

runRepro();


// Return value:
// {
//   "lowerBound": 1000,
//   "upperBound": 1000000
// }

OS

macOS

Node version

Node v21.7.1

Library version

openai v4.93.0

[davidjwiner]

Note: I was able to get it to work correctly by replacing the Zod schema with:

const schema = z.object({
  lowerBound: z.union([z.number(), z.null()]),
  upperBound: z.union([z.number(), z.null()]),
});

[Lucas-Levandoski]

I was investigating this, but it seems to be entirely based on the generated response, system prompt.

On this parsed response it is passing a header specifically for this tool

    return this._client.chat.completions
      .create(body, {
        ...options,
        headers: {
          ...options?.headers,
          'X-Stainless-Helper-Method': 'beta.chat.completions.parse', //this parse seems to be the tool
        },
      })
      ._thenUnwrap((completion) => parseChatCompletion(completion, body));

and inside the parseChatCompletion there is a check if it is using the tool

which is not being used even for the example that did work with z.union, so I am guessing it is more of a fine tunning/system prompt problem to understand the nullable zod schema?

[Lucas-Levandoski]

@RobertCraigie you think this could be a problem on the API side too?

[davidjwiner]

Seem to also run into this with z.boolean().nullable(), which is never null. z.union([z.boolean(), z.null()]) does produce null as expected, though.

[murtuzaalisurti]

@davidjwiner @Lucas-Levandoski

I don't think it's just with number datatype, I ran into the same issue while trying to use z.string().optional().nullable(), and fixed it by making it z.union([z.string(), z.null()]). The z.optional() property is also now not supported by the API, leading to a deprecation warning as shown below. More context here: murtuzaalisurti/better#14

Zod field at `#/definitions/json_diff_response/properties/commentsToAdd/items/properties/previously` uses `.optional()` without `.nullable()` which is not supported by the API. See: https://platform.openai.com/docs/guides/structured-outputs?api-mode=responses#all-fields-must-be-required
This will become an error in a future version of the SDK.