OpenAI with search tool - failing save in validator

[jedimonkey]

Just upgraded to 0.2.1 and found that OpenAI is having a bug with using the web search tool.

The error I'm getting is:

8/20/2025, 11:13:03 AM [CONVEX A(agents:executePromptRun)] Uncaught Error: ArgumentValidationError: Value does not match validator.
Path: .messages[0].message
Value: {content: [{args: "<prompt removed>", toolCallId: "ws_68a53d3a37848193a9b1f4b664d3e97603d3ba2133a3a382", toolName: "web_search_preview", type: "tool-call"}, {result: {type: "json", value: {query: "<prompt removed>", status: "completed"}}, toolCallId: "ws_68a53d3a37848193a9b1f4b664d3e97603d3ba2133a3a382", toolName: "web_search_preview", type: "tool-result"}, {providerOptions: {openai: {itemId: "msg_68a53d3c85708193a32374be3fccc95f03d3ba2133a3a382"}}, text: "<Text response>", type: "text"}], role: "assistant"}
Validator: v.union(v.object({content: v.union(v.string(), v.array(v.union(v.object({providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), text: v.string(), type: v.literal("text")}), v.object({image: v.union(v.string(), v.bytes()), mimeType: v.optional(v.string()), providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), type: v.literal("image")}), v.object({data: v.union(v.string(), v.bytes()), filename: v.optional(v.string()), mimeType: v.string(), providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), type: v.literal("file")})))), providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), role: v.literal("user")}), v.object({content: v.union(v.string(), v.array(v.union(v.object({providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), text: v.string(), type: v.literal("text")}), v.object({data: v.union(v.string(), v.bytes()), filename: v.optional(v.string()), mimeType: v.string(), providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), type: v.literal("file")}), v.object({providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), signature: v.optional(v.string()), state: v.optional(v.union(v.literal("streaming"), v.literal("done"))), text: v.string(), type: v.literal("reasoning")}), v.object({data: v.string(), providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), type: v.literal("redacted-reasoning")}), v.object({args: v.any(), providerExecuted: v.optional(v.boolean()), providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), toolCallId: v.string(), toolName: v.string(), type: v.literal("tool-call")})))), providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), role: v.literal("assistant")}), v.object({content: v.array(v.object({args: v.optional(v.any()), experimental_content: v.optional(v.array(v.union(v.object({text: v.string(), type: v.literal("text")}), v.object({data: v.string(), mimeType: v.optional(v.string()), type: v.literal("image")})))), isError: v.optional(v.boolean()), providerExecuted: v.optional(v.boolean()), providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), result: v.any(), toolCallId: v.string(), toolName: v.string(), type: v.literal("tool-result")})), providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), role: v.literal("tool")}), v.object({content: v.string(), providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), role: v.literal("system")}))

Here is the structure a bit better laid out:

{
  "content": [
    {
      "args": "<Prompt>",
      "toolCallId": "ws*68a53749ec888194aac7fa1386ad722a099480cbbec2232f",
      "toolName": "web_search_preview",
      "type": "tool-call"
    },
    {
      "result": {
        "type": "json",
        "value": {
          "query": "<Prompt>",
          "status": "completed"
        }
      },
      "toolCallId": "ws_68a53749ec888194aac7fa1386ad722a099480cbbec2232f",
      "toolName": "web_search_preview",
      "type": "tool-result"
    },
    {
      "providerOptions": {
        "openai": {
          "itemId": "msg_68a5374cec30819496c5bbf1cdc48c61099480cbbec2232f"
        }
      },
      "text": "Response",
      "type": "text"
    }
  ],
  "role": "assistant"
}

I've dug into the code and found that that assistant only expects to have the tool-call, not the tool-result.
https://github.com/get-convex/agent/blob/main/src/validators.ts#L96C1-L107C3
I checked out the code and tried to add that vToolResultPart but the addMessages needs updating to handle that case and I wasn't sure if I was on the right track.

This is using gpt-4.1 with the following config:

await thread.generateText({
  prompt: 'Who is the latest company by market value?',
  tools: {
    web_search_preview: openai.tools.webSearchPreview({
      searchContextSize: 'high',
      userLocation: {
        type: 'approximate',
        city: 'Perth',
        region: 'Western Australia',
        country: 'AU',
        timezone: 'Australia/Perth',
      },
    }),
  },
  toolChoice: {
    type: 'tool',
    toolName: 'web_search_preview',
  },
})

Also tried GPT-5 (on a whim) and after removing the toolChoice I get this error:

8/20/2025, 11:40:55 AM [CONVEX A(agents:executePromptRun)] Uncaught Error: ArgumentValidationError: Value does not match validator.
Path: .messages[0].reasoningDetails[0]
Value: {providerMetadata: {openai: {itemId: "rs_68a543c1b20c8190b487ed7cdd309d760c2d24ffe2fb3ff3", reasoningEncryptedContent: null}}, text: "", type: "reasoning"}
Validator: v.union(v.object({providerOptions: v.optional(v.record(v.string(), v.record(v.string(), v.any()))), signature: v.optional(v.string()), state: v.optional(v.union(v.literal("streaming"), v.literal("done"))), text: v.string(), type: v.literal("reasoning")}), v.object({signature: v.optional(v.string()), text: v.string(), type: v.literal("text")}), v.object({data: v.string(), type: v.literal("redacted")}))

Happy to investigate / test further, just need a steer if adding more pathways to the addMessage is the correct solution.

[jedimonkey]

Just noticed 0.2.2 has been published, error still remains in this version.

[ianmacartney]

Thanks for the report @jedimonkey . It looks like there's two issues:

1. The assistant message needs to allow for tool responses - that was a change from v4 I hadn't seen
2. The types seem to indicate reasoning details should have providerOptions, but it seems to be having providerMetadata.. I'm ma

I've started a PR and it'll have a package you can try out - they should both be fixed in npm i https://pkg.pr.new/get-convex/agent/@convex-dev/agent@126 - if that works I can merge & release it!

[jedimonkey]

Hot damn that was quick! Just in the middle of hooking up perplexity, will switch to this build very shortly.

[jedimonkey]

Something isn't quite right. Convex isn't failing and I've debugged what is coming back as there was 3 messages in convex (1 - my prompt, 2nd - text response, 3rd, tool call error message):
This is what is in the error:

Invalid input for tool web_search_preview: JSON parsing failed: Text: most mentioned marketing agencies VML AKQA Freeman Burson-Marsteller Publicis Groupe Omnicom Group Edelman mc Group Cheil Worldwide.
Error message: Unexpected token 'm', "most menti"... is not valid JSON

The full response from openai is:

 content: [
        {
          type: 'tool-call',
          toolCallId: 'ws_68a57a409d0881979da2b6f52cd99cc60ecd29e4fec598b6',
          toolName: 'web_search_preview',
          input: 'largest independent marketing agency by number of clients worldwide',
          dynamic: true,
          invalid: true,
          error: [Object]
        },
        {
          type: 'tool-result',
          toolCallId: 'ws_68a57a409d0881979da2b6f52cd99cc60ecd29e4fec598b6',
          toolName: 'web_search_preview',
          input: 'largest independent marketing agency by number of clients worldwide',
          output: [Object],
          providerExecuted: true,
          dynamic: true
        },
        {
          type: 'text',
          text: 'Determining the independent marketing agency with the most clients worldwide is challenging due to limited publicly available data. However, **Edelman** stands out as the world\'s largest independent public relations agency, partnering with approximately 2,000 leading companies globally. ([best-marketing-companies.com](https://www.best-marketing-companies.com/who-is-the-best-marketing-company-in-the-world?utm_source=openai))\n\nAnother notable independent agency is **mc Group**, recognized as the largest independent advertising and public relations agency in Germany and Europe. They have collaborated with major brands such as Visa, Vodafone, Gazprom, CNN, and Real Madrid. With over 80 offices worldwide, mc Group is uniquely present in all 28 EU member states and key economic centers globally. ([best-marketing-companies.com](https://www.best-marketing-companies.com/who-is-the-best-marketing-company-in-the-world?utm_source=openai))\n\nWhile specific client numbers for mc Group are not readily available, their extensive global presence and diverse client portfolio suggest a substantial client base.\n\nIn summary, **Edelman** is likely the independent marketing agency with the most clients worldwide, given its extensive partnerships and global reach. ',
          providerMetadata: [Object]
        },
        {
          type: 'source',
          sourceType: 'url',
          id: 'Mn0liWV6CfV0LJml',
          url: 'https://www.best-marketing-companies.com/who-is-the-best-marketing-company-in-the-world?utm_source=openai',
          title: 'Discover the World\'s Largest Marketing Companies'
        },
        {
          type: 'source',
          sourceType: 'url',
          id: 'An6DKR7fCRvJHK1n',
          url: 'https://www.best-marketing-companies.com/who-is-the-best-marketing-company-in-the-world?utm_source=openai',
          title: 'Discover the World\'s Largest Marketing Companies'
        },
        {
          type: 'tool-error',
          toolCallId: 'ws_68a57a409d0881979da2b6f52cd99cc60ecd29e4fec598b6',
          toolName: 'web_search_preview',
          input: 'largest independent marketing agency by number of clients worldwide',
          error: 'Invalid input for tool web_search_preview: JSON parsing failed: Text: largest independent marketing agency by number of clients worldwide.\nError message: Unexpected token \'l\', "largest in"... is not valid JSON',
          dynamic: true
        }
      ],
      finishReason: 'tool-calls',

[jedimonkey]

oh, in that 'broken' tool-result record, it does contain the sources - so it looks like it's working, even though there is an error. In the v4 version it was just a single message with the text and sources.

[ianmacartney]

@jedimonkey so is there a bug here?
Is it incorrectly getting marked as a failed message? Should we mark a message as success if there's a single non-error message in there?

[jedimonkey]

it's weird, as it's effectively the same config from the v4 code to the v5, so feels like there is an implementation issue. I'll throw in a test with just the AI library making the call, to rule out this library being an issue.

[jedimonkey]

ahh, okay managed to replicate it in the AI sdk, and they have released a new update that fixes that. However the data I'm after is lost (annotations). Seeing if there is a way to recover it. It used to appear in sources, but even the AI SDK say that sources is only supported in gemini/perplexity. :(

[jedimonkey]

I've managed to work around the lost annotations. I guess this is a limitation or bug within the ai sdk. My workaround is grabbing the result from generateText, and extract all the step's content that is source entries and use that to patch back to the providerOptions on the response message record. A bit hacky, but I couldn't patch the sources. Maybe the fix is to scan for sources and populate the sources field?

[ianmacartney]

The step.sources should be already stored on the tool response in the messages table - can you confirm if that's the data you're looking for? And if it's supposed to be in the assistant sources instead?

[jedimonkey]

Sorry @ianmacartney - just got back to this aspect and I still can't see them being saved. sources is just an empty array.

[ianmacartney]

Are you getting sources back in the final `steps` / from AI SDK directly? I'm curious if it's not being saved correctly from the Step in mapping.ts - the latest version loosens the schema on title being optional for url sources, maybe that helps..

…

On Mon, Sep 15, 2025 at 8:36 PM Marcus Collier ***@***.***> wrote: *jedimonkey* left a comment (get-convex/agent#125) <#125 (comment)> Sorry @ianmacartney <https://github.com/ianmacartney> - just got back to this aspect and I still can't see them being saved. sources is just an empty array. — Reply to this email directly, view it on GitHub <#125 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACZQW7P77WZTE4B5JHBB3D3S6AU7AVCNFSM6AAAAACEKFBGY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEOJUG4ZTQNBSGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jedimonkey]

Yep, I'm actually looping through the result steps to extract the sources manually and then patching it back to the providerOptions. it's hacky but working around the issue

const patchMissingSources = async (result: Awaited<ReturnType<typeof thread.generateText>>) => {
        const sources = result.steps.flatMap((step) => step.content.filter((c) => c.type === 'source'))
        const responseMessage = result.savedMessages.find((msg) => msg.status === 'pending' && msg.text === '')
        if (responseMessage) {
          await ctx.runMutation(components.agent.messages.updateMessage, {
            messageId: responseMessage._id,
            patch: {
              providerOptions: {
                  sources,
              },
            },
          })
        }
      }