[Vertex AI] Invalid JSON payload received. Unknown name "optionalProperties" at 'tools[0].function_declarations[0].parameters': Cannot find field.

[brandonfranke]

Operating System

Windows 11

Environment (if applicable)

Chrome

Firebase SDK Version

11.1.0

Firebase SDK Product(s)

VertexAI

Project Tooling

Next.js app

Detailed Problem Description

When trying to provide optionalProperties to a function as part of the Vertex AI tools, I get this error:

Invalid JSON payload received. Unknown name "optionalProperties" at 'tools[0].function_declarations[0].parameters': Cannot find field.

It is typed correctly as I can see optionalProperties is available to provide to the function declaration. This is an issue as Vertex AI in Firebase treats all parameters as required unless specified as optional but I am unable to actually specify anything as optional.

Steps and code to reproduce issue

1. Create a function declaration that has optionalProperties defined and pass it to the Vertex AI model creation
2. Attempt to create a new chat and send a message.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[hsubox76]

You didn't provide any code but here's a version of it that seems to be working fine. Are you using the Schema.object() method? optionalProperties only works with Schema.object. We should probably add that in the documentation and/or fix the typing so it doesn't look like it's an available property if you are passing a raw object.

async function functionCallingWithSchema() {
  const model = init();
  const chat = model.startChat({
    tools: [
      {
        functionDeclarations: [
          {
            name: "getWeather",
            description: "Gets the weather of a given location",
            parameters: Schema.object({
              properties: {
                location: Schema.string({
                  description: "location of which the weather is returned",
                }),
                horoscope: Schema.string({
                  description: "user's horoscope sign",
                }),
              },
              optionalProperties: ["horoscope"],
            }),
          },
        ],
      },
    ],
  });
  const result = await chat.sendMessage(
    "\"I'm trying to decide whether to go to London or Zurich this weekend. How hot are those cities? How about Singapore? Or maybe Tokyo. I want to go somewhere not that cold but not too hot either. Suggest me."
  );
  result.response.candidates[0].content.parts.push({
    functionCall: { name: "fakeFunction", args: { fakeArg: "fakeValue" } },
  });
  result.response.candidates[0].content.parts.push({ text: "fakeText" });
  console.log(result.response.text());
  console.log(result.response.functionCalls());
  console.log(result.response);
}

[brandonfranke]

Thanks for the reply.

I've copied the exact function declaration you've provided and looks like that works however I am now getting type errors.

Type 'ObjectSchema' is not assignable to type 'ObjectSchemaInterface'. Types of property 'type' are incompatible. Type 'SchemaType' is not assignable to type 'SchemaType.OBJECT'.

Works but with type errors:

{
        name: "goToItem",
        description: "Go to a player, team, league, manager or fixture page",
        parameters: Schema.object({
          type: SchemaType.OBJECT,
          properties: {
            type: Schema.string({
              description: "The type of item to go to",
              enum: ["player", "team", "league", "manager", "fixture"],
            }),
            id: Schema.number({
              description:
                "The id of the player, team, league, manager or fixture to go to",
            }),
            sport: Schema.string({
              description:
                "The sport of the player, team, league, manager or fixture to go to",
              enum: ["football"],
            }),
          },
          optionalProperties: ["sport"],
        }),
      },

No type errors but causes described error:

{
        name: "goToItem",
        description: "Go to a player, team, league, manager or fixture page",
        parameters: {
          type: SchemaType.OBJECT,
          properties: {
            type: Schema.string({
              description: "The type of item to go to",
              enum: ["player", "team", "league", "manager", "fixture"],
            }),
            id: Schema.number({
              description:
                "The id of the player, team, league, manager or fixture to go to",
            }),
            sport: Schema.string({
              description:
                "The sport of the player, team, league, manager or fixture to go to",
              enum: ["football"],
            }),
          },
          optionalProperties: ["sport"],
        },
      }

[hsubox76]

I see that, let me see what I can do to fix it. In the meantime, to unblock yourself, you can temporarily use // @ts-ignore on that line and remove it when we release a fix.