Why not use strict mode in the test code `validator.ts`

[ryoppippi]

https://platform.openai.com/docs/guides/function-calling/function-calling-with-structured-outputs?api-mode=responses&strict-mode=enabled#strict-mode

in the test code, the strict mode is not enabled. I think we need to enable it and prove that the OpenAI’s claim is false!

However, when I add srict=true here

openapi/test/benchmark/validate.ts

Lines 12 to 14 in d08bb86

	config: {
	reference: process.argv.includes("--reference"),
	},

It goes an error like this

Error: Failed to convert the JSON schema to the ChatGPT schema.
    at /Users/ryoppippi/ghq/github.com/samchon/openapi/bin/test/utils/ChatGptFunctionCaller.js:978:19
    at Generator.next (<anonymous>)
    at /Users/ryoppippi/ghq/github.com/samchon/openapi/bin/test/utils/ChatGptFunctionCaller.js:41:71
    at new Promise (<anonymous>)
    at __awaiter (/Users/ryoppippi/ghq/github.com/samchon/openapi/bin/test/utils/ChatGptFunctionCalle
r.js:37:12)
    at step (/Users/ryoppippi/ghq/github.com/samchon/openapi/bin/test/utils/ChatGptFunctionCaller.js:
84:47)
    at /Users/ryoppippi/ghq/github.com/samchon/openapi/bin/test/utils/ChatGptFunctionCaller.js:71:28
    at Generator.next (<anonymous>)
    at /Users/ryoppippi/ghq/github.com/samchon/openapi/bin/test/utils/ChatGptFunctionCaller.js:41:71
    at new Promise (<anonymous>)

According to this error message, I cannot figure out what is the causes of the error

[ryoppippi]

Related to this
#158

[ryoppippi]

Also, when I debug the test function all the validation errors I got is that the argument is undefine.

~/g/g/s/openapi master• 5.2s ❱ pnpm node bin/test/benchmark/validate.js
{ path: '$input', expected: 'IShoppingSale.ICreate', value: undefined }
{ path: '$input', expected: 'IShoppingSale.ICreate', value: undefined }
{ path: '$input', expected: 'IShoppingSale.ICreate', value: undefined }
{ path: '$input', expected: 'IShoppingSale.ICreate', value: undefined }
{ path: '$input', expected: 'IShoppingSale.ICreate', value: undefined }
{ path: '$input', expected: 'IShoppingSale.ICreate', value: undefined }
[
  1, 3, 1, 1, 3,
  1, 1, 1, 1, 1
]

(I added a console.log in the retry loop)

This is a little bit weird.
So in this experiment, there are only two situations

• the input argument is valid
• there are no input argument

[samchon]

The reasons why not to using strict mode, and why you have failed to convert to the chatgpt schema are:

• does not allow optional property
• does not allow Record<string, T> type
• too much time spending
• type error can be fixed by validation strategy

cc @AcrylicShrimp

[ryoppippi]

When strict mode is used, it is assumed that validation is performed within the openAI API server.
If the data is of a type that allows strict mode, wouldn't it be faster to use strict mode than to exchange data with the LLM over and over again?
If the conversion fails, the conventional method seems to be fine.

[samchon]

Also, there have been many other type error cases in development of shopping chatbot, or our team's connector services, so don't need to deep dive only into the IShoppingSale.ICreate type.

If you add a console.log() statement to the https://github.com/wrtnlabs/agentica/blob/main/packages/core/src/chatgpt/ChatGptCallFunctionAgent.ts file and test shopping chatbot with https://github.com/wrtnlabs/agentica/blob/main/packages/chat/src/examples/shopping/script.md prompts, you can see how OpenAI failed arguments composition a lot in various ways.

Not filling arguments at all, it is just the tip of the iceberg.

[samchon]

I already tested the strict mode case, but it also occurs the type error case. It would be interesting to reflect this when creating benchmark score charts for tool calling of overall models.

[samchon]

I also initially trusted OpenAI's documentation on strict mode and developed by bypassing optional and dictionary types as much as possible, but I was disappointed when I realized that it was all useless. @AcrylicShrimp suggested validation feedback as an alternative, so let's gather these know-hows and create a proper reference material or benchmark chart to use in a paper after the April seminar.

[ryoppippi]

That would be the case for complex moulds.
But what I am trying to convey is that I hypothesised that there might be something wrong or something to be improved in the benchmark that the accuracy of existing function calling is 0%, and in fact the hypothesis was true to some extent.

One of my current tasks is to investigate how reliable the figures on which Agentica is based are in order to demonstrate its usefulness. This is not an attack, but a follow-up test to ensure reproducibility.

[samchon]

If it's something you need to do, such as revising a technical article or readme document that you're currently writing, I'll prepare the datasets and benchmark programs next week. Please let me know the priority of ryoppippi.