fix(providers): consolidate response_format loading and fix Azure nested schema bug #6880
Conversation
…ted schema bug - Add maybeLoadFromExternalFileWithVars for simple config fields with variable rendering - Add maybeLoadResponseFormatFromExternalFile to handle nested schema file references - Fix bug in Azure responses where nested schema: file://... was not being loaded - Update OpenAI, Azure, and xAI providers to use consolidated helper functions - Add comprehensive unit tests including chained file references and error handling - Document nested schema file references and variable rendering in provider docs Closes #3227
mldangelo
requested review from
a team,
MrFlounder,
swarnap and
typpo
as code owners
![promptfoo-scanner[bot]](https://avatars.githubusercontent.com/in/2172945?s=60&v=4){kind=small}
There was a problem hiding this comment.
👍 All Clear
I reviewed this PR for LLM security vulnerabilities, focusing on the new response format loading functions and their usage across OpenAI, Azure, and XAI providers. The changes consolidate configuration loading logic and add support for nested schema file references. No LLM security issues were identified.
Minimum severity threshold for this scan: 🟡 Medium | Learn more
📝 WalkthroughWalkthroughThe PR introduces three new utility functions in Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing touches
📜 Recent review detailsConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro 📒 Files selected for processing (13)
🧰 Additional context used📓 Path-based instructions (11)**/*.{ts,tsx}📄 CodeRabbit inference engine (AGENTS.md)
Files:
**/*.{ts,tsx,js,jsx}📄 CodeRabbit inference engine (AGENTS.md)
Files:
**/*.{ts,tsx,js,jsx,json,yaml,yml}📄 CodeRabbit inference engine (AGENTS.md)
Files:
src/**/*.{ts,tsx}📄 CodeRabbit inference engine (AGENTS.md)
Files:
site/docs/**/*.md📄 CodeRabbit inference engine (site/AGENTS.md)
Files:
site/**/*.{md,ts,tsx,js,json}📄 CodeRabbit inference engine (site/AGENTS.md)
Files:
test/**/*.test.{ts,tsx}📄 CodeRabbit inference engine (test/AGENTS.md)
Files:
{test/**/*.{ts,tsx},src/app/**/*.{test,spec}.{ts,tsx}}📄 CodeRabbit inference engine (AGENTS.md)
Files:
test/**/*.{test,spec}.{ts,tsx}📄 CodeRabbit inference engine (AGENTS.md)
Files:
src/providers/**/*.ts📄 CodeRabbit inference engine (src/providers/AGENTS.md)
Files:
src/providers/**/*.{ts,tsx}📄 CodeRabbit inference engine (AGENTS.md)
Files:
🧠 Learnings (34)📚 Learning: 2025-12-09T06:08:55.096ZApplied to files:
📚 Learning: 2025-07-18T17:25:38.445ZApplied to files:
📚 Learning: 2025-07-18T17:25:38.444ZApplied to files:
📚 Learning: 2025-12-09T06:08:22.578ZApplied to files:
📚 Learning: 2025-12-09T06:08:22.578ZApplied to files:
📚 Learning: 2025-07-18T17:25:38.444ZApplied to files:
📚 Learning: 2025-07-18T17:25:38.445ZApplied to files:
📚 Learning: 2025-12-09T06:08:22.578ZApplied to files:
📚 Learning: 2025-07-18T17:25:38.444ZApplied to files:
📚 Learning: 2025-12-09T06:08:22.578ZApplied to files:
📚 Learning: 2025-07-18T17:25:38.444ZApplied to files:
📚 Learning: 2025-12-11T07:30:27.904ZApplied to files:
📚 Learning: 2025-12-28T00:26:53.706ZApplied to files:
📚 Learning: 2025-12-09T06:09:06.028ZApplied to files:
📚 Learning: 2025-12-11T07:30:27.904ZApplied to files:
📚 Learning: 2025-12-11T07:30:27.904ZApplied to files:
📚 Learning: 2025-12-11T07:30:27.904ZApplied to files:
📚 Learning: 2025-12-11T07:30:27.904ZApplied to files:
📚 Learning: 2025-12-11T07:30:27.904ZApplied to files:
📚 Learning: 2025-12-11T07:30:27.904ZApplied to files:
📚 Learning: 2025-12-29T16:16:50.330ZApplied to files:
📚 Learning: 2025-12-29T16:16:50.330ZApplied to files:
📚 Learning: 2025-12-29T16:16:50.330ZApplied to files:
📚 Learning: 2025-12-09T06:09:06.028ZApplied to files:
📚 Learning: 2025-12-11T07:30:27.904ZApplied to files:
📚 Learning: 2025-12-09T06:09:06.028ZApplied to files:
📚 Learning: 2025-07-18T17:25:38.444ZApplied to files:
📚 Learning: 2025-12-09T06:09:06.028ZApplied to files:
📚 Learning: 2025-07-18T17:25:38.445ZApplied to files:
📚 Learning: 2025-07-18T17:25:38.445ZApplied to files:
📚 Learning: 2025-12-09T06:09:21.914ZApplied to files:
📚 Learning: 2025-12-09T06:09:06.028ZApplied to files:
📚 Learning: 2025-12-29T16:16:50.330ZApplied to files:
📚 Learning: 2025-12-20T02:05:23.936ZApplied to files:
🧬 Code graph analysis (8)src/providers/openai/chat.ts (2)
src/providers/azure/responses.ts (2)
src/providers/openai/responses.ts (1)
src/providers/openai/util.ts (2)
src/providers/xai/responses.ts (2)
src/providers/azure/chat.ts (2)
src/util/file.ts (1)
test/util/file.test.ts (2)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (25)
🔇 Additional comments (17)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/providers/openai/util.ts (1)
382-478: Remove duplicategpt-realtimeentry or clarify versioning with distinct IDs
OPENAI_REALTIME_MODELShas two entries withid: 'gpt-realtime'. The.find()method in the cost calculation returns only the first match, so the second entry is never used. More critically, the first entry's prices ($32/$64 per 1M tokens) significantly exceed OpenAI's current pricing of $0.50/$16.00 per 1M tokens for gpt-realtime, causing incorrect cost calculations. If the second entry represents an older pricing snapshot, use a distinct versioned ID (e.g.,'gpt-realtime-2024-10-01'); otherwise, remove it and update the first entry with correct current pricing.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (13)
site/docs/providers/azure.mdsite/docs/providers/openai.mdsite/docs/providers/xai.mdsrc/providers/azure/chat.tssrc/providers/azure/responses.tssrc/providers/openai/chat.tssrc/providers/openai/responses.tssrc/providers/openai/util.tssrc/providers/xai/responses.tssrc/util/file.tssrc/util/index.tstest/providers/azure/responses.test.tstest/util/file.test.ts
🧰 Additional context used
📓 Path-based instructions (11)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use TypeScript with strict type checking
Files:
src/util/index.tstest/providers/azure/responses.test.tssrc/providers/openai/chat.tssrc/providers/azure/responses.tssrc/providers/openai/responses.tssrc/providers/openai/util.tssrc/providers/xai/responses.tssrc/providers/azure/chat.tssrc/util/file.tstest/util/file.test.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{ts,tsx,js,jsx}: Follow consistent import order enforced by Biome
Use consistent curly braces for all control statements
Preferconstoverlet; avoidvar
Use object shorthand syntax whenever possible
Useasync/awaitfor asynchronous code
Use consistent error handling with proper type checks
Avoid re-exporting from files; import directly from the source module
Use the logger with object context (auto-sanitized) for logging:logger.debug('[Component] Message', { headers, body, config })
Files:
src/util/index.tstest/providers/azure/responses.test.tssrc/providers/openai/chat.tssrc/providers/azure/responses.tssrc/providers/openai/responses.tssrc/providers/openai/util.tssrc/providers/xai/responses.tssrc/providers/azure/chat.tssrc/util/file.tstest/util/file.test.ts
**/*.{ts,tsx,js,jsx,json,yaml,yml}
📄 CodeRabbit inference engine (AGENTS.md)
Never commit the
.envfile or expose API keys in code or commit messages
Files:
src/util/index.tstest/providers/azure/responses.test.tssrc/providers/openai/chat.tssrc/providers/azure/responses.tssrc/providers/openai/responses.tssrc/providers/openai/util.tssrc/providers/xai/responses.tssrc/providers/azure/chat.tssrc/util/file.tstest/util/file.test.ts
src/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use Drizzle ORM for database operations
Files:
src/util/index.tssrc/providers/openai/chat.tssrc/providers/azure/responses.tssrc/providers/openai/responses.tssrc/providers/openai/util.tssrc/providers/xai/responses.tssrc/providers/azure/chat.tssrc/util/file.ts
site/docs/**/*.md
📄 CodeRabbit inference engine (site/AGENTS.md)
site/docs/**/*.md: Don't modify heading text in documentation (often externally linked)
Avoid embellishment words like 'sophisticated' in documentation
Include front matter with title (under 60 chars), description (150-160 chars), and sidebar_position on all documentation pages
Addtitle="filename.yaml"attribute only to code blocks containing complete, runnable files
Don't add titles to code block fragments (only complete, runnable files)
Use// highlight-next-linedirective to emphasize specific lines in code blocks
Never remove existing highlight directives from code blocks
Use admonition syntax (:::note, :::warning, :::danger) with empty lines around content
Files:
site/docs/providers/xai.mdsite/docs/providers/azure.mdsite/docs/providers/openai.md
site/**/*.{md,ts,tsx,js,json}
📄 CodeRabbit inference engine (site/AGENTS.md)
site/**/*.{md,ts,tsx,js,json}: Use 'eval' not 'evaluation' in commands and documentation
Use 'Promptfoo' when referring to the company or product, 'promptfoo' when referring to the CLI command or in code
Files:
site/docs/providers/xai.mdsite/docs/providers/azure.mdsite/docs/providers/openai.md
test/**/*.test.{ts,tsx}
📄 CodeRabbit inference engine (test/AGENTS.md)
test/**/*.test.{ts,tsx}: Never increase test timeouts - fix the slow test instead
Never use.only()or.skip()in committed code
Always clean up mocks inafterEachusingvi.resetAllMocks()
All tests require explicit imports from vitest:import { afterEach, describe, expect, it, vi } from 'vitest'
Use Vitest's mocking utilities (vi.mock,vi.fn,vi.spyOn) and prefer shallow mocking over deep mocking
Usevi.clearAllMocks()to clear call history, but usemockReset()for full isolation when mocking withmockReturnValue()orvi.hoisted()inbeforeEach
Clean up any test data or mocks after each test
Globals are disabled in vitest configuration. All test utilities must be explicitly imported fromvitest
Files:
test/providers/azure/responses.test.tstest/util/file.test.ts
{test/**/*.{ts,tsx},src/app/**/*.{test,spec}.{ts,tsx}}
📄 CodeRabbit inference engine (AGENTS.md)
{test/**/*.{ts,tsx},src/app/**/*.{test,spec}.{ts,tsx}}: Use Vitest for all tests
Test both success and error cases for all functionality
Files:
test/providers/azure/responses.test.tstest/util/file.test.ts
test/**/*.{test,spec}.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Backend tests in
test/should use Vitest with globals enabled (describe,it,expectavailable without imports)
Files:
test/providers/azure/responses.test.tstest/util/file.test.ts
src/providers/**/*.ts
📄 CodeRabbit inference engine (src/providers/AGENTS.md)
src/providers/**/*.ts: ImplementApiProviderinterface fromsrc/types/providers.tsfor LLM provider integrations
Provider must transform prompts to provider-specific API format and return normalizedProviderResponse
If your provider allocates resources (Python workers, connections, child processes), implement acleanup()method and register withproviderRegistryfor automatic cleanup
Use logger with object context (auto-sanitized) for logging as specified indocs/agents/logging.md
OpenAI-compatible providers should extendOpenAiChatCompletionProviderbase class
Config priority order: Explicit options > Environment variables > Provider defaults
Files:
src/providers/openai/chat.tssrc/providers/azure/responses.tssrc/providers/openai/responses.tssrc/providers/openai/util.tssrc/providers/xai/responses.tssrc/providers/azure/chat.ts
src/providers/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Document provider configurations following examples in existing code
Files:
src/providers/openai/chat.tssrc/providers/azure/responses.tssrc/providers/openai/responses.tssrc/providers/openai/util.tssrc/providers/xai/responses.tssrc/providers/azure/chat.ts
🧠 Learnings (34)
📚 Learning: 2025-12-09T06:08:55.096Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/commands/AGENTS.md:0-0
Timestamp: 2025-12-09T06:08:55.096Z
Learning: Applies to src/commands/**/*.ts : Use `setupEnv()` for environment setup in CLI commands
Applied to files:
src/util/index.ts
📚 Learning: 2025-07-18T17:25:38.445Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.445Z
Learning: Applies to examples/*/promptfooconfig*.yaml : When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features in configuration files
Applied to files:
site/docs/providers/xai.md
📚 Learning: 2025-07-18T17:25:38.444Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.444Z
Learning: Applies to examples/*/promptfooconfig*.yaml : Always include the YAML schema reference at the top of configuration files: '# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json'
Applied to files:
site/docs/providers/xai.mdsite/docs/providers/azure.mdsite/docs/providers/openai.md
📚 Learning: 2025-12-09T06:08:22.578Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: examples/AGENTS.md:0-0
Timestamp: 2025-12-09T06:08:22.578Z
Learning: Applies to examples/**/promptfooconfig.yaml : Each example directory must include a `promptfooconfig.yaml` with schema reference `https://promptfoo.dev/config-schema.json`
Applied to files:
site/docs/providers/xai.mdsite/docs/providers/azure.mdsite/docs/providers/openai.md
📚 Learning: 2025-12-09T06:08:22.578Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: examples/AGENTS.md:0-0
Timestamp: 2025-12-09T06:08:22.578Z
Learning: Applies to examples/**/promptfooconfig.yaml : Use current model identifiers for OpenAI, Anthropic, and Google providers from the documentation in `site/docs/providers/`
Applied to files:
site/docs/providers/xai.mdsite/docs/providers/openai.md
📚 Learning: 2025-07-18T17:25:38.444Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.444Z
Learning: Applies to examples/*/promptfooconfig*.yaml : Include a working promptfooconfig.yaml (or equivalent) file in each example
Applied to files:
site/docs/providers/xai.mdsite/docs/providers/openai.md
📚 Learning: 2025-07-18T17:25:38.445Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.445Z
Learning: Applies to examples/*/promptfooconfig*.yaml : Include a mix of providers when comparing model performance in configuration files
Applied to files:
site/docs/providers/xai.md
📚 Learning: 2025-12-09T06:08:22.578Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: examples/AGENTS.md:0-0
Timestamp: 2025-12-09T06:08:22.578Z
Learning: Applies to examples/**/promptfooconfig.yaml : Use `file://` prefix for external file references in promptfooconfig.yaml
Applied to files:
site/docs/providers/xai.mdsite/docs/providers/azure.mdsite/docs/providers/openai.md
📚 Learning: 2025-07-18T17:25:38.444Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.444Z
Learning: Applies to examples/*/promptfooconfig*.yaml : For OpenAI, prefer models like 'openai:o3-mini' and 'openai:gpt-4o-mini' in configuration files
Applied to files:
site/docs/providers/xai.mdsite/docs/providers/openai.md
📚 Learning: 2025-12-09T06:08:22.578Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: examples/AGENTS.md:0-0
Timestamp: 2025-12-09T06:08:22.578Z
Learning: Applies to examples/**/promptfooconfig.yaml : Use YAML field order in promptfooconfig.yaml: description, env, prompts, providers, defaultTest, scenarios, tests
Applied to files:
site/docs/providers/xai.md
📚 Learning: 2025-07-18T17:25:38.444Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.444Z
Learning: Applies to examples/*/promptfooconfig*.yaml : Follow the specific field order in all configuration files: description, env (optional), prompts, providers, defaultTest (optional), scenarios (optional), tests
Applied to files:
site/docs/providers/xai.md
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/**/*.test.{ts,tsx} : Use Vitest's mocking utilities (`vi.mock`, `vi.fn`, `vi.spyOn`) and prefer shallow mocking over deep mocking
Applied to files:
test/providers/azure/responses.test.tstest/util/file.test.ts
📚 Learning: 2025-12-28T00:26:53.706Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/app/AGENTS.md:0-0
Timestamp: 2025-12-28T00:26:53.706Z
Learning: Applies to src/app/src/**/*.test.{ts,tsx} : Use Vitest with `vi.fn()` for mocks and `vi.mock()` for modules. See 'src/app/src/hooks/usePageMeta.test.ts' for patterns
Applied to files:
test/providers/azure/responses.test.tstest/util/file.test.ts
📚 Learning: 2025-12-09T06:09:06.028Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/providers/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:06.028Z
Learning: Applies to src/providers/test/providers/**/*.ts : Provider tests must NEVER make real API calls - mock all HTTP requests using `vi.mock`
Applied to files:
test/providers/azure/responses.test.tstest/util/file.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/**/*.test.{ts,tsx} : Use `vi.clearAllMocks()` to clear call history, but use `mockReset()` for full isolation when mocking with `mockReturnValue()` or `vi.hoisted()` in `beforeEach`
Applied to files:
test/providers/azure/responses.test.tstest/util/file.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/**/*.test.{ts,tsx} : All tests require explicit imports from vitest: `import { afterEach, describe, expect, it, vi } from 'vitest'`
Applied to files:
test/providers/azure/responses.test.tstest/util/file.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/**/*.test.{ts,tsx} : Always clean up mocks in `afterEach` using `vi.resetAllMocks()`
Applied to files:
test/providers/azure/responses.test.tstest/util/file.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/**/*.test.{ts,tsx} : Clean up any test data or mocks after each test
Applied to files:
test/providers/azure/responses.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/providers/*.test.{ts,tsx} : For provider tests, cover: success case, error cases (4xx, 5xx, rate limits), configuration validation, and token usage tracking
Applied to files:
test/providers/azure/responses.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Applies to test/**/*.test.{ts,tsx} : Globals are disabled in vitest configuration. All test utilities must be explicitly imported from `vitest`
Applied to files:
test/providers/azure/responses.test.tstest/util/file.test.ts
📚 Learning: 2025-12-29T16:16:50.330Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-29T16:16:50.330Z
Learning: Applies to test/**/*.{test,spec}.{ts,tsx} : Backend tests in `test/` should use Vitest with globals enabled (`describe`, `it`, `expect` available without imports)
Applied to files:
test/providers/azure/responses.test.tstest/util/file.test.ts
📚 Learning: 2025-12-29T16:16:50.330Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-29T16:16:50.330Z
Learning: Applies to src/app/**/*.{test,spec}.{ts,tsx} : Frontend tests in `src/app/` should use Vitest with explicit imports
Applied to files:
test/providers/azure/responses.test.tstest/util/file.test.ts
📚 Learning: 2025-12-29T16:16:50.330Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-29T16:16:50.330Z
Learning: Applies to {test/**/*.{ts,tsx},src/app/**/*.{test,spec}.{ts,tsx}} : Use Vitest for all tests
Applied to files:
test/providers/azure/responses.test.tstest/util/file.test.ts
📚 Learning: 2025-12-09T06:09:06.028Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/providers/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:06.028Z
Learning: Applies to src/providers/test/providers/**/*.ts : Test provider success AND error cases, including rate limits, timeouts, and invalid configs
Applied to files:
test/providers/azure/responses.test.tstest/util/file.test.ts
📚 Learning: 2025-12-11T07:30:27.904Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: test/AGENTS.md:0-0
Timestamp: 2025-12-11T07:30:27.904Z
Learning: Mock external dependencies but not the code being tested
Applied to files:
test/providers/azure/responses.test.ts
📚 Learning: 2025-12-09T06:09:06.028Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/providers/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:06.028Z
Learning: Applies to src/providers/site/docs/providers/**/*.md : Provider documentation should be added in `site/docs/providers/`
Applied to files:
site/docs/providers/azure.md
📚 Learning: 2025-07-18T17:25:38.444Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.444Z
Learning: Applies to examples/*/promptfooconfig*.yaml : When referencing external files in configuration, always use the 'file://' prefix
Applied to files:
site/docs/providers/azure.mdsite/docs/providers/openai.md
📚 Learning: 2025-12-09T06:09:06.028Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/providers/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:06.028Z
Learning: Applies to src/providers/**/*.ts : OpenAI-compatible providers should extend `OpenAiChatCompletionProvider` base class
Applied to files:
src/providers/openai/chat.ts
📚 Learning: 2025-07-18T17:25:38.445Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.445Z
Learning: Applies to examples/*/promptfooconfig*.yaml : When creating examples for specific providers, always use the latest available model versions for that provider in configuration files
Applied to files:
site/docs/providers/openai.md
📚 Learning: 2025-07-18T17:25:38.445Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-07-18T17:25:38.445Z
Learning: Applies to examples/*/README.md : When creating examples for specific providers, compare to similar providers where appropriate in the README
Applied to files:
site/docs/providers/openai.md
📚 Learning: 2025-12-09T06:09:21.914Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/server/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:21.914Z
Learning: Applies to src/server/**/*.{ts,tsx} : Wrap responses in { success, data/error } format
Applied to files:
src/providers/xai/responses.ts
📚 Learning: 2025-12-09T06:09:06.028Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/providers/AGENTS.md:0-0
Timestamp: 2025-12-09T06:09:06.028Z
Learning: Applies to src/providers/**/*.ts : Provider must transform prompts to provider-specific API format and return normalized `ProviderResponse`
Applied to files:
src/providers/xai/responses.ts
📚 Learning: 2025-12-29T16:16:50.330Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-29T16:16:50.330Z
Learning: Applies to {test/**/*.{ts,tsx},src/app/**/*.{test,spec}.{ts,tsx}} : Test both success and error cases for all functionality
Applied to files:
test/util/file.test.ts
📚 Learning: 2025-12-20T02:05:23.936Z
Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: src/redteam/AGENTS.md:0-0
Timestamp: 2025-12-20T02:05:23.936Z
Learning: Applies to src/redteam/test/redteam/**/*.{ts,js} : Add tests for new plugins in the test/redteam/ directory following the reference pattern in src/redteam/plugins/pii.ts
Applied to files:
test/util/file.test.ts
🧬 Code graph analysis (8)
src/providers/openai/chat.ts (2)
src/util/file.ts (2)
maybeLoadFromExternalFileWithVars(387-393)maybeLoadResponseFormatFromExternalFile(408-445)src/util/index.ts (2)
maybeLoadFromExternalFileWithVars(9-9)maybeLoadResponseFormatFromExternalFile(10-10)
src/providers/azure/responses.ts (2)
src/util/file.ts (1)
maybeLoadResponseFormatFromExternalFile(408-445)src/util/index.ts (1)
maybeLoadResponseFormatFromExternalFile(10-10)
src/providers/openai/responses.ts (1)
src/util/file.ts (1)
maybeLoadResponseFormatFromExternalFile(408-445)
src/providers/openai/util.ts (2)
src/util/file.ts (1)
maybeLoadFromExternalFileWithVars(387-393)src/util/index.ts (1)
maybeLoadFromExternalFileWithVars(9-9)
src/providers/xai/responses.ts (2)
src/util/file.ts (1)
maybeLoadResponseFormatFromExternalFile(408-445)src/util/index.ts (1)
maybeLoadResponseFormatFromExternalFile(10-10)
src/providers/azure/chat.ts (2)
src/util/file.ts (2)
maybeLoadResponseFormatFromExternalFile(408-445)maybeLoadFromExternalFileWithVars(387-393)src/util/index.ts (2)
maybeLoadResponseFormatFromExternalFile(10-10)maybeLoadFromExternalFileWithVars(9-9)
src/util/file.ts (1)
src/util/render.ts (1)
renderVarsInObject(100-123)
test/util/file.test.ts (2)
src/util/file.ts (2)
maybeLoadFromExternalFileWithVars(387-393)maybeLoadResponseFormatFromExternalFile(408-445)src/util/index.ts (2)
maybeLoadFromExternalFileWithVars(9-9)maybeLoadResponseFormatFromExternalFile(10-10)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (25)
- GitHub Check: Test on Node 22.x and windows-latest (shard 1/3)
- GitHub Check: Test on Node 24.x and ubuntu-latest
- GitHub Check: Test on Node 24.x and windows-latest (shard 3/3)
- GitHub Check: Test on Node 20.x and macOS-latest
- GitHub Check: Test on Node 22.x and windows-latest (shard 3/3)
- GitHub Check: Test on Node 20.x and windows-latest (shard 2/3)
- GitHub Check: Test on Node 24.x and windows-latest (shard 2/3)
- GitHub Check: Test on Node 22.x and ubuntu-latest
- GitHub Check: Test on Node 20.x and windows-latest (shard 3/3)
- GitHub Check: Test on Node 22.x and windows-latest (shard 2/3)
- GitHub Check: Test on Node 24.x and windows-latest (shard 1/3)
- GitHub Check: webui tests
- GitHub Check: Test on Node 20.x and windows-latest (shard 1/3)
- GitHub Check: Test on Node 22.x and macOS-latest
- GitHub Check: Build on Node 22.x
- GitHub Check: Build on Node 24.x
- GitHub Check: Test on Node 20.x and ubuntu-latest
- GitHub Check: Redteam (Staging API)
- GitHub Check: Redteam (Production API)
- GitHub Check: Build on Node 20.x
- GitHub Check: Build Docs
- GitHub Check: Style Check
- GitHub Check: Share Test
- GitHub Check: security-scan
- GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (17)
src/util/file.ts (1)
371-445: New helpers cleanly centralize var rendering and nested response_format loading
maybeLoadFromExternalFileWithVarsandmaybeLoadResponseFormatFromExternalFilecorrectly composerenderVarsInObjectwithmaybeLoadFromExternalFile, and the latter safely:
- Early-returns for
undefined/null.- Handles both top-level
schemaandjson_schema.schema.- Loads nested
file://...schemas without mutating the original object.This matches the documented behavior and should fix the nested-schema bug without regressions.
site/docs/providers/xai.md (1)
435-443: xAI structured-output docs align with new external schema behaviorThe added example and note about nested file references + variable rendering accurately reflect the new
response_formatloading semantics and are consistent with the OpenAI docs reference.site/docs/providers/openai.md (1)
1039-1055: Nested schema and templated path examples correctly describe implementationThe new
response_format.jsonexample with"schema": "file://./schemas/event-schema.json"and the templatedresponse_format: file://./schemas/{{ schema_name }}.jsonsnippet map directly onto the logic inmaybeLoadResponseFormatFromExternalFile, so users will see behavior consistent with the docs.src/providers/openai/util.ts (1)
2-2: validateFunctionCall helper swap preserves behavior and centralizes loadingSwitching to
maybeLoadFromExternalFileWithVars(functions, vars)keeps the previous behavior (var rendering + optional external file load) while reusing the new shared utility. The optional chaining oninterpolatedFunctions?.find(...)still guardsfunctionsbeingundefined, so the change is safe.Also applies to: 640-643
site/docs/providers/azure.md (1)
183-199: Azure Responses API docs correctly describe nested schema and templated pathsThe new nested
schema: "file://./schemas/output-schema.json"example and the templatedresponse_formatpath align withmaybeLoadResponseFormatFromExternalFile, so Azure docs now reflect the same external-schema behavior as OpenAI/XAI.src/util/index.ts (1)
8-15: Exposing new file helpers via the util barrel is consistent and usefulRe-exporting
maybeLoadFromExternalFileWithVarsandmaybeLoadResponseFormatFromExternalFilealongside the existing file utilities keeps provider imports consistent (../../util/index) and matches this file’s backwards‑compatibility purpose.src/providers/openai/chat.ts (1)
15-20: Chat provider now correctly uses shared helpers for functions and response_formatWiring
functionsthroughmaybeLoadFromExternalFileWithVarsandresponse_formatthroughmaybeLoadResponseFormatFromExternalFilekeeps semantics identical to the prior inline render+load logic, while adding nested schema support and centralizing behavior with other providers. As both helpers are synchronous, this fits cleanly intogetOpenAiBody.Also applies to: 259-273
src/providers/openai/responses.ts (1)
4-8: Responses provider correctly delegates response_format loading to nested-schema helperUsing
maybeLoadResponseFormatFromExternalFile(config.response_format, context?.vars)to deriveresponseFormat, then readingschemafromresponseFormat.schema || responseFormat.json_schema?.schema, ensures:
- Top-level and nested
file://schema references are resolved once.- The resulting
text.formatis always built from the loaded schema.
Storingresponse_format: responseFormatback into the returnedconfigkeeps downstream processing consistent.Also applies to: 174-193
src/providers/xai/responses.ts (1)
4-7: LGTM! Clean integration of consolidated response_format loading.The changes successfully integrate the new
maybeLoadResponseFormatFromExternalFilehelper, which handles both variable rendering and nested schema loading in a single call. The updated comment on line 244 correctly notes that schema loading is handled by the helper, eliminating the previous double-loading issue.Also applies to: 232-236, 244-245
test/util/file.test.ts (2)
1262-1306: Excellent test coverage for maybeLoadFromExternalFileWithVars.The test suite comprehensively covers all expected scenarios: variable rendering in file paths and objects, pass-through behavior, and edge cases with undefined/null values.
1308-1520: Comprehensive test coverage for maybeLoadResponseFormatFromExternalFile.The test suite thoroughly validates all critical scenarios including:
- Basic file loading and variable rendering
- Nested schema loading (both
schemaandjson_schema.schemapaths)- Chained file references (lines 1460-1504) - this is the key test for the Azure bug fix where an outer file contains a nested file reference
- Error propagation when nested files are missing
The chained file reference test is particularly valuable as it validates the fix for the Azure double-loading bug mentioned in the PR objectives.
src/providers/azure/chat.ts (1)
10-15: LGTM! Consistent adoption of consolidated loading helpers.The code correctly uses:
maybeLoadResponseFormatFromExternalFilefor response_format (with nested schema support)maybeLoadFromExternalFileWithVarsfor the simpler functions fieldBoth helpers properly pass
context?.varsfor variable rendering, and the comment on line 135 accurately describes the nested schema loading capability.Also applies to: 135-143, 182-182
test/providers/azure/responses.test.ts (3)
6-9: LGTM! Test setup correctly updated for new helpers.The test properly imports and mocks
maybeLoadResponseFormatFromExternalFile. The change fromvi.clearAllMocks()tovi.resetAllMocks()on line 30 is correct per coding guidelines, which specify usingmockReset()orvi.resetAllMocks()for full isolation when usingmockReturnValue()in test setup.Also applies to: 18-24, 30-30
130-130: Excellent validation of the double-loading bug fix.Line 144 explicitly verifies that
maybeLoadResponseFormatFromExternalFileis called only once, which validates the fix for the Azure nested schema double-loading bug mentioned in the PR objectives. This is a critical test case.Also applies to: 140-144
155-157: Test scenarios correctly updated for new loading path.The mock implementations properly simulate:
- Line 155-157: Pass-through behavior for inline schemas
- Line 339-341: Error propagation for missing files
Both scenarios align with the expected behavior of
maybeLoadResponseFormatFromExternalFile.Also applies to: 339-341
src/providers/azure/responses.ts (2)
4-8: LGTM! Proper implementation of consolidated response_format loading.The code correctly uses
maybeLoadResponseFormatFromExternalFilewithcontext?.varsfor variable rendering. The comment on line 142 accurately notes that the schema is already loaded by the helper, and line 151 correctly uses the loaded schema without attempting to reload it. This fix addresses the Azure nested schema double-loading bug mentioned in the PR objectives.Also applies to: 127-131, 142-151
231-238: Improved error handling with detailed error messages.The validation logic now uses
maybeLoadResponseFormatFromExternalFileconsistently, and the error message on lines 234-238 provides helpful context including the specific loading error and guidance for resolution.
Changes SummaryThis PR consolidates the Type: bugfix Components Affected: Azure Chat Provider, Azure Responses Provider, OpenAI Chat Provider, OpenAI Responses Provider, OpenAI Util, xAI Responses Provider, File utilities, Provider documentation Files Changed
Architecture Impact
Risk Areas: Azure Responses Provider behavior change: nested schema file references were silently failing before; now they are properly loaded. This could affect existing configurations that worked around this bug., File loading pattern changes across all three providers (Azure, OpenAI, xAI) - ensures consistent behavior but requires verification that all path resolution and variable rendering scenarios work correctly, Double-loading bug fix in OpenAI and xAI: previously nested schemas were loaded twice; now single load. Performance improvement but could expose edge cases if double-load had side effects Suggestions
Full review in progress... | Powered by diffray |
| ); | ||
|
|
||
| // Build text format configuration | ||
| let textFormat; |
There was a problem hiding this comment.
🟠 HIGH - Missing default textFormat when responseFormat is falsy
Agent: bugs
Category: bug
Description:
When config.response_format is falsy, textFormat remains undefined and is silently omitted from the request body. Azure and OpenAI providers both have an else clause to set textFormat to { format: { type: 'text' } } when responseFormat is undefined.
Suggestion:
Add an else clause after line 259 to initialize textFormat to a default text format when responseFormat is falsy: } else { textFormat = { format: { type: 'text' } }; }
Why this matters: Inconsistent API request format across providers may cause unexpected behavior.
Confidence: 90%
Rule: bug_null_pointer
Review ID: 9851952d-5055-47c1-b5e9-d7128d79d411
Rate it 👍 or 👎 to improve future reviews | Powered by diffray
| maybeLoadResponseFormatFromExternalFile as MockedFunction< | ||
| typeof maybeLoadResponseFormatFromExternalFile | ||
| >; | ||
| const _mockFs = fs as Mocked<typeof fs>; |
There was a problem hiding this comment.
🟡 MEDIUM - Unused variable declaration
Agent: refactoring
Category: quality
Description:
_mockFs is declared but never used in the test file. The underscore prefix indicates intentional dead code that should be removed.
Suggestion:
Remove the unused variable declaration: const _mockFs = fs as Mocked;
Why this matters: Dead code adds confusion and maintenance burden.
Confidence: 85%
Rule: quality_unused_variable
Review ID: 9851952d-5055-47c1-b5e9-d7128d79d411
Rate it 👍 or 👎 to improve future reviews | Powered by diffray
| maybeLoadResponseFormatFromExternalFile(this.config.response_format, {}); | ||
| } catch (error) { | ||
| throw new Error( | ||
| `Failed to load response_format file: ${this.config.response_format}\n` + |
There was a problem hiding this comment.
🟡 MEDIUM - Magic number for timeout without named constant
Agent: quality
Category: quality
Description:
Hard-coded timeout value 600_000 is duplicated from openai/responses.ts without a shared constant.
Suggestion:
Define a shared constant for this timeout value: const DEEP_RESEARCH_TIMEOUT_MS = 600_000;
Why this matters: Magic numbers obscure intent and make changes error-prone.
Confidence: 60%
Rule: qual_magic_numbers_js
Review ID: 9851952d-5055-47c1-b5e9-d7128d79d411
Rate it 👍 or 👎 to improve future reviews | Powered by diffray
| const _mockMaybeLoadFromExternalFile = maybeLoadFromExternalFile as MockedFunction< | ||
| typeof maybeLoadFromExternalFile | ||
| >; |
There was a problem hiding this comment.
🟠 HIGH - Unused variable declaration
Agent: refactoring
Category: bug
Description:
_mockMaybeLoadFromExternalFile is declared with underscore prefix indicating intentionally unused, but variable serves no purpose and adds confusion. Grep confirms no other usages in the file.
Suggestion:
Remove the unused variable declaration on lines 18-20, keeping only mockMaybeLoadResponseFormatFromExternalFile which is used.
Confidence: 90%
Rule: quality_unused_variable
Review ID: 9851952d-5055-47c1-b5e9-d7128d79d411
Rate it 👍 or 👎 to improve future reviews | Powered by diffray
Review Summary
Validated 44 issues: 12 kept, 32 filtered Issues Found: 12💬 See 4 individual line comment(s) for details. 📊 9 unique issue type(s) across 12 location(s) 📋 Full issue list (click to expand)🟠 HIGH - Missing default textFormat when responseFormat is falsyAgent: bugs Category: bug Why this matters: Inconsistent API request format across providers may cause unexpected behavior. File: Description: When config.response_format is falsy, textFormat remains undefined and is silently omitted from the request body. Azure and OpenAI providers both have an else clause to set textFormat to { format: { type: 'text' } } when responseFormat is undefined. Suggestion: Add an else clause after line 259 to initialize textFormat to a default text format when responseFormat is falsy: } else { textFormat = { format: { type: 'text' } }; } Confidence: 90% Rule: 🟠 HIGH - Missing null check on message before accessing propertiesAgent: react Category: bug Why this matters: Prevents runtime TypeError crashes and makes fallback behavior explicit. File: Description: Line 363 assigns message from choice?.message (which can be undefined), but lines 385-386 access message.tool_calls and message.function_call without null check. Suggestion: Add null check: if (message) { const toolCalls = message.tool_calls; ... } or use optional chaining: message?.tool_calls Confidence: 85% Rule: 🟠 HIGH - Async function called without awaitAgent: testing Category: bug File: Description: Test calls maybeLoadToolsFromExternalFile() which is async (declared on line 459 of file.ts), but line 1082 does not await the result. Other tests in same file (lines 1075, 1090, 1098) correctly use await. Suggestion: Add 'await' before the function call: 'await maybeLoadToolsFromExternalFile(tools, vars);' on line 1082 Confidence: 95% Rule: 🟠 HIGH - Unsafe any type for config parameterAgent: typescript Category: quality File: Description: Function maybeLoadConfigFromExternalFile uses Suggestion: Use 'config: unknown' and return type 'unknown', or define a Config interface. Add runtime type guards for safe processing. Confidence: 85% Rule: 🟡 MEDIUM - Environment variable pollution in parsePathOrGlob suite (2 occurrences)Agent: microservices Category: bug Why this matters: Live I/O introduces slowness, nondeterminism, and external failures unrelated to the code. 📍 View all locations
Rule: 🟡 MEDIUM - Duplicate file URL parsing logicAgent: consistency Category: quality Why this matters: Duplicated code increases maintenance burden when fixing bugs or adding features. File: Description: The loadExternalFunction method reimplements file:// URL parsing that already exists in src/util/functions/loadFunction.ts (parseFileUrl). While the implementations differ slightly, they solve the same problem. Suggestion: Consider refactoring to use parseFileUrl from src/util/functions/loadFunction.ts for consistency, or document why different parsing logic is needed here. Confidence: 65% Rule: 🟡 MEDIUM - Magic number for timeout without named constant (2 occurrences)Agent: quality Category: quality Why this matters: Magic numbers obscure intent and make changes error-prone. 📍 View all locations
Rule: 🟡 MEDIUM - Missing test coverage for context parameterAgent: testing Category: quality File: Description: getAzureResponsesBody accepts context with vars (line 91 of responses.ts: Suggestion: Add test case: Confidence: 78% Rule: 🟡 MEDIUM - Unused variable declaration (2 occurrences)Agent: refactoring Category: quality Why this matters: Dead code adds confusion and maintenance burden. 📍 View all locations
Rule: ℹ️ 8 issue(s) outside PR diff (click to expand)
🟠 HIGH - Missing null check on message before accessing propertiesAgent: react Category: bug Why this matters: Prevents runtime TypeError crashes and makes fallback behavior explicit. File: Description: Line 363 assigns message from choice?.message (which can be undefined), but lines 385-386 access message.tool_calls and message.function_call without null check. Suggestion: Add null check: if (message) { const toolCalls = message.tool_calls; ... } or use optional chaining: message?.tool_calls Confidence: 85% Rule: 🟠 HIGH - Async function called without awaitAgent: testing Category: bug File: Description: Test calls maybeLoadToolsFromExternalFile() which is async (declared on line 459 of file.ts), but line 1082 does not await the result. Other tests in same file (lines 1075, 1090, 1098) correctly use await. Suggestion: Add 'await' before the function call: 'await maybeLoadToolsFromExternalFile(tools, vars);' on line 1082 Confidence: 95% Rule: 🟠 HIGH - Unsafe any type for config parameterAgent: typescript Category: quality File: Description: Function maybeLoadConfigFromExternalFile uses Suggestion: Use 'config: unknown' and return type 'unknown', or define a Config interface. Add runtime type guards for safe processing. Confidence: 85% Rule: 🟡 MEDIUM - Environment variable pollution in parsePathOrGlob suite (2 occurrences)Agent: microservices Category: bug Why this matters: Live I/O introduces slowness, nondeterminism, and external failures unrelated to the code. 📍 View all locations
Rule: 🟡 MEDIUM - Duplicate file URL parsing logicAgent: consistency Category: quality Why this matters: Duplicated code increases maintenance burden when fixing bugs or adding features. File: Description: The loadExternalFunction method reimplements file:// URL parsing that already exists in src/util/functions/loadFunction.ts (parseFileUrl). While the implementations differ slightly, they solve the same problem. Suggestion: Consider refactoring to use parseFileUrl from src/util/functions/loadFunction.ts for consistency, or document why different parsing logic is needed here. Confidence: 65% Rule: 🟡 MEDIUM - Magic number for timeout without named constantAgent: quality Category: quality Why this matters: Magic numbers obscure intent and make changes error-prone. File: Description: Hard-coded timeout value 600_000 is used without a named constant. Same value appears in azure/responses.ts. Suggestion: Extract to a shared constant: const LONG_RUNNING_MODEL_TIMEOUT_MS = 600_000; Confidence: 60% Rule: 🟡 MEDIUM - Missing test coverage for context parameterAgent: testing Category: quality File: Description: getAzureResponsesBody accepts context with vars (line 91 of responses.ts: Suggestion: Add test case: Confidence: 78% Rule: Review ID: |
…t constant - Add missing else clause in xAI responses provider to set default text format when no response_format is specified (matches Azure/OpenAI behavior) - Extract LONG_RUNNING_MODEL_TIMEOUT_MS constant to shared.ts - Remove unused test mocks and imports in azure/responses.test.ts - Simplify timeout logic with ternary operators 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Summary
This PR completes the work started in #3642 by consolidating the
renderVarsInObject+maybeLoadFromExternalFilepattern forresponse_formatandfunctionsfields across all providers.Key changes:
maybeLoadFromExternalFileWithVarshelper for simple config fields (e.g.,functions)maybeLoadResponseFormatFromExternalFilehelper that handles nestedschema: file://...referencesBug fixed: Azure Responses API was not loading nested schema file references. For example:
{ "type": "json_schema", "name": "my_schema", "schema": "file://./actual-schema.json" }The nested
schemafile was returned as a string instead of being loaded. OpenAI and xAI already handled this correctly; Azure now matches their behavior.Test plan
examples/openai-responses/promptfooconfig.external-format.yaml(4/4 pass)examples/openai-structured-output/promptfooconfig.responses.yaml(4/4 pass)examples/openai-structured-output/promptfooconfig.chat.yaml(8/8 pass)Closes #3227