Skip to content

fix(core): warn when a preview model is silently substituted - #28828

Open
chelsealong wants to merge 2 commits into
google-gemini:mainfrom
chelsealong:fix/warn-on-silent-model-substitution-28825
Open

fix(core): warn when a preview model is silently substituted#28828
chelsealong wants to merge 2 commits into
google-gemini:mainfrom
chelsealong:fix/warn-on-silent-model-substitution-28825

Conversation

@chelsealong

Copy link
Copy Markdown

Summary

Fixes #28825.

When a user requests a preview model (e.g. gemini-3.1-pro-preview) but their
authenticated account has no preview-model entitlement, Config silently
rewrites the active model to the auto-gemini-2.5 alias with zero
indication to the caller — no error, no warning, nothing in the CLI output.
The substitute model is then chosen by the auto-routing/classifier logic,
which is why different callers can observe different substitutes
(gemini-2.5-pro vs gemini-2.5-flash) for the same requested model,
as described in the issue.

This is a minimal fix for the smallest actionable ask in the issue ("Failing
that, warn on stderr whenever the served model differs from the requested
one."). The larger asks (hard error, --strict-model flag) require a
cross-component refactor across packages/cli argument parsing and multiple
packages/core fallback/routing components — the same scope called out as
effort/medium and requiring further design in the related, closed issue
#26938. This PR keeps things scoped to the specific code path this report
demonstrates: Config's post-auth preview-access check
(packages/core/src/config/config.ts), which silently reset this.model via
setModel(DEFAULT_GEMINI_MODEL_AUTO) with no log of any kind.

Change

In packages/core/src/config/config.ts, when this reset fires, log a
warning (via the existing debugLogger.warn, which surfaces to stderr in
non-interactive mode and to the console message log in interactive mode)
naming the originally-requested model:

[Config] Requested model "gemini-3.1-pro-preview" is not available to the
current account and will be substituted with an automatically selected
model instead.

Test plan

Added a regression test in packages/core/src/config/config.test.ts
(Model Persistence Bug Fix (#19864) suite) that:

  • Constructs a Config with the initial model set to the Gemini 3.1
    preview model.
  • Mocks getCodeAssistServer/retrieveUserQuota so the account's quota
    buckets do not include a preview model (i.e. no entitlement).
  • Calls refreshAuth(AuthType.LOGIN_WITH_GOOGLE) (the personal OAuth flow
    from the issue's repro).
  • Asserts the model is substituted to DEFAULT_GEMINI_MODEL_AUTO (existing
    behavior, unchanged) and that debugLogger.warn was called with a
    message naming the originally-requested model (the new behavior).

Verified the test fails without the fix:

$ git stash push -- packages/core/src/config/config.ts   # revert only the fix
$ npx vitest run src/config/config.test.ts -t "silently substituted"
 × should warn when a requested preview model is silently substituted due to lack of entitlement
   → expected "warn" to be called with arguments: [ StringContaining{…} ]
   Number of calls: 0
$ git stash pop                                            # restore the fix
$ npx vitest run src/config/config.test.ts -t "silently substituted"
 ✓ should warn when a requested preview model is silently substituted due to lack of entitlement

Full suite run after the fix:

$ npx vitest run src/config/config.test.ts
 ✓ src/config/config.test.ts (237 tests) 730ms
 Test Files  1 passed (1)
      Tests  237 passed (237)

$ npx vitest run src/config/
 Test Files  14 passed (14)
      Tests  450 passed (450)

Also ran, with no new errors:

$ npx eslint packages/core/src/config/config.ts packages/core/src/config/config.test.ts --max-warnings 0
(clean)
$ npm run typecheck   # in packages/core
(clean)

AI assistance disclosure

This change was implemented with the assistance of an AI coding agent
(Claude, Anthropic), which located the root cause, wrote the fix, and wrote
and verified the regression test described above.

When a saved/requested model is a preview model (e.g. gemini-3.1-pro-preview)
but the authenticated account has no preview access, Config silently
rewrites the active model to the auto alias with no indication to the
caller. Log a warning naming the requested model so the substitution is
visible instead of silent.

Fixes google-gemini#28825
@chelsealong
chelsealong requested a review from a team as a code owner August 15, 2026 15:03
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue where the system would silently fall back to a default model when a user requested a preview model they were not entitled to use. By introducing a warning message, the system now provides necessary transparency to the user when their requested model is automatically substituted, improving debuggability and user awareness.

Highlights

  • Improved User Feedback: Added a warning log when a requested preview model is silently substituted due to a lack of account entitlement.
  • Regression Testing: Introduced a new test case in the configuration suite to verify that the warning is correctly triggered when model substitution occurs.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@github-actions github-actions Bot added the size/m A medium sized PR label Aug 15, 2026
@github-actions

github-actions Bot commented Aug 15, 2026

Copy link
Copy Markdown

📊 PR Size: size/M

  • Lines changed: 53
  • Additions: +53
  • Deletions: -0
  • Files changed: 2

@gemini-cli gemini-cli Bot added priority/p1 Important and should be addressed in the near term. area/agent Issues related to Core Agent, Tools, Memory, Sub-Agents, Hooks, Agent Quality labels Aug 15, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a warning log when a requested preview model is substituted with a default model due to a lack of account entitlement, and adds a corresponding unit test to verify this behavior. The feedback suggests restoring the debugLogger.warn spy at the end of the new test to prevent potential test leakage, as debugLogger is a shared module-level singleton.

Comment on lines +4335 to +4338
expect(warnSpy).toHaveBeenCalledWith(
expect.stringContaining(PREVIEW_GEMINI_3_1_MODEL),
);
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The spy on debugLogger.warn is not restored at the end of the test. Since debugLogger is a shared module-level singleton, not restoring the spy can leak into other tests in the suite, potentially swallowing warnings or causing unexpected side effects. Please call warnSpy.mockRestore() at the end of the test to clean up the spy, matching the pattern used elsewhere in this file.

Suggested change
expect(warnSpy).toHaveBeenCalledWith(
expect.stringContaining(PREVIEW_GEMINI_3_1_MODEL),
);
});
expect(warnSpy).toHaveBeenCalledWith(
expect.stringContaining(PREVIEW_GEMINI_3_1_MODEL),
);
warnSpy.mockRestore();
});

…n test

Prevents the spy from leaking into other tests in the shared config.test.ts suite, per review feedback.
@chelsealong

Copy link
Copy Markdown
Author

Addressed: restored the debugLogger.warn spy with warnSpy.mockRestore() at the end of the new test in packages/core/src/config/config.test.ts to prevent it leaking into other tests, matching the existing pattern in this file. Full src/config/ suite (450 tests) and lint pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/agent Issues related to Core Agent, Tools, Memory, Sub-Agents, Hooks, Agent Quality priority/p1 Important and should be addressed in the near term. size/m A medium sized PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v0.55.1: --model gemini-3.1-pro-preview silently served by a 2.5-series model under oauth-personal auth

1 participant