Assistant: unify provider enablement and model settings#11490
Open
sharon-wang wants to merge 61 commits intomainfrom
Open
Assistant: unify provider enablement and model settings#11490sharon-wang wants to merge 61 commits intomainfrom
sharon-wang wants to merge 61 commits intomainfrom
Conversation
- replaces positron.assistant.enabledProviders (array of IDs) with positron.assistant.providers (object with UI names with boolean enablement) - shows non-blocking migration prompt to move user's settings from the old setting to the new one - still supports old setting in conjunction with new one for backwards compatibility, merging both settings for enablement even if the user hasn't migrated yet - removes old setting after successful migration - also updates models.custom and models.preference.byProvider to account for new provider UI names - added tests for provider configuration, provider setting migration, and updated modelDefinitions test - adds settings UI enforcement for new provider UI names, to help users avoid misconfiguration or having to know the internal provider IDs
…gureProviders - update tests and other files to use new command name
- use positron.assistant.providers object instead of positron.assistant.enabledProviders array
Key changes:
1. **Track extension-to-vendor mapping from package.json**
- Populate _providerExtensions map when loading vendor contributions
- Enables getExtensionIdentifierForProvider() to work immediately on extension load, not just after runtime registration
- Remove redundant tracking during runtime registration
2. **Add provider configuration to Positron Assistant service**
- Implement getEnabledProviders() in PositronAssistantConfigurationService
- Use getVendors() to read from package.json declarations (available immediately) instead of getLanguageModelProviders() (requires auth)
- Filter out non-copilot vendors from GitHub Copilot extension's BYOK feature to avoid confusion with Positron's own provider implementations with similar names
3. **Expose via Positron API**
- Add positron.ai.getEnabledProviders()
- Provides single source of truth shared between core and extensions
4. **Update extension to use Positron API**
- Replace duplicate logic in providerConfiguration.ts with API calls
- Eliminates code duplication and ensures consistency
5. **Refactor model registration**
- Extract model registration logic to new modelRegistration.ts
- Add listener for provider enable/disable with restart notification
still need to: - determine whether to automatically remove enabledProviders setting after migration - fill out settings description for each individual provider enable setting - perhaps modify the configureProviders command to take a provider and pre-select it in the modal - how to show provider status (experimental, preview, etc.) in the provider modal? - update positron extension tests for provider migration and configuration
…ption - Updated `showConfigurationDialog` to accept an optional `preselectedProviderId` parameter for preselecting a provider in the dialog. - Modified command registration to pass the provider ID when invoking the configuration dialog. - Updated the Positron API to include options for showing the language model configuration modal. - Refactored related components to support the new preselection functionality, including UI adjustments for scrolling into view.
- Replace `models.preference.byProvider` object with individual `models.preference.<settingName>` settings for each provider - Replace `models.custom` object with individual `models.custom.<settingName>` settings for each provider - Implement automatic migration from old to new settings format with user notification
The settings migration code was reading 'models.custom' to detect if the legacy setting exists. However, VS Code's config API returns a value for 'models.custom' when any child setting like 'models.custom.anthropic' is set, even though these are semantically different settings. This caused the migration to repeatedly trigger because it couldn't distinguish between the old format and the new per-provider settings. Renaming to 'models.overrides' creates a clean namespace separation: - Old setting: models.custom (object with provider id keys) - New settings: models.overrides.<providerName> (array per provider) Additional changes: - Add deprecation notices for old settings (models.custom and models.preference.byProvider) that link to the new settings - Change migration behavior so old values always take precedence, ensuring user intent from the original setting is preserved - Update provider migration tests and model definition test
- share test providers between tests - Add providerSettingsMapping.test.ts for real provider validation of settingName - Add edge case tests for empty/undefined migration scenarios - Remove redundant per-provider inspect mocks from migration tests
|
E2E Tests 🚀 |
Member
Author
|
Note The issue this PR addresses is now targeted for 2026.03, so this should be merged after we branch for 2026.02! |
- wait for some models to be loaded before clicking the pick model dropdown - close the model picker dropdown at the end of each attempt so that next attempt can open it again with refreshed models
src/vs/workbench/contrib/chat/browser/chatManagement/chatModelsViewModel.ts
Show resolved
Hide resolved
Co-authored-by: Melissa Barca <5323711+melissa-barca@users.noreply.github.com>
Co-authored-by: Melissa Barca <5323711+melissa-barca@users.noreply.github.com>
Co-authored-by: Melissa Barca <5323711+melissa-barca@users.noreply.github.com> Signed-off-by: sharon <sharon-wang@users.noreply.github.com>
Co-authored-by: Melissa Barca <5323711+melissa-barca@users.noreply.github.com> Signed-off-by: sharon <sharon-wang@users.noreply.github.com>
update tests to work after 1c7917e
Move Anthropic sign-in and settings configuration from the test body to beforeAll hook. Add polling to wait for models to load asynchronously before running assertions. This fixes potential race conditions where the model picker is checked before providers have fully loaded.
a few other nls strings cleanup/rewording
add test to verify posit ai is first provider in modal
Member
Author
|
Confirmed with Jon that the e2e test failure with Anthropic is a known issue to be addressed separately, so I think this is ready for another review pass! (this test passes for me locally) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Main changes
Individual provider enable/disable settings: Replace
positron.assistant.enabledProvidersarray with individualpositron.assistant.provider.<name>.enableboolean settings for each providerPer-provider model preferences: Replace
positron.assistant.models.preference.byProviderobject with individualpositron.assistant.models.preference.<name>settingsPer-provider custom model overrides: Replace
positron.assistant.models.customobject with individualpositron.assistant.models.overrides.<name>settingsChat panel state awareness: Welcome message now reflects enabled providers state
All providers can now be disabled: Previously, Copilot and Anthropic were always shown in the model picker regardless of settings. Now all providers (including Copilot and Anthropic) respect the enable/disable settings and are properly filtered from the model picker when disabled.
Command
positron-assistant.configureModelsrenamed topositron-assistant.configureProviders: better matches what it does!Provider config modal now takes provider id: when a provider id is passed to the
configureProviderscommand, the modal will scroll to the selected provider. This is available via the provider enablement settings by clicking on the "language model provider dialog" link, and also via the gear icon in the Manage models... > Language Models editoropen_provider_modal_to_provider.mp4
Remove unused providers: Removed deprecated/unused language model providers that were never exposed in the UI:
Remove
getSupportedProviders()API method: replaced with dynamic provider registration throughILanguageModelsService. This didn't appear to be used anywhere anyways.Implementation decisions to call out
models.customrenamed tomodels.overrides: Originally I wanted to usemodels.custom.<providerName>for the new per-provider settings, but ran into an issue with VS Code's configuration API: it returns a value formodels.customwhen any child setting likemodels.custom.anthropicis set, even though these are semantically different settings. This caused the migration to repeatedly trigger because it couldn't distinguish between the old object format and the new per-provider settings. Renaming tomodels.overridescreates clean setting namespace separation, though a different name than before.overridesto something else -- let me know if you have suggestions!Settings organized by domain, not by provider: Settings are grouped by their function (
provider.*for enablement,models.*for model config) rather than having all settings for a provider under one prefix. This means a provider's settings are spread across setting namespaces, but provides better semantic grouping and discoverability in the Settings UI. Users can still find all settings for a specific provider by searching for the provider name.Migration preserves old values: When migrating from the old settings format, old values always take precedence. For example, if a user had
"positron.assistant.enabledProviders": ["anthropic-api", "openai-api"], after migration they'll havepositron.assistant.provider.anthropic.enable: trueandpositron.assistant.provider.openAI.enable: true. The old setting is removed after successful migration, and a one-time notification is shown.enabledProviderssettings (since the admin setting can't be removed by migration). There is a "Don't Show Again" button to dismiss the notification though.Settings change summary
Deprecated (with automatic migration):
positron.assistant.enabledProviders→positron.assistant.provider.<name>.enablepositron.assistant.models.preference.byProvider→positron.assistant.models.preference.<name>positron.assistant.models.custom→positron.assistant.models.overrides.<name>Provider setting names: Each provider class defines its
settingNameproperty (e.g.,anthropic,amazonBedrock,openAI). Seeextensions/positron-assistant/src/providers/*/for the full list.Future possible enhancements
Release Notes
New Features
Bug Fixes
QA Notes
@:positron-notebooks
@:assistant
Added/updated Assistant extension integration tests:
providerMigration.test.ts- new tests for migrating legacy settings to new per-provider formatconfig.test.ts- tests for settings validationmodelDefinitions.test.ts- updated for newmodels.overridessetting nameRun assistant extension tests locally with
npm run test-extension -- -l positron-assistant.Additional notes on changes
extensions/positron-assistant/src/providers/*: changes to files in here were to add the settingName to each provider, and remove any providers not in useextensions/positron-assistant/src/test/*: updated existing tests to match the new settings; added new tests for settings migration, provider enablement, whethersettingNameis adequately set for providers (should fail if a new provider is added but doesn't have a corresponding setting in package.json)extensions/positron-assistant/src/config.ts:getEnabledProvidersis now handled via API, and we also pass the selected provider ID to the config dialog if a user clicks on the config dialog link in the settings UIsrc/vs/workbench/contrib/chat/*: various fixes to ensure that the chat UI shows only enabled provider information. I wanted to avoid having to add too many changes here in the upstream code, but we need to check for enabled providers and invalidate the cached provider/model in order for the chat UI to properly reflect the provider enablement state.