Skip to content

Assistant: unify provider enablement and model settings#11490

Open
sharon-wang wants to merge 61 commits intomainfrom
assistant/providers-setting
Open

Assistant: unify provider enablement and model settings#11490
sharon-wang wants to merge 61 commits intomainfrom
assistant/providers-setting

Conversation

@sharon-wang
Copy link
Member

@sharon-wang sharon-wang commented Jan 22, 2026

Summary

Main changes

  • Individual provider enable/disable settings: Replace positron.assistant.enabledProviders array with individual positron.assistant.provider.<name>.enable boolean settings for each provider

  • Per-provider model preferences: Replace positron.assistant.models.preference.byProvider object with individual positron.assistant.models.preference.<name> settings

  • Per-provider custom model overrides: Replace positron.assistant.models.custom object with individual positron.assistant.models.overrides.<name> settings

  • Chat panel state awareness: Welcome message now reflects enabled providers state

    • When no providers enabled
      image
    • When at least one provider is enabled, but none are authenticated
      image
  • All providers can now be disabled: Previously, Copilot and Anthropic were always shown in the model picker regardless of settings. Now all providers (including Copilot and Anthropic) respect the enable/disable settings and are properly filtered from the model picker when disabled.

  • Command positron-assistant.configureModels renamed to positron-assistant.configureProviders: better matches what it does!

  • Provider config modal now takes provider id: when a provider id is passed to the configureProviders command, the modal will scroll to the selected provider. This is available via the provider enablement settings by clicking on the "language model provider dialog" link, and also via the gear icon in the Manage models... > Language Models editor

    open_provider_modal_to_provider.mp4
  • Remove unused providers: Removed deprecated/unused language model providers that were never exposed in the UI:

    • Chat providers: Vertex AI, Mistral, Ollama, OpenRouter -- these providers may be compatible with the Custom Provider provider
    • Completion providers: DeepSeek, Ollama and related completion code that is unused
  • Remove getSupportedProviders() API method: replaced with dynamic provider registration through ILanguageModelsService. This didn't appear to be used anywhere anyways.

Implementation decisions to call out

  • models.custom renamed to models.overrides: Originally I wanted to use models.custom.<providerName> for the new per-provider settings, but ran into an issue with VS Code's configuration API: it returns a value for models.custom when any child setting like models.custom.anthropic is set, even though these are semantically different settings. This caused the migration to repeatedly trigger because it couldn't distinguish between the old object format and the new per-provider settings. Renaming to models.overrides creates clean setting namespace separation, though a different name than before.

    • Happy to switch from overrides to something else -- let me know if you have suggestions!
  • Settings organized by domain, not by provider: Settings are grouped by their function (provider.* for enablement, models.* for model config) rather than having all settings for a provider under one prefix. This means a provider's settings are spread across setting namespaces, but provides better semantic grouping and discoverability in the Settings UI. Users can still find all settings for a specific provider by searching for the provider name.

  • Migration preserves old values: When migrating from the old settings format, old values always take precedence. For example, if a user had "positron.assistant.enabledProviders": ["anthropic-api", "openai-api"], after migration they'll have positron.assistant.provider.anthropic.enable: true and positron.assistant.provider.openAI.enable: true. The old setting is removed after successful migration, and a one-time notification is shown.

    • Note: I'm a bit worried that the migration notification will show repeatedly for Workbench users who have admin-configured enabledProviders settings (since the admin setting can't be removed by migration). There is a "Don't Show Again" button to dismiss the notification though.

Settings change summary

Deprecated (with automatic migration):

  • positron.assistant.enabledProviderspositron.assistant.provider.<name>.enable
    • image
    • image
  • positron.assistant.models.preference.byProviderpositron.assistant.models.preference.<name>
    • image
    • image
    • image
  • positron.assistant.models.custompositron.assistant.models.overrides.<name>
    • image
    • image
    • image

Provider setting names: Each provider class defines its settingName property (e.g., anthropic, amazonBedrock, openAI). See extensions/positron-assistant/src/providers/*/ for the full list.

Future possible enhancements

  • Auto-enable provider when clicking its settings gear icon (currently requires enabling first, then configuring)
  • Display provider maturity status (experimental, preview, GA) in configuration modal

Release Notes

New Features

Bug Fixes

  • Assistant: ensure chat panel state reflects enabled providers

QA Notes

@:positron-notebooks
@:assistant

  • updated e2e test fixtures and tests with updated command and settings names

Added/updated Assistant extension integration tests:

  • providerMigration.test.ts - new tests for migrating legacy settings to new per-provider format
  • config.test.ts - tests for settings validation
  • modelDefinitions.test.ts - updated for new models.overrides setting name

Run assistant extension tests locally with npm run test-extension -- -l positron-assistant.


Additional notes on changes

  • extensions/positron-assistant/src/providers/*: changes to files in here were to add the settingName to each provider, and remove any providers not in use
  • extensions/positron-assistant/src/test/*: updated existing tests to match the new settings; added new tests for settings migration, provider enablement, whether settingName is adequately set for providers (should fail if a new provider is added but doesn't have a corresponding setting in package.json)
  • extensions/positron-assistant/src/config.ts: getEnabledProviders is now handled via API, and we also pass the selected provider ID to the config dialog if a user clicks on the config dialog link in the settings UI
  • src/vs/workbench/contrib/chat/*: various fixes to ensure that the chat UI shows only enabled provider information. I wanted to avoid having to add too many changes here in the upstream code, but we need to check for enabled providers and invalidate the cached provider/model in order for the chat UI to properly reflect the provider enablement state.

- replaces positron.assistant.enabledProviders (array of IDs) with positron.assistant.providers (object with UI names with boolean enablement)
- shows non-blocking migration prompt to move user's settings from the old setting to the new one
- still supports old setting in conjunction with new one for backwards compatibility, merging both settings for enablement even if the user hasn't migrated yet
- removes old setting after successful migration
- also updates models.custom and models.preference.byProvider to account for new provider UI names
- added tests for provider configuration, provider setting migration, and updated modelDefinitions test
- adds settings UI enforcement for new provider UI names, to help users avoid misconfiguration or having to know the internal provider IDs
…gureProviders

- update tests and other files to use new command name
- use positron.assistant.providers object instead of positron.assistant.enabledProviders array
Key changes:

1. **Track extension-to-vendor mapping from package.json**
   - Populate _providerExtensions map when loading vendor contributions
   - Enables getExtensionIdentifierForProvider() to work immediately on extension load, not just after runtime registration
   - Remove redundant tracking during runtime registration

2. **Add provider configuration to Positron Assistant service**
    - Implement getEnabledProviders() in PositronAssistantConfigurationService
    - Use getVendors() to read from package.json declarations (available immediately) instead of getLanguageModelProviders() (requires auth)
    - Filter out non-copilot vendors from GitHub Copilot extension's BYOK feature to avoid confusion with Positron's own provider implementations with similar names

3. **Expose via Positron API**
   - Add positron.ai.getEnabledProviders()
   - Provides single source of truth shared between core and extensions

4. **Update extension to use Positron API**
   - Replace duplicate logic in providerConfiguration.ts with API calls
   - Eliminates code duplication and ensures consistency

5. **Refactor model registration**
   - Extract model registration logic to new modelRegistration.ts
   - Add listener for provider enable/disable with restart notification
still need to:
- determine whether to automatically remove enabledProviders setting after migration
- fill out settings description for each individual provider enable setting
- perhaps modify the configureProviders command to take a provider and pre-select it
in the modal
- how to show provider status (experimental, preview, etc.) in the provider modal?
- update positron extension tests for provider migration and configuration
…ption

- Updated `showConfigurationDialog` to accept an optional `preselectedProviderId` parameter for preselecting a provider in the dialog.
- Modified command registration to pass the provider ID when invoking the configuration dialog.

- Updated the Positron API to include options for showing the language model configuration modal.
- Refactored related components to support the new preselection functionality, including UI adjustments for scrolling into view.
- Replace `models.preference.byProvider` object with individual `models.preference.<settingName>` settings for each provider
- Replace `models.custom` object with individual `models.custom.<settingName>` settings for each provider
- Implement automatic migration from old to new settings format with user notification
The settings migration code was reading 'models.custom' to detect if the legacy setting exists. However, VS Code's config API returns a value for 'models.custom' when any child setting like 'models.custom.anthropic' is set, even though these are semantically different settings. This caused the migration to repeatedly trigger because it couldn't distinguish between the old format and the new per-provider settings.

Renaming to 'models.overrides' creates a clean namespace separation:
- Old setting: models.custom (object with provider id keys)
- New settings: models.overrides.<providerName> (array per provider)

Additional changes:
- Add deprecation notices for old settings (models.custom and models.preference.byProvider) that link to the new settings
- Change migration behavior so old values always take precedence, ensuring user intent from the original setting is preserved
- Update provider migration tests and model definition test
- share test providers between tests
- Add providerSettingsMapping.test.ts for real provider validation of settingName
- Add edge case tests for empty/undefined migration scenarios
- Remove redundant per-provider inspect mocks from migration tests
@github-actions
Copy link

github-actions bot commented Jan 22, 2026

E2E Tests 🚀
This PR will run tests tagged with: @:critical @:positron-notebooks @:assistant

readme  valid tags

@sharon-wang sharon-wang marked this pull request as ready for review January 23, 2026 22:25
@sharon-wang
Copy link
Member Author

Note

The issue this PR addresses is now targeted for 2026.03, so this should be merged after we branch for 2026.02!

- wait for some models to be loaded before clicking the pick model dropdown
- close the model picker dropdown at the end of each attempt so that next attempt can open it again with refreshed models
sharon-wang and others added 10 commits February 4, 2026 17:25
Co-authored-by: Melissa Barca <5323711+melissa-barca@users.noreply.github.com>
Co-authored-by: Melissa Barca <5323711+melissa-barca@users.noreply.github.com>
Signed-off-by: sharon <sharon-wang@users.noreply.github.com>
Co-authored-by: Melissa Barca <5323711+melissa-barca@users.noreply.github.com>
Signed-off-by: sharon <sharon-wang@users.noreply.github.com>
Move Anthropic sign-in and settings configuration from the test body to beforeAll hook. Add polling to wait for models to load asynchronously before running assertions. This fixes potential race conditions where the model picker is checked before providers have fully loaded.
a few other nls strings cleanup/rewording
add test to verify posit ai is first provider in modal
@sharon-wang
Copy link
Member Author

Confirmed with Jon that the e2e test failure with Anthropic is a known issue to be addressed separately, so I think this is ready for another review pass!

1) [***-electron] › test/***/tests/positron-assistant/positron-assistant.test.ts:344:6 › Positron Assistant Model Picker Default Indicator - Multiple Providers › Verify default model indicators and ordering for multiple providers @:win @:assistant @:web 

    Error: Expected Anthropic models to be present

    expect(received).toBe(expected) // Object.is equality

    Expected: true
    Received: false

    Call Log:
    - Timeout 30000ms exceeded while waiting on the predicate

      307 | 				expect(hasAnthropic, 'Expected Anthropic models to be present').toBe(true);
      308 | 			}
    > 309 | 		}).toPass({ timeout: 30000 });
          | 		   ^
      310 | 	});
      311 |
      312 | 	test.afterAll('Sign out of providers and clean up', async function ({ app, settings }) {
        at /__w/positron/positron/test/***/tests/positron-assistant/positron-assistant.test.ts:309:6

(this test passes for me locally)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants