refactor: refactored LLM Providers: Adapting Modular Approach

[thecodacus]

LLM Providers Refactor: Modular Approach

Overview

This PR refactors the LLM providers section, which was previously dispersed across multiple files, into a single modular structure. Each provider is now encapsulated within its own dedicated file, containing all the logic required to support both client-side and server-side functionality. This reorganization aims to enhance maintainability, scalability, and clarity in how providers are managed.

Key Changes

1. Introduction of a Modular Provider Structure

• Change: Each LLM provider now exists as a standalone file under app/lib/modules/llm/providers/.
• Impact: New providers can be added simply by creating a new file, following the established structure.
• Benefits: Enhanced code organization, easier provider additions, and reduced risk of breaking changes.

2. Centralized Provider Management via LLMManager

• Change: Added LLMManager, a singleton responsible for loading, managing, and retrieving LLM providers.
• Impact: All provider-related logic is now managed in one place.
• Benefits: Simplified provider registration and lookup, making it easier to maintain and extend.

3. Base Provider Class

• Change: A BaseProvider class was introduced to standardize provider implementation.
• Impact: All providers extend from BaseProvider, enforcing a consistent contract for API interactions.
• Benefits: Enforces consistent methods, such as getModelInstance and getDynamicModels, which promotes uniformity across providers.

4. New getDynamicModels Support

• Change: Providers can now define the getDynamicModels method to retrieve model information dynamically.
• Impact: Providers can now fetch available models dynamically at runtime.
• Benefits: Supports real-time updates to available models, enhancing flexibility and responsiveness.

5. Removal of Legacy Code

• Change: Deprecated provider logic was removed from scattered files.
• Impact: Reduces technical debt and eliminates duplicate logic.
• Benefits: Cleaner, more maintainable codebase.

Technical Details

File Structure and Organization

app/lib/modules/llm/
  |- base-provider.ts        // Abstract base class for providers
  |- manager.ts              // LLMManager for centralized provider control
  |- providers/              // Directory containing all provider implementations
      |- dummy-provider.ts    // Example provider file
      |- ... (other providers) 

LLMManager Singleton

• Responsibilities:
  • Dynamically loads all providers from the providers/ directory.
  • Registers each provider instance and maintains a lookup map for quick access.
  • Exposes methods to retrieve models, default providers, and registered providers.

Simplified Code Example

// manager.ts
class LLMManager {
  private static _instance: LLMManager;
  private _providers: Map<string, BaseProvider> = new Map();

  static getInstance(): LLMManager {
    if (!LLMManager._instance) {
      LLMManager._instance = new LLMManager();
    }
    return LLMManager._instance;
  }

  registerProvider(provider: BaseProvider) {
    this._providers.set(provider.name, provider);
  }

  getProvider(name: string): BaseProvider | undefined {
    return this._providers.get(name);
  }
}

BaseProvider Class

• Responsibilities:
  • Enforces consistent methods like getModelInstance and getDynamicModels for all providers.
  • Simplifies shared logic for API interactions and model retrieval.

Simplified Code Example

// base-provider.ts
export abstract class BaseProvider {
  abstract name: string;
  abstract getModelInstance(options: { model: string }): any;
  
  // New optional method for dynamic model retrieval
  getDynamicModels?(options: { apiKeys?: Record<string, string>; providerSettings?: any; serverEnv?: Record<string, string> }): Promise<ModelInfo[]> {
    return Promise.resolve([]);
  }

  getProviderBaseUrlAndKey() {
    // Shared logic to retrieve API keys and base URLs
  }
}

Example Provider Implementation: DummyProvider

• File Location: app/lib/modules/llm/providers/dummy-provider.ts
• Responsibilities: Implements a Dummy provider, extending BaseProvider, and includes support for dynamic model fetching.

Simplified Code Example

// dummy-provider.ts
import { BaseProvider } from '~/lib/modules/llm/base-provider';
import type { ModelInfo } from '~/lib/modules/llm/types';
import { createOpenAI } from '@ai-sdk/openai';

export class DummyProvider extends BaseProvider {
  name = 'DummyProvider';
  
  async getDynamicModels(options: { apiKeys?: Record<string, string>; providerSettings?: any; serverEnv?: Record<string, string> }): Promise<ModelInfo[]> {
    const apiKey = options.apiKeys?.['DummyProvider'];
    if (!apiKey) throw new Error('API key is missing');
    
    const response = await fetch('https://dummy.api.com/v1/models', {
      headers: { 'Authorization': `Bearer ${apiKey}` }
    });
    const data = await response.json();
    return data.data.map((model: any) => ({
      name: model.id,
      label: model.id,
      provider: this.name,
      maxTokenAllowed: 8000,
    }));
  }
  
  // To be used by the server
  getModelInstance(options: { 
        model: string, serverEnv: Env;
        apiKeys?: Record<string, string>;
        providerSettings?: Record<string, IProviderSetting>; 
    }): LanguageModelV1 {
    
    const { apiKeys, providerSettings, serverEnv, model } = options;
    
    const { baseUrl, apiKey } = this.getProviderBaseUrlAndKey({
      apiKeys,
      providerSettings,
      serverEnv: serverEnv as any,
      defaultBaseUrlKey: 'DUMMY_API_BASE_URL',
      defaultApiTokenKey: '',
    });
    
    // create a model instance
    return createOpenAI({ baseURL, apiKey })(options.model);
  }
}

System Changes

• Infrastructure: No changes required.
• Integration Points: No changes required.
• Performance: Improved performance due to a more modular and efficient lookup system.

Testing

• Unit Tests: New unit tests have been added for LLMManager, BaseProvider, and individual providers.
• Integration Tests: End-to-end tests validate provider functionality.

Migration Impact

• Breaking Changes: None.
• Required Updates: None for consumers of the system, but provider logic was relocated.
• Migration Steps: None required.
• Backward Compatibility: Fully backward compatible.

Future Improvements

• Additional Provider Support: New providers can now be added easily without major refactoring.
• Dynamic Provider Loading: Consider auto-loading providers from an external configuration file.
• Improved Error Handling: Enhance error reporting for failed provider registration.

[mrsimpson]

That's a great PR! Thanks!

I was just implementing a test-provider and got really annoyed by the current architecture.

I have not tested it yet, but a registry is definitely the way to go!

The only thing I'm a bit nervous about is that we still have a mix of frontend related information (like the model names) and configuration which should be available to the server only (for hiding credentials).
Currently, it's not possible to define both with env variables.
But this is actually not the core of this PR (and this PR will make it easier to add such mechanisms), so I don't see this as a blocker.

[aliasfoxkde]

Just tested it and I like the code and the direction..

Some thoughts though:

• Should we include a template.ts file for adding providers easier (or something) (or will this be taken care of somewhere else)?
• Like it would be nice to keep the average user from needing to modify code and whatnot.
• Why so many levels deep? Consider moving to ./app/modules/providers (normal users will be possibly modifying this)?

It doesn't appear that any of the providers are "dynamic" getting models:

Note: It appears that only the static models are loading.

Though they do seem to be getting loaded in (unless I'm not understanding when this happens)?

And this is kind of weird (blanks at the end of the list):

[thecodacus]

• Should we include a template.ts file for adding providers easier (or something) (or will this be taken care of somewhere else)?

better we add the steps to the docs

Why so many levels deep? Consider moving to ./app/modules/providers (normal users will be possibly modifying this)?

the idea is to modularize everything.
so llm folder is one module, which holds the loading provider code and base providers
and providers folder is where only the dynamic codes will reside. so we have llm->providers

and the modules folder holds parent for all future modules not just llm modules,

Note: It appears that only the static models are loading.

Though they do seem to be getting loaded in (unless I'm not understanding when this happens)?

its only loading the code file not the model dynamic models at the moment. so the provider code files are dynamically loaded at runtime.. the manager looks for files in that folder and looks into the code and checks for a Provider in that file.

so you just add the file and the manager will automatically detects that and loads it.

but the model loading happens after you opens the webpage

[thecodacus]

It doesn't appear that any of the providers are "dynamic" getting models:

I haven't added any dynamic model loading for any provider except ollama,lmstudio, openrouter, and togatherai
these already had dynamic model loading, and I am not trying to add/update model providers right now but only the refactoring