Skip to content

Feat/local llm #178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Feat/local llm #178

wants to merge 3 commits into from

Conversation

rezawr
Copy link
Contributor

@rezawr rezawr commented May 16, 2025

Feature: Support Custom baseUrl and Local LLM Endpoints

Summary

This change extends the zerox SDK by adding a baseUrl field to the OpenAICredentials interface and model-factory logic. Users can now:

  • Specify custom OpenAI-compatible endpoints (proxy, staging, internal)
  • Connect to self-hosted or local LLM servers (for example, http://localhost:1234/v1)

When baseUrl is omitted, the SDK continues to use the default https://api.openai.com/v1.

Changes

  1. Types

    • File: node-zerox/src/types.ts
      export interface OpenAICredentials {
        apiKey:  string;
        baseUrl: string;    // newly added field
      }
  2. Credential Validation

    • File: node-zerox/src/models/index.ts
      Updated the isOpenAICredentials() type guard to require baseUrl in addition to apiKey.
  3. OpenAI Model Implementation

    • File: node-zerox/src/models/openai.ts
      • Constructor reads credentials.baseUrl (falling back if necessary).
      • All HTTP requests (chat completions, embeddings, etc.) now use this.baseUrl.
  4. Factory Logic

    • File: node-zerox/src/models/index.ts
      • createModel() passes through the new baseUrl property to the model implementation.
  5. Examples and Tests

    • Added unit tests in node-zerox/src/models/openai.test.ts:
      • Verify default base URL when baseUrl is not provided.
      • Verify custom base URL is respected.
    • Updated example in examples/ocr.ts:
      import { zerox } from "zerox";
      
      const result = await zerox({
        filePath: "invoice.png",
        model:    "qwen-2-vl-7b-ocr",
        credentials: {
          apiKey:  "",
          baseUrl: "http://localhost:1234/v1"
        }
      });

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant