Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 23 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,26 @@
# ResilientLLM

A simple but robust LLM integration layer designed to ensure reliable, seamless interactions across multiple APIs by intelligently handling failures and rate limits.
A minimalist but robust LLM integration layer designed to ensure reliable, seamless interactions across multiple LLM providers by intelligently handling failures and rate limits.

---

This library solves challenges in building production-ready AI Agents due to:

- ❌ Unstable network conditions
- ⚠️ Inconsistent error handling
- ⏳ Unpredictable LLM API rate limit errors

### Key Features

- **Token estimation**: You don’t need to calculate LLM tokens, they are estimated for each request
- **Rate limiting**: You don't need to manage the token bucket rate algorithm yourself to follow the rate limits by LLM service providers, it is done for you automatically
- **Retries, backoff, and circuit breaker**: All are handled internally by the `ResilientOperation`.

## Installation

```bash
npm i resilient-llm
```

## Quickstart

Expand All @@ -16,7 +36,7 @@ const llm = new ResilientLLM({
requestsPerMinute: 60, // Limit to 60 requests per minute
llmTokensPerMinute: 90000 // Limit to 90,000 LLM tokens per minute
},
retries: 3, // Number of times to retry if req. fails for reasons possible to fix by retry
retries: 3, // Number of times to retry when req. fails and only if it is possible to fix by retry
backoffFactor: 2 // Increase delay between retries by this factor
});

Expand All @@ -35,32 +55,6 @@ const conversationHistory = [
})();
```

---

### Key Points

- **Rate limiting is automatic**: You don’t need to pass token counts or manage rate limits yourself.
- **Token estimation**: The number of LLM tokens is estimated for each request and enforced.
- **Retries, backoff, and circuit breaker**: All are handled internally by the `ResilientOperation`.

---

### Advanced: With Custom Options

```js
const response = await llm.chat(
[
{ role: 'user', content: 'Summarize the plot of Inception.' }
],
{
maxTokens: 512,
temperature: 0.5,
aiService: 'anthropic', // override default
model: 'claude-3-5-sonnet-20240620'
}
);
```

## Motivation

ResilientLLM is a resilient, unified LLM interface featuring circuit breaker, token bucket rate limiting, caching, and adaptive retry with dynamic backoff support.
Expand All @@ -76,7 +70,7 @@ The final solution was to extract tiny LLM orchestration class out of all my AI
This library solves my challenges in building production-ready AI Agents such as:
- unstable network conditions
- inconsistent error handling
- unpredictable LLM API rate limit errrors
- unpredictable LLM API rate limit errors

This library aims to solve the same challenges for you by providing a resilient layer that intelligently manages failures and rate limits, enabling you (developers) to integrate LLMs confidently and effortlessly at scale.

Expand Down