Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added docs/media/plugins/circuit-breaker-states.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
270 changes: 270 additions & 0 deletions plugins/circuitbreaker/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,270 @@
# Bifrost Circuit Breaker Plugin

The Circuit Breaker plugin for Bifrost provides automatic failure detection and recovery for AI provider requests. It monitors request failures and slow calls, automatically opening the circuit when thresholds are exceeded to prevent cascading failures.

## Quick Start

### Download the Plugin

```bash
go get github.com/maximhq/bifrost/plugins/circuitbreaker
```

### Basic Usage

```go
package main

import (
"context"
"time"
bifrost "github.com/maximhq/bifrost/core"
"github.com/maximhq/bifrost/core/schemas"
circuitbreaker "github.com/maximhq/bifrost/plugins/circuitbreaker"
)

func main() {
// Create plugin with default configuration
circuitbreakerPlugin, err := circuitbreaker.NewCircuitBreakerPlugin(circuitbreaker.CircuitBreakerConfig{
FailureRateThreshold: 0.5, // 50% failure rate threshold
SlowCallRateThreshold: 0.5, // 50% slow call rate threshold
SlowCallDurationThreshold: 5 * time.Second,
MinimumNumberOfCalls: 10,
SlidingWindowType: circuitbreaker.CountBased, // Track last N calls
SlidingWindowSize: 100, // Track last 100 calls
PermittedNumberOfCallsInHalfOpenState: 5,
MaxWaitDurationInHalfOpenState: 60 * time.Second,
})
if err != nil {
panic(err)
}

// Initialize Bifrost with the plugin
client, err := bifrost.Init(schemas.BifrostConfig{
Account: &yourAccount,
Plugins: []schemas.Plugin{circuitbreakerPlugin},
})
if err != nil {
panic(err)
}
defer client.Cleanup()

// Circuit breaker will automatically protect your requests
response, err := client.ChatCompletionRequest(context.Background(), &schemas.BifrostRequest{
Provider: schemas.OpenAI,
Model: "gpt-4",
Input: schemas.RequestInput{
ChatCompletionInput: &[]schemas.BifrostMessage{
{
Role: schemas.ModelChatMessageRoleUser,
Content: schemas.MessageContent{
ContentStr: bifrost.Ptr("Hello!"),
},
},
},
},
})
}
```

### State Diagram of Circuit Breaker
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick (assertive)

Your markdown spacing is giving me commitment issues!

The markdown formatting needs some TLC. Add blank lines around headings and lists for better readability.

Example fixes:

 ### State Diagram of Circuit Breaker
+
 ![Circuit Breaker States](../../docs/media/plugins/circuit-breaker-states.png)
 #### Count-Based Sliding Window
+
 - **Type**: `"count-based"`
 ### CLOSED (Normal Operation)
+
 - Requests are sent to providers normally

Also applies to: 98-106, 146-158

🧰 Tools
🪛 markdownlint-cli2 (0.17.2)

69-69: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)

🤖 Prompt for AI Agents
In plugins/circuitbreaker/README.md at line 69 and also lines 98-106 and
146-158, the markdown headings and lists lack proper blank lines around them,
which affects readability. Add blank lines before and after each heading and
list to improve markdown formatting and ensure consistent spacing for better
visual structure.

![Circuit Breaker States](../../docs/media/plugins/circuit-breaker-states.png)

## Configuration

### CircuitBreakerConfig

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `FailureRateThreshold` | `float64` | `0.5` | Failure rate threshold (0.0 to 1.0) |
| `SlowCallRateThreshold` | `float64` | `0.5` | Slow call rate threshold (0.0 to 1.0) |
| `SlowCallDurationThreshold` | `time.Duration` | `5s` | Duration threshold for slow calls |
| `MinimumNumberOfCalls` | `int` | `10` | Minimum calls before evaluation |
| `SlidingWindowType` | `string` | `"count-based"` | `"count-based"` or `"time-based"` |
| `SlidingWindowSize` | `int` | `100` | Size of sliding window (calls for count-based, seconds for time-based) |
| `PermittedNumberOfCallsInHalfOpenState` | `int` | `5` | Calls allowed in half-open state |
| `MaxWaitDurationInHalfOpenState` | `time.Duration` | `60s` | Wait time before half-open transition |
| `Logger` | `schemas.Logger` | `bifrost.NewDefaultLogger(schemas.LogLevelInfo)` | Logger for circuit breaker operations |

### Sliding Window Types

The circuit breaker supports two types of sliding windows for collecting metrics:

#### Count-Based Sliding Window
- **Type**: `"count-based"`
- **Size**: Number of most recent calls to track
- **Behavior**: Maintains a fixed-size circular buffer of the last N calls
- **Use Case**: When you want to evaluate based on a specific number of recent requests
- **Example**: Track the last 100 calls to evaluate failure rates

#### Time-Based Sliding Window
- **Type**: `"time-based"`
- **Size**: Duration in seconds to look back
- **Behavior**: Maintains all calls within the specified time window
- **Use Case**: When you want to evaluate based on a time period
- **Example**: Track all calls in the last 5 minutes to evaluate failure rates

### Configuration Examples

#### Count-Based Sliding Window (Default)

```go
config := circuitbreaker.CircuitBreakerConfig{
FailureRateThreshold: 0.3, // 30% failure rate threshold
SlowCallRateThreshold: 0.4, // 40% slow call rate threshold
SlowCallDurationThreshold: 10 * time.Second,
MinimumNumberOfCalls: 20,
SlidingWindowType: circuitbreaker.CountBased, // Track last N calls
SlidingWindowSize: 200, // Track last 200 calls
PermittedNumberOfCallsInHalfOpenState: 3,
MaxWaitDurationInHalfOpenState: 30 * time.Second,
}
```

#### Time-Based Sliding Window

```go
config := circuitbreaker.CircuitBreakerConfig{
FailureRateThreshold: 0.3, // 30% failure rate threshold
SlowCallRateThreshold: 0.4, // 40% slow call rate threshold
SlowCallDurationThreshold: 10 * time.Second,
MinimumNumberOfCalls: 20,
SlidingWindowType: circuitbreaker.TimeBased, // Track calls in time window
SlidingWindowSize: 300, // Track calls in last 300 seconds (5 minutes)
PermittedNumberOfCallsInHalfOpenState: 3,
MaxWaitDurationInHalfOpenState: 30 * time.Second,
}
```

### Logging Configuration

The circuit breaker plugin includes comprehensive logging to help you monitor its behavior. By default, it uses Bifrost's default logger with `Info` level logging. You can customize the logger by providing your own implementation:

```go
// Use custom logger
customLogger := yourCustomLoggerImplementation
config := circuitbreaker.CircuitBreakerConfig{
FailureRateThreshold: 0.3,
// ... other config options
Logger: customLogger, // Use your custom logger
}

// Or use Bifrost's default logger with different log level
config := circuitbreaker.CircuitBreakerConfig{
FailureRateThreshold: 0.3,
// ... other config options
Logger: bifrost.NewDefaultLogger(schemas.LogLevelDebug), // More verbose logging
}
```

## Circuit States

### CLOSED (Normal Operation)
- Requests are sent to providers normally
- Circuit breaker monitors failures and slow calls
- Metrics are collected in sliding window

### OPEN (Failure Protection)
- All requests are immediately rejected
- No provider calls are made
- Prevents cascading failures
- Automatically transitions to HALF_OPEN after wait duration

### HALF_OPEN (Recovery Testing)
- Limited number of requests are allowed through
- Success/failure determines next state
- Success → CLOSED (recovery complete)
- Failure → OPEN (still failing)

### CircuitState Type

The `CircuitState` type is an `int32` enum that represents the three possible states of the circuit breaker. It includes a `String()` method that provides human-readable string representations:

- `StateClosed` → `"CLOSED"`
- `StateOpen` → `"OPEN"`
- `StateHalfOpen` → `"HALF_OPEN"`

This is useful for logging, debugging, and displaying circuit breaker status in monitoring dashboards.

### Error Classification

The circuit breaker distinguishes between different types of errors:
- **Server Errors (5xx)**: Considered failures that contribute to the failure rate
- **Rate Limit Errors (429)**: Considered failures that contribute to the failure rate
- **Other Client Errors (4xx)**: Considered successful for circuit breaker purposes (e.g., invalid requests, authentication errors)

This classification ensures that rate limiting issues and server-side problems trigger circuit breaker protection, while other client-side issues (like invalid API keys or malformed requests) don't.

## Monitoring

### Get Circuit State

```go
state := plugin.GetState(schemas.OpenAI)
switch state {
case circuitbreaker.StateClosed:
fmt.Println("Circuit is CLOSED - normal operation")
case circuitbreaker.StateOpen:
fmt.Println("Circuit is OPEN - requests blocked")
case circuitbreaker.StateHalfOpen:
fmt.Println("Circuit is HALF_OPEN - testing recovery")
}
```

### Get Metrics

```go
metrics, err := plugin.GetMetrics(schemas.OpenAI)
if err == nil {
fmt.Printf("Total Calls: %d\n", metrics.TotalCalls)
fmt.Printf("Failed Calls: %d\n", metrics.FailedCalls)
fmt.Printf("Failure Rate: %.2f%%\n", metrics.FailureRate*100)
fmt.Printf("Slow Call Rate: %.2f%%\n", metrics.SlowCallRate*100)
}
```

## Advanced Operations

### Manual Circuit Control

The circuit breaker provides manual control functions for testing and emergency situations:

```go
// Force the circuit to open state (blocks all requests)
err := plugin.ForceOpen(schemas.OpenAI)
if err != nil {
fmt.Printf("Error forcing circuit open: %v\n", err)
}

// Force the circuit to closed state (allows all requests)
err = plugin.ForceClose(schemas.OpenAI)
if err != nil {
fmt.Printf("Error forcing circuit closed: %v\n", err)
}

// Reset the circuit breaker (clears all metrics and returns to closed state)
err = plugin.Reset(schemas.OpenAI)
if err != nil {
fmt.Printf("Error resetting circuit: %v\n", err)
}
```

**Note**: Manual control should be used sparingly and primarily for testing or emergency situations. The automatic circuit breaker logic is designed to handle most scenarios optimally.

## Performance

The Circuit Breaker plugin is optimized for high-performance scenarios:

- **Atomic Operations**: Uses atomic counters for thread-safe statistics
- **Lock-Free Reads**: Read operations don't block other operations
- **Memory Efficient**: Pre-allocated data structures with minimal allocations

## Best Practices

1. **Monitor Metrics**: Regularly check circuit states and failure rates
2. **Adjust Thresholds**: Lower thresholds for critical services, higher for non-critical
3. **Test Recovery**: Verify half-open state works correctly in your environment
4. **Use Fallbacks**: Combine with Bifrost's fallback providers for maximum resilience

**Need help?** Check the [Bifrost documentation](../../docs/plugins.md) or open an issue on GitHub.

36 changes: 36 additions & 0 deletions plugins/circuitbreaker/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
module github.com/maximhq/bifrost/plugins/circuitbreaker

go 1.24.1

toolchain go1.24.4

require github.com/maximhq/bifrost/core v1.1.6

require (
cloud.google.com/go/compute/metadata v0.3.0 // indirect
github.com/andybalholm/brotli v1.1.1 // indirect
github.com/aws/aws-sdk-go-v2 v1.36.3 // indirect
github.com/aws/aws-sdk-go-v2/config v1.29.14 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.17.67 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.30 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.34 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.34 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.15 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.25.3 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.30.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.33.19 // indirect
github.com/aws/smithy-go v1.22.3 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/mark3labs/mcp-go v0.32.0 // indirect
github.com/spf13/cast v1.7.1 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasthttp v1.60.0 // indirect
github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
golang.org/x/net v0.39.0 // indirect
golang.org/x/oauth2 v0.30.0 // indirect
golang.org/x/text v0.24.0 // indirect
)
58 changes: 58 additions & 0 deletions plugins/circuitbreaker/go.sum
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
cloud.google.com/go/compute/metadata v0.3.0 h1:Tz+eQXMEqDIKRsmY3cHTL6FVaynIjX2QxYC4trgAKZc=
cloud.google.com/go/compute/metadata v0.3.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1hT1fIilQDwofLpJ20k=
github.com/andybalholm/brotli v1.1.1 h1:PR2pgnyFznKEugtsUo0xLdDop5SKXd5Qf5ysW+7XdTA=
github.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=
github.com/aws/aws-sdk-go-v2 v1.36.3 h1:mJoei2CxPutQVxaATCzDUjcZEjVRdpsiiXi2o38yqWM=
github.com/aws/aws-sdk-go-v2 v1.36.3/go.mod h1:LLXuLpgzEbD766Z5ECcRmi8AzSwfZItDtmABVkRLGzg=
github.com/aws/aws-sdk-go-v2/config v1.29.14 h1:f+eEi/2cKCg9pqKBoAIwRGzVb70MRKqWX4dg1BDcSJM=
github.com/aws/aws-sdk-go-v2/config v1.29.14/go.mod h1:wVPHWcIFv3WO89w0rE10gzf17ZYy+UVS1Geq8Iei34g=
github.com/aws/aws-sdk-go-v2/credentials v1.17.67 h1:9KxtdcIA/5xPNQyZRgUSpYOE6j9Bc4+D7nZua0KGYOM=
github.com/aws/aws-sdk-go-v2/credentials v1.17.67/go.mod h1:p3C44m+cfnbv763s52gCqrjaqyPikj9Sg47kUVaNZQQ=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.30 h1:x793wxmUWVDhshP8WW2mlnXuFrO4cOd3HLBroh1paFw=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.30/go.mod h1:Jpne2tDnYiFascUEs2AWHJL9Yp7A5ZVy3TNyxaAjD6M=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.34 h1:ZK5jHhnrioRkUNOc+hOgQKlUL5JeC3S6JgLxtQ+Rm0Q=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.34/go.mod h1:p4VfIceZokChbA9FzMbRGz5OV+lekcVtHlPKEO0gSZY=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.34 h1:SZwFm17ZUNNg5Np0ioo/gq8Mn6u9w19Mri8DnJ15Jf0=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.34/go.mod h1:dFZsC0BLo346mvKQLWmoJxT+Sjp+qcVR1tRVHQGOH9Q=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.3 h1:bIqFDwgGXXN1Kpp99pDOdKMTTb5d2KyU5X/BZxjOkRo=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.3/go.mod h1:H5O/EsxDWyU+LP/V8i5sm8cxoZgc2fdNR9bxlOFrQTo=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.3 h1:eAh2A4b5IzM/lum78bZ590jy36+d/aFLgKF/4Vd1xPE=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.3/go.mod h1:0yKJC/kb8sAnmlYa6Zs3QVYqaC8ug2AbnNChv5Ox3uA=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.15 h1:dM9/92u2F1JbDaGooxTq18wmmFzbJRfXfVfy96/1CXM=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.15/go.mod h1:SwFBy2vjtA0vZbjjaFtfN045boopadnoVPhu4Fv66vY=
github.com/aws/aws-sdk-go-v2/service/sso v1.25.3 h1:1Gw+9ajCV1jogloEv1RRnvfRFia2cL6c9cuKV2Ps+G8=
github.com/aws/aws-sdk-go-v2/service/sso v1.25.3/go.mod h1:qs4a9T5EMLl/Cajiw2TcbNt2UNo/Hqlyp+GiuG4CFDI=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.30.1 h1:hXmVKytPfTy5axZ+fYbR5d0cFmC3JvwLm5kM83luako=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.30.1/go.mod h1:MlYRNmYu/fGPoxBQVvBYr9nyr948aY/WLUvwBMBJubs=
github.com/aws/aws-sdk-go-v2/service/sts v1.33.19 h1:1XuUZ8mYJw9B6lzAkXhqHlJd/XvaX32evhproijJEZY=
github.com/aws/aws-sdk-go-v2/service/sts v1.33.19/go.mod h1:cQnB8CUnxbMU82JvlqjKR2HBOm3fe9pWorWBza6MBJ4=
github.com/aws/smithy-go v1.22.3 h1:Z//5NuZCSW6R4PhQ93hShNbyBbn8BWCmCVCt+Q8Io5k=
github.com/aws/smithy-go v1.22.3/go.mod h1:t1ufH5HMublsJYulve2RKmHDC15xu1f26kHCp/HgceI=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/mark3labs/mcp-go v0.32.0 h1:fgwmbfL2gbd67obg57OfV2Dnrhs1HtSdlY/i5fn7MU8=
github.com/mark3labs/mcp-go v0.32.0/go.mod h1:rXqOudj/djTORU/ThxYx8fqEVj/5pvTuuebQ2RC7uk4=
github.com/maximhq/bifrost/core v1.1.3 h1:EBxwxqpCNNs3ck44qBwqrTiKGD+1Avyb57fM3/2wTKs=
github.com/maximhq/bifrost/core v1.1.3/go.mod h1:8ycaWQ9bjQezoUT/x6a82VmPjoqLzyGglQ0RnnlZjqo=
github.com/maximhq/bifrost/core v1.1.6 h1:rZrfPVcAfNggfBaOTdu/w+xNwDhW79bfexXsw8LRoMQ=
github.com/maximhq/bifrost/core v1.1.6/go.mod h1:yMRCncTgKYBIrECSRVxMbY3BL8CjLbipJlc644jryxc=
github.com/spf13/cast v1.7.1 h1:cuNEagBQEHWN1FnbGEjCXL2szYEXqfJPbP2HNUaca9Y=
github.com/spf13/cast v1.7.1/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.60.0 h1:kBRYS0lOhVJ6V+bYN8PqAHELKHtXqwq9zNMLKx1MBsw=
github.com/valyala/fasthttp v1.60.0/go.mod h1:iY4kDgV3Gc6EqhRZ8icqcmlG6bqhcDXfuHgTO4FXCvc=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zIM+UJPGz4=
github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT040gmv5BQTMR2HP4=
golang.org/x/net v0.39.0 h1:ZCu7HMWDxpXpaiKdhzIfaltL9Lp31x/3fCP11bc6/fY=
golang.org/x/net v0.39.0/go.mod h1:X7NRbYVEA+ewNkCNyJ513WmMdQ3BineSwVtN2zD/d+E=
golang.org/x/oauth2 v0.30.0 h1:dnDm7JmhM45NNpd8FDDeLhK6FwqbOf4MLCM9zb1BOHI=
golang.org/x/oauth2 v0.30.0/go.mod h1:B++QgG3ZKulg6sRPGD/mqlHQs5rB3Ml9erfeDY7xKlU=
golang.org/x/text v0.24.0 h1:dd5Bzh4yt5KYA8f9CJHCP4FB4D51c2c6JvN37xJJkJ0=
golang.org/x/text v0.24.0/go.mod h1:L8rBsPeo2pSS+xqN0d5u2ikmjtmoJbDBT1b7nHvFCdU=
Loading