feat: Add dynamic burn rate alerting for traffic-aware SLO thresholds

[yairst]

Overview

This PR implements dynamic burn rate alerting that adapts alert thresholds based on actual traffic patterns, preventing false positives during low traffic and false negatives during high traffic periods.

Motivation

Traditional static burn rate multipliers (14x, 7x, 2x, 1x) don't account for traffic variations, leading to:

• False positives during low traffic (few errors trigger alerts due to small sample sizes)
• False negatives during high traffic (many errors go undetected)

Dynamic burn rates solve this by calculating thresholds that maintain consistent absolute error budget consumption regardless of traffic volume:

dynamic_threshold = (N_SLO / N_alert) × E_budget_percent × (1 - SLO_target)

Key Insight: This formula ensures alerts fire at the same absolute number of errors regardless of traffic. The threshold percentage adapts to traffic: lower during high traffic, higher during low traffic, but always requiring the same absolute error count.

This methodology is based on my blog post, "Error Budget Is All You Need - Part 2".

Implementation Summary

Backend Changes

Core Implementation (slo/rules.go):

• Added buildDynamicAlertExpr() method implementing traffic-aware threshold calculation
• Enhanced Burnrates() method to route between static and dynamic expressions
• Integrated dynamic window logic with proper E_budget_percent mapping (1/48, 1/16, 1/14, 1/7)
• Multi-window consistency: Both short and long windows use N_long for traffic scaling

CRD Changes (kubernetes/api/v1alpha1/servicelevelobjective_types.go):

• Added BurnRateType field to SLO spec (values: "static", "dynamic")
• Default: "static" (preserves existing behavior)
• Backward compatible: Existing SLOs continue working unchanged

Indicator Type Support:

• ✅ Ratio: Uses increase() for traffic calculation
• ✅ Latency: Uses histogram _count metrics with le="" label selector
• ✅ LatencyNative: Uses histogram_count(sum(increase(...))) for native histograms
• ✅ BoolGauge: Uses count_over_time() for boolean gauge observations

API Changes

Protobuf (proto/objectives/v1alpha1/objectives.proto):

• Added burn_rate_type field to Objective message
• Values: "static" (default), "dynamic"
• Full end-to-end transmission from CRD → Backend → API → UI

UI Changes

Core Components:

• List Page (ui/src/List.tsx): Added "Burn Rate" column with sortable badges
• Detail Page (ui/src/Detail.tsx): Added burn rate type badge with traffic context
• Alerts Table (ui/src/AlertsTable.tsx): Added "Error Budget Consumption" column
• Threshold Display (ui/src/components/BurnRateThresholdDisplay.tsx): Real-time dynamic threshold calculation
• Burn Rate Graph (ui/src/components/BurnrateGraph.tsx): Dynamic threshold visualization

User Experience Enhancements:

• Visual Indicators: Green "Dynamic" badges vs gray "Static" badges with appropriate icons
• Enhanced Tooltips: Context-aware explanations showing traffic impact on alert sensitivity
• Real-Time Calculations: Live threshold values instead of placeholder text
• Traffic Context: Shows current traffic ratio and above/below average status
• Error Handling: Graceful degradation for missing metrics with meaningful error messages

Backend Alert Rules:

• Alert rules optimized to use recording rules for SLO window calculation
• Reduces Prometheus evaluation load (rules evaluated every 30s)
• Maintains accuracy while improving performance

Testing Evidence

Mathematical Validation

Core Concept: Dynamic burn rates maintain consistent absolute error budget consumption regardless of traffic volume.

Mathematical Proof:

Alert fires when: error_rate > (N_SLO / N_alert) × E_budget_percent × (1 - SLO_target)

Since error_rate = errors / N_alert, we can substitute:

errors / N_alert > (N_SLO / N_alert) × E_budget_percent × (1 - SLO_target)

Multiply both sides by N_alert:

errors > N_SLO × E_budget_percent × (1 - SLO_target)

Since N_SLO × (1 - SLO_target) = E_budget (absolute error budget for SLO period):

errors > E_budget_percent × E_budget

Result: The N_alert terms cancel out! Alerts fire at the same absolute error count regardless of traffic.

Example Validation:

Given:

• SLO target: 99% (so 1 - SLO_target = 0.01)
• E_budget_percent: 0.02 (2% of error budget per alert window)
• N_SLO (30d): 1,000,000 requests
• E_budget (absolute): 1,000,000 × 0.01 = 10,000 errors allowed in 30d

High Traffic Scenario:

• N_alert (1h): 10,000 requests
• Traffic Ratio: 1,000,000 / 10,000 = 100x
• Dynamic Threshold: 100 × 0.02 × 0.01 = 0.02 (2%)
• Absolute errors needed: 10,000 × 0.02 = 200 errors

Low Traffic Scenario:

• N_alert (1h): 1,000 requests
• Traffic Ratio: 1,000,000 / 1,000 = 1,000x
• Dynamic Threshold: 1,000 × 0.02 × 0.01 = 0.2 (20%)
• Absolute errors needed: 1,000 × 0.2 = 200 errors

Same absolute threshold (200 errors), vastly different error rate thresholds (2% vs 20%)!

Benefits:

• Prevents false positives: During low traffic, 20 errors out of 1,000 (2%) won't alert because threshold is 20%
• Maintains sensitivity: During high traffic, 200 errors out of 10,000 (2%) will alert because threshold is 2%
• Consistent behavior: Always alerts when 200 errors occur (2% of the 10,000 error budget)

Validation Results:

• ✅ Window scaling correctly adapts to different SLO periods (28d → 30d)
• ✅ Recording rules use appropriate PromQL functions (rate, increase)
• ✅ Alert thresholds correctly implement dynamic formula
• ✅ E_budget_percent thresholds correctly map from static factors (14→1/48, 7→1/16, 2→1/14, 1→1/7)
• ✅ Multi-window logic uses consistent traffic scaling
• ✅ All indicator types (ratio, latency, latencyNative, boolGauge) validated

UI Regression Testing

Test Environment:

• 16 SLOs total (4 static, 12 dynamic)
• Multiple indicator types tested
• Both working and broken metrics scenarios
• Minikube cluster with kube-prometheus stack

Regression Testing Results:

• ✅ Zero regressions found - All original Pyrra functionality preserved
• ✅ Static SLO behavior identical to baseline (except intentional enhancements)
• ✅ 6 intentional new features successfully integrated
• ✅ No visual glitches, layout issues, or console errors
• ✅ Mixed static/dynamic environment stable

Production Build Validation:

• ✅ All production build tests passed
• ✅ Critical missing metrics fixes working perfectly (no white page crash)
• ✅ All indicator types working correctly
• ✅ Graceful error handling for missing/broken metrics
• ✅ Performance acceptable (< 3 seconds page load)

Alert Firing Validation

Validated Results:

• ✅ Synthetic traffic generation working (20 req/sec with configurable error rate)
• ✅ Alert state transitions detected: inactive → pending → firing
• ✅ Both static and dynamic alerts fire correctly
• ✅ Dynamic alerts demonstrate improved sensitivity

Browser Compatibility Testing

Browsers Tested:

• ✅ Chrome (primary development browser)
• ✅ Firefox (full compatibility confirmed)
• ⚠️ Edge (not tested - assumed compatible as Chromium-based)

Graceful Degradation Testing:

• ✅ Network throttling: Proper loading states and retry logic
• ✅ API failures: Meaningful error messages
• ✅ Prometheus unavailability: Graceful fallback displays
• ✅ Missing metrics: No crashes, appropriate error states

Breaking Changes

None. This feature is completely opt-in and backward compatible:

• Default behavior: burnRateType: static (existing behavior)
• Existing SLOs: Continue working unchanged
• Migration: Add burnRateType: dynamic to enable new behavior
• No schema changes that break existing deployments

Migration Guide

Enabling Dynamic Burn Rates

For new SLOs, add burnRateType: dynamic to the alerting section:

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: my-service-slo
spec:
  target: "99"
  window: 28d
  indicator:
    ratio:
      errors:
        metric: http_requests_total{code=~"5.."}
      total:
        metric: http_requests_total
  alerting:
    name: MyServiceErrorBudgetBurn
    burnRateType: dynamic  # Add this line
    burnrates: true

For existing SLOs, edit the SLO YAML and add burnRateType: dynamic:

kubectl edit slo my-service-slo -n monitoring

Validation

After enabling dynamic burn rates:

1. Check Prometheus Rules: Verify dynamic expressions generated

   kubectl get prometheusrule -n monitoring

2. Check UI: Verify green "Dynamic" badge appears on SLO list page

3. Check Thresholds: Verify calculated threshold values display in alerts table

4. Monitor Alerts: Observe alert behavior with traffic variations

Rollback

To revert to static burn rates:

alerting:
  burnRateType: static  # Change back to static

Or remove the field entirely (defaults to static).

Examples

Four comprehensive examples are provided in the examples/ directory:

1. examples/dynamic-burn-rate-ratio.yaml - Ratio indicator (API success rate)
2. examples/dynamic-burn-rate-latency.yaml - Latency indicator (histogram-based)
3. examples/dynamic-burn-rate-latency-native.yaml - Native histogram latency
4. examples/dynamic-burn-rate-bool-gauge.yaml - Boolean gauge (availability)

Each example includes:

• Clear comments explaining use cases
• Proper metric selectors
• Recommended configuration values
• Traffic-aware alerting benefits

Design Decisions

1. Opt-In Feature (Not Default)

Decision: Dynamic burn rates require explicit burnRateType: dynamic configuration

Rationale:

• Preserves existing behavior for current users
• Allows gradual adoption and testing
• Reduces risk of unexpected alert behavior changes
• Users can evaluate feature before full deployment

2. Latency Indicator Label Selector

Decision: Always add le="" label selector when querying latency recording rules

Rationale:

• Latency indicators create TWO recording rules: total (le="") and success (le="0.1")
• Without le="", sum() aggregation includes BOTH rules (2x traffic)
• Explicit le="" selector ensures only total traffic is counted
• Critical for accurate dynamic threshold calculation

3. Error Handling Strategy

Decision: Graceful degradation with fallback displays instead of crashes

Rationale:

• Production environments may have missing or misconfigured metrics
• Users need visibility into SLO configuration even with data issues
• Fallback to "Traffic-Aware" or "No data" better than white page crash
• Console warnings help debugging without breaking UI

Documentation

User-Facing Documentation

Updated Files:

• README.md - Added dynamic burn rate feature section
• examples/README.md - Added dynamic SLO examples with explanations
• examples/*.yaml - Four comprehensive example configurations

Development Documentation (Fork Only)

Comprehensive Internal Documentation (40+ documents in .dev-docs/):

• Implementation summaries and session notes
• Testing procedures and validation reports
• Mathematical correctness validation
• Performance benchmarks and optimization analysis
• Browser compatibility matrices
• Migration guides and troubleshooting
• Development workflow and standards

Development Tools (Fork Only - cmd/ directory):

• Query performance validation tools
• Threshold calculation testing tools
• Alert rule validation tools
• Recording rule validation tools
• Synthetic metric generation for testing
• Performance monitoring tools

References

Methodology:

• "Error Budget Is All You Need - Part 2" - Blog post explaining dynamic burn rate methodology

---

Before/After Examples

Example 1: Static vs Dynamic Threshold Comparison

Scenario: API service with 99% SLO target, 30d window

Static Burn Rate (Factor 14):

Threshold = 14 × (1 - 0.99) = 0.14 (14% error rate)

• Same threshold regardless of traffic
• 14% error rate required to trigger alert
• Does not adapt to traffic patterns

Dynamic Burn Rate (High Traffic):

N_SLO (30d): 1,000,000 requests
N_alert (1h): 10,000 requests
Traffic Ratio: 100x
E_budget_percent: 0.02 (2% of error budget)

Threshold = 100 × 0.02 × 0.01 = 0.02 (2% error rate)
Absolute errors needed = 10,000 × 0.02 = 200 errors

• Lower threshold percentage (2%)
• Same absolute errors needed (200 errors)

Dynamic Burn Rate (Low Traffic):

N_SLO (30d): 1,000,000 requests
N_alert (1h): 1,000 requests
Traffic Ratio: 1,000x
E_budget_percent: 0.02 (2% of error budget)

Threshold = 1,000 × 0.02 × 0.01 = 0.2 (20% error rate)
Absolute errors needed = 1,000 × 0.2 = 200 errors

• Higher threshold percentage (20%)
• Same absolute errors needed (200 errors)
• Prevents false positives from small sample sizes

Key Insight: Both scenarios require the same absolute number of errors (200), but the error rate thresholds differ dramatically (2% vs 20%).

Example 2: UI Display Comparison

Static SLO - List Page:

┌─────────────────────────────────────────────────┐
│ Name: apiserver-requests-static                 │
│ Burn Rate: [Static] 🔒                          │
│ Availability: 99.95%                            │
│ Budget: 95.2%                                   │
└─────────────────────────────────────────────────┘

Dynamic SLO - List Page:

┌─────────────────────────────────────────────────┐
│ Name: apiserver-requests-dynamic                │
│ Burn Rate: [Dynamic] 👁                         │
│ Availability: 99.95%                            │
│ Budget: 95.2%                                   │
└─────────────────────────────────────────────────┘

Static SLO - Alerts Table:

┌──────────┬──────────┬────────────┬────────┬───────────┐
│ Severity │ Exhaust  │ Factor     │ Thresh │ Short     │
├──────────┼──────────┼────────────┼────────┼───────────┤
│ critical │ 2d       │ 14         │ 0.700  │ 0.123     │
│ critical │ 6d       │ 7          │ 0.350  │ 0.456     │
│ warning  │ 12d      │ 2          │ 0.100  │ 0.789     │
│ warning  │ 30d      │ 1          │ 0.050  │ 0.234     │
└──────────┴──────────┴────────────┴────────┴───────────┘

Dynamic SLO - Alerts Table:

┌──────────┬──────────┬────────────┬────────┬───────────┐
│ Severity │ Exhaust  │ Error Bdgt │ Thresh │ Short     │
├──────────┼──────────┼────────────┼────────┼───────────┤
│ critical │ 2d       │ 2.08%      │ 0.0046 │ 0.123     │
│ critical │ 6d       │ 6.25%      │ 0.0137 │ 0.456     │
│ warning  │ 12d      │ 7.14%      │ 0.0156 │ 0.789     │
│ warning  │ 30d      │ 14.29%     │ 0.0313 │ 0.234     │
└──────────┴──────────┴────────────┴────────┴───────────┘

Tooltip Comparison:

Static SLO Tooltip:

Static Burn Rate
Uses fixed multipliers (14x, 7x, 2x, 1x) for alert thresholds.
Threshold = 14 × (1 - 0.99) = 0.700

Dynamic SLO Tooltip:

Dynamic Burn Rate
Adapts thresholds based on traffic patterns.

Error Budget: 2.08% (burns 2.08% of budget per alert window)
Traffic Ratio: 100x (high traffic)
Dynamic Threshold: 0.02 (2%) - Adapts to traffic
Static Threshold: 0.14 (14%) - Same threshold regardless of traffic

Formula: (N_SLO / N_alert) × E_budget_percent × (1 - SLO_target)

Example 3: Alert Rule Comparison

Static Alert Rule (PrometheusRule):

- alert: ApiserverRequestsStaticErrorBudgetBurn
  expr: |
    (
      apiserver_request:burnrate5m{slo="apiserver-requests-static"} > (14 * (1 - 0.99))
      and
      apiserver_request:burnrate1h4m{slo="apiserver-requests-static"} > (14 * (1 - 0.99))
    )
  labels:
    severity: critical
    long: 1h4m
    short: 5m

Dynamic Alert Rule (PrometheusRule):

- alert: ApiserverRequestsDynamicErrorBudgetBurn
  expr: |
    (
      apiserver_request:burnrate5m{slo="apiserver-requests-dynamic"} > 
      scalar((sum(apiserver_request:increase30d{slo="apiserver-requests-dynamic"}) / 
              sum(increase(apiserver_request_total{verb="GET"}[1h4m]))) * 0.020833 * (1 - 0.99))
      and
      apiserver_request:burnrate1h4m{slo="apiserver-requests-dynamic"} > 
      scalar((sum(apiserver_request:increase30d{slo="apiserver-requests-dynamic"}) / 
              sum(increase(apiserver_request_total{verb="GET"}[1h4m]))) * 0.020833 * (1 - 0.99))
    )
  labels:
    severity: critical
    long: 1h4m
    short: 5m

Key Differences:

1. Static uses fixed multiplier (14)
2. Dynamic calculates traffic ratio (N_SLO / N_alert)
3. Dynamic uses E_budget_percent (0.020833 = 1/48)
4. Dynamic uses recording rules for SLO window (optimized)
5. Both use same burn rate recording rules for error rate

[metalmatze]

Wow!
This is incredible!

First thing, I'll have to read your blog post and fully understand how things are working.
Then, if we decide on adding the feature to Pyrra, I would like to know if we should go and split the features into various PRs. Currently, it's going to be quite hard to add with a gigantic PR, and probably won't be a good experience reviewing all files within one PR.