Self-hosted AI SRE agent for incident response automation across Prometheus, Grafana, Loki, Tempo, Alertmanager, Kubernetes, Slack, and an Obsidian-compatible memory vault.
flowchart LR
A[Alertmanager / Webhook] --> B[OBSERVE]
B --> C[REASON]
C --> D{Approval Gate}
D -->|read-only| E[ACT]
D -->|risky action| F[Human Approval]
F --> E
E --> G[LEARN]
G --> H[Markdown Vault]
H --> C
B --> I[Prometheus]
B --> J[Loki]
B --> K[Tempo]
E --> L[Kubernetes / Runbooks]
E --> M[Slack / PagerDuty / GitHub]
Production incidents are noisy, fragmented, and repetitive. SRE Agent gives teams a transparent AI incident investigator that runs locally, keeps memory in plain Markdown, and uses approval gates before touching production.
| Problem | SRE Agent approach |
|---|---|
| Alerts lack context | Pulls logs, metrics, traces, topology, and deploys into one loop |
| AI tools feel like black boxes | Explicit OBSERVE -> REASON -> ACT -> LEARN state machine |
| Vendor lock-in | Self-hosted, adapter-based, LiteLLM-compatible model routing |
| Lost incident knowledge | Writes reusable runbooks, service notes, and patterns to a vault |
| Risky remediation | Human approval gates for destructive or production-changing actions |
| Capability | SRE Agent | HolmesGPT | Microsoft SRE Agent | Lucas |
|---|---|---|---|---|
| Self-hosted | Yes | Yes | Partial | Yes |
| Markdown/Obsidian memory | Yes | No | No | No |
| Approval gates | Yes | Partial | Yes | Partial |
| LiteLLM model routing | Yes | No | No | No |
| Grafana stack demo | Yes | Yes | No | Partial |
| Extensible adapter pattern | Yes | Yes | Partial | Partial |
- Alertmanager and generic webhook triggers
- Deterministic
OBSERVE -> REASON -> ACT -> LEARNinvestigation loop - Context-aware reasoning with token budget management and compression
- Human approval gates for risky remediation actions
- Prometheus metrics queries
- Loki log search
- Tempo trace lookups
- Grafana-stack adapter interface
- Deployment and topology discovery
- Kubernetes pod restart, scaling, and rollback tools
- Database lock inspection and query termination tools
- Runbook execution with approval controls
- Code sandbox for safe diagnostic scripts
- Obsidian-compatible Markdown vault
- Incident, service, pattern, and runbook templates
- Vault search and recall tools
- Baseline and topology history tracking
- Composio-powered integrations for Slack, PagerDuty, GitHub, and Linear
- Structured incident summaries
- Tool-call audit trail for postmortems
| Tool | Version |
|---|---|
| Python | 3.12+ |
| uv | latest |
| Docker | 24.0+ |
| Docker Compose | v2+ |
You also need:
- An Anthropic or OpenAI API key for LiteLLM
- A Composio API key for external notifications and ticketing integrations
git clone https://github.com/alparn/sre-agent.git
cd sre-agent
cp .env.example .env
docker compose -f docker-compose.yaml -f demo/docker-compose.demo.yaml up --buildOpen the local services:
| Service | URL |
|---|---|
| SRE Agent API | http://localhost:8000 |
| Grafana | http://localhost:3001 |
| Prometheus | http://localhost:9090 |
| Alertmanager | http://localhost:9093 |
| LiteLLM | http://localhost:4000 |
git clone https://github.com/alparn/sre-agent.git
cd sre-agent
uv sync --group dev
cp .env.example .env
uv run uvicorn src.main:app --host 0.0.0.0 --port 8000 --reloadKubernetes deployment manifests and Helm charts are planned. Until then, use the Docker image and configure the same environment variables documented below.
Start the demo stack:
docker compose -f docker-compose.yaml -f demo/docker-compose.demo.yaml up --buildTrigger a synthetic incident:
uv run python scripts/run_demo_incident.pyExpected result:
Investigation started
Alert: HighErrorRate
Service: demo-service
Agent loop: OBSERVE -> REASON -> ACT -> LEARN
Vault entry written under vault/incidents/
You can also trigger the webhook directly:
curl -X POST http://localhost:8000/webhook \
-H "Content-Type: application/json" \
-d '{
"title": "High API error rate",
"service": "demo-service",
"severity": "critical",
"description": "5xx error rate exceeded threshold"
}'SRE Agent is intentionally built without a heavyweight agent framework. The runtime is a small, inspectable loop with typed tools, a model router, and an approval gate.
| Component | Responsibility |
|---|---|
src/main.py |
FastAPI webhooks, health checks, dependency wiring |
src/agent/ |
Agent loop, context, state transitions, approval gates |
src/tools/observe/ |
Logs, metrics, traces, alerts, topology, deploys |
src/tools/diagnose/ |
Correlation, anomaly detection, metric diffs |
src/tools/act/ |
Kubernetes, database, runbook, and sandbox actions |
src/tools/memory/ |
Vault read/write/search/recall tools |
src/adapters/ |
Observability backend abstraction |
src/vault/ |
Markdown vault writer, reader, search, and indexer |
src/compression/ |
Token budgets, rule summaries, LLM compression |
src/integrations/ |
Composio-backed external integrations |
SRE Agent reads YAML configuration from config/ and secrets from environment
variables. Never commit a real .env file.
cp .env.example .env| Variable | Required | Description |
|---|---|---|
ANTHROPIC_API_KEY |
one LLM key required | Anthropic key used by LiteLLM |
OPENAI_API_KEY |
one LLM key required | OpenAI key used by LiteLLM |
COMPOSIO_API_KEY |
yes | Enables Slack, PagerDuty, GitHub, Linear tools |
GRAFANA_ADMIN_USER |
no | Local Grafana user, defaults to admin |
GRAFANA_ADMIN_PASSWORD |
no | Local Grafana password, defaults to admin |
GRAFANA_PORT |
no | Local Grafana port, defaults to 3001 |
SRE_AGENT_CONFIG_PATH |
no | Override runtime config path |
agent:
max_iterations: 8
default_severity: warning
observability:
provider: grafana
grafana:
prometheus_url: http://prometheus:9090
loki_url: http://loki:3100
tempo_url: http://tempo:3200
vault:
path: /app/vault
index_path: /app/vault/.index.sqlite
approval:
require_human_for:
- restart_pod
- scale_deployment
- rollback_deployment
- db_kill_query| Category | Supported | Planned |
|---|---|---|
| Metrics | Prometheus | Datadog, CloudWatch |
| Logs | Loki | Elastic, Splunk |
| Traces | Tempo | Jaeger |
| Alerts | Alertmanager, webhook | PagerDuty, Opsgenie |
| LLMs | Anthropic, OpenAI, Ollama via LiteLLM | Gemini |
| Chat and tickets | Slack, GitHub, Linear via Composio | Teams, Discord |
| Runtime | Docker Compose | Helm, Kubernetes manifests |
- Trigger: Alertmanager or a webhook sends an incident payload.
- Observe: The agent collects metrics, logs, traces, deploy history, and topology for the affected service.
- Reason: The model router asks the configured LLM what evidence to gather next or which safe action to propose.
- Act: Read-only tools run automatically. Risky tools go through the approval gate.
- Learn: The final summary, evidence, and reusable patterns are written to the local Markdown vault.
Example investigation output
Incident: High API error rate
Evidence:
- Prometheus: 5xx rate increased from 0.2% to 8.7%
- Loki: token validation failures started after deployment api-7f9d4
- Tempo: p95 latency increased on /auth/validate
Likely cause:
Recent auth middleware deployment rejects valid tokens for one tenant.
Recommended action:
Rollback deployment api-7f9d4 after human approval.
Vault updates:
- incidents/2026-05-19-high-api-error-rate.md
- patterns/token-validation-failures.md
uv sync --group dev
uv run ruff check .
uv run ruff format .
uv run pytest- Do not commit
.env, API keys, vault data, logs, or database files. - All secrets must come from environment variables or a secret manager.
- Destructive actions require explicit approval.
- See SECURITY.md for responsible disclosure.
- Datadog and CloudWatch adapters
- Helm chart and production Kubernetes manifests
- GitHub Codespaces demo environment
- Postmortem generation
- Additional memory backends
- Multi-tenant approval policies
Contributions are welcome. Start with CONTRIBUTING.md, look
for good first issue, and keep PRs small and reviewable.
SRE Agent is licensed under the Apache License 2.0.