Modular Nostr Network Observatory
Discovers relays across clearnet, Tor, I2P, and Lokinet. Validates connectivity, runs NIP-11 and NIP-66 health checks, archives events, materializes analytics views, and exposes data through a REST API and a NIP-90 Data Vending Machine.
BigBrotr answers three questions about the Nostr network:
- What relays exist? — Seeder bootstraps from a seed file, Finder discovers new relays from event tag values and external APIs.
- How healthy are they? — Validator confirms WebSocket connectivity, Monitor runs 7 health checks (RTT, SSL, DNS, Geo, Net, HTTP, NIP-11) and publishes NIP-66 events.
- What events are they publishing? — Synchronizer connects to relays, streams events, and archives them with cursor-based resumption.
Eight independent async services share a PostgreSQL database. Each runs on its own schedule, can be started or stopped individually, and has no direct dependency on any other service.
┌──────────────────────────────────────────────────────┐
│ PostgreSQL Database │
│ │
│ relay ─── event_relay ─── event │
│ metadata ─── relay_metadata │
│ service_state 11 materialized views │
└──┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──┘
│ │ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
Seeder Finder Valid. Monitor Sync. Refresh. Api Dvm
│ │ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ │ ▼ ▼
seed HTTP Relays Relays Relays (no I/O) HTTP Nostr
file APIs (WS) (NIP-11, (fetch clients
NIP-66) events) │
│ ▼
▼ Nostr Network
Nostr Network (kind 5050/6050)
(kind 10166/30166)
| Service | What it does | External I/O |
|---|---|---|
| Seeder | Loads relay URLs from a seed file (one-shot) | Seed file |
| Finder | Discovers relay URLs from event tag values and external APIs | HTTP (nostr.watch) |
| Validator | Tests candidates via WebSocket handshake, promotes valid relays | WebSocket |
| Monitor | Runs NIP-11 + 6 NIP-66 health checks, publishes kind 10166/30166 events | HTTP, WS, DNS, SSL, GeoIP |
| Synchronizer | Connects to relays, streams and archives signed events with cursor-based resumption | WebSocket |
| Refresher | Refreshes 11 materialized views in dependency order | None |
| Api | Read-only REST API with auto-generated paginated endpoints | HTTP (FastAPI) |
| Dvm | NIP-90 Data Vending Machine for database queries over Nostr | WebSocket (Nostr) |
All continuous services default to a 5-minute cycle interval (interval=300.0), configurable per deployment.
Services are loosely coupled through the database: Seeder and Finder populate candidates, Validator promotes them to relays, Monitor and Synchronizer operate on validated relays, Refresher materializes analytics. Stopping one does not break the others.
Imports flow strictly downward:
services src/bigbrotr/services/
/ | \
core nips utils src/bigbrotr/{core,nips,utils}/
\ | /
models src/bigbrotr/models/
- models — Pure frozen dataclasses (Relay, Event, Metadata, ServiceState). Zero I/O, stdlib logging only.
- core — Pool (asyncpg with retry), Brotr (DB facade), BaseService (lifecycle), Logger (structured kv/JSON), Metrics (Prometheus), YAML loader.
- nips — NIP-11 relay info fetch/parse, NIP-66 health checks (RTT, SSL, DNS, Geo, Net, HTTP). Never raises — errors captured in structured logs.
- utils — DNS resolution, Nostr key management, WebSocket/HTTP transport, SSL fallback, SOCKS5 proxy support, event streaming with binary-split windowing.
- services — 8 independent services + shared queries, configs, mixins (ConcurrentStream, NetworkSemaphores, GeoReader, Clients, CatalogAccess).
┌─────────────────────┐ ┌──────────────────────────────────────┐
│ relay │ │ event │
│─────────────────────│ │──────────────────────────────────────│
│ url PK │◄──┐ ┌──►│ id PK (BYTEA, 32B) │
│ network TEXT │ │ │ │ pubkey BYTEA (32B) │
│ discovered_at BIGINT│ │ │ │ created_at BIGINT │
└─────────┬───────────┘ │ │ │ kind INTEGER │
│ │ │ │ tags JSONB │
│ │ │ │ tagvalues TEXT[] │
│ │ │ │ content TEXT │
│ │ │ │ sig BYTEA (64B) │
│ │ │ └──────────────────────────────────────┘
│ │ │
│ ┌──────────┴─┴──────────────────┐
│ │ event_relay │
│ │───────────────────────────────│
├───►│ relay_url FK ──► relay.url |
│ │ event_id FK ──► event.id |
│ │ seen_at BIGINT |
│ │ PK(event_id, relay_url) |
│ └───────────────────────────────┘
│
│ ┌───────────────────────────────────────────────┐
│ │ relay_metadata │
│ │───────────────────────────────────────────────│
└───►│ relay_url FK ──► relay.url |
│ metadata_id FK ──► metadata.id |
│ metadata_type FK ──► metadata.type |
│ generated_at BIGINT |
│ PK(relay_url, generated_at, metadata_type) |
└──────────┬────────────────────────────────────┘
│
┌──────────┴────────────────────┐
│ metadata │
│───────────────────────────────│
│ id PK (BYTEA, SHA-256) |
│ type PK (TEXT, 7 types) |
│ data JSONB |
└───────────────────────────────┘
┌───────────────────────┐
│ service_state │
│───────────────────────│
│ service_name PK (TEXT)│
│ state_type PK (TEXT)│
│ state_key PK (TEXT)│
│ state_value JSONB │
└───────────────────────┘
Key relationships:
relayis the central entity. Cascade deletes propagate toevent_relayandrelay_metadata.metadatais content-addressed: SHA-256 hash of canonical JSON + type as composite PK. Same data = same hash.service_stateis a generic key-value store used by Finder (cursors), Validator (candidates), Monitor (checkpoints), Synchronizer (cursors).event.tagvaluesis computed at insert time byevent_insert()(fromtags_to_tagvalues(tags)) and indexed with GIN for fast containment queries.
relay event event_ meta- relay_ service_ materialized
relay data metadata state views (11)
─────────────┬────────┬───────┬────────┬───────┬─────────┬─────────┬────────────
Seeder │ W(1) │ │ │ │ │ W │
Finder │ R │ │ R │ │ │ R/W │
Validator │ W │ │ │ │ │ R/W │
Monitor │ R │ │ │ W │ W │ R/W │
Synchronizer │ R │ W │ W │ │ │ R/W │
Refresher │ │ │ │ │ │ │ W
Api │ R │ R │ R │ R │ R │ R │ R
Dvm │ R │ R │ R │ R │ R │ R │ R
─────────────┴────────┴───────┴────────┴───────┴─────────┴─────────┴────────────
R = reads W = writes (1) = only when to_validate=False
- Docker and Docker Compose
- (Optional) Python 3.11+ and uv for local development
git clone https://github.com/BigBrotr/bigbrotr.git
cd bigbrotr/deployments/bigbrotr
# Configure secrets
cp .env.example .env
# Edit .env: set DB_ADMIN_PASSWORD, DB_WRITER_PASSWORD, DB_REFRESHER_PASSWORD, DB_READER_PASSWORD, NOSTR_PRIVATE_KEY, GRAFANA_PASSWORD
# Start everything
docker compose up -d
# Watch services start
docker compose logs -f seederThis starts PostgreSQL 18, PGBouncer, Tor proxy, all 8 services, Prometheus, Alertmanager, and Grafana.
| Endpoint | URL |
|---|---|
| Grafana | http://localhost:3000 |
| Prometheus | http://localhost:9090 |
| Alertmanager | http://localhost:9093 |
| PostgreSQL | localhost:5432 |
| PGBouncer | localhost:6432 |
uv sync --group dev
cd deployments/bigbrotr
# One cycle
python -m bigbrotr seeder --once
# Continuous with debug logging
python -m bigbrotr finder --log-level DEBUGBigBrotr supports multiple deployment configurations from the same codebase via a single parametric Dockerfile (deployments/Dockerfile with ARG DEPLOYMENT).
Stores complete Nostr events (id, pubkey, created_at, kind, tags, content, sig).
cd deployments/bigbrotr && docker compose up -dSame eight services and schema, but tags, content, and sig columns are nullable and never populated — approximately 60% disk savings while retaining all metadata and relay health data.
cd deployments/lilbrotr && docker compose up -dcp -r deployments/bigbrotr deployments/myrelay
# Edit config, SQL schema, docker-compose.yaml
cd deployments/myrelay && docker compose up -dPostgreSQL 18 with PGBouncer (transaction-mode pooling) and asyncpg async driver. All mutations via stored functions with bulk array parameters.
| Table | Purpose |
|---|---|
relay |
Validated relay URLs with network type and discovery timestamp |
event |
Nostr events (BYTEA ids/pubkeys/sigs for space efficiency) |
event_relay |
Junction: which events were seen at which relays (with seen_at) |
metadata |
Content-addressed NIP-11/NIP-66 documents (SHA-256 dedup, composite PK (id, type)) |
relay_metadata |
Time-series snapshots linking relays to metadata records |
service_state |
Per-service operational data (candidates, cursors, checkpoints) |
- 1 utility:
tags_to_tagvalues(extracts key-prefixed single-char tag values for GIN indexing) - 10 CRUD:
relay_insert,event_insert,metadata_insert,event_relay_insert,relay_metadata_insert,event_relay_insert_cascade,relay_metadata_insert_cascade,service_state_upsert,service_state_get,service_state_delete - 2 cleanup:
orphan_event_delete,orphan_metadata_delete(batched) - 12 refresh: one per materialized view +
all_statistics_refresh
All functions use SECURITY INVOKER, bulk array parameters, and ON CONFLICT DO NOTHING.
relay_metadata_latest, event_stats, relay_stats, kind_counts, kind_counts_by_relay, pubkey_counts, pubkey_counts_by_relay, network_stats, relay_software_counts, supported_nip_counts, event_daily_counts — all support REFRESH CONCURRENTLY via unique indexes.
Every service exposes /metrics on its configured port with four metric types:
| Metric | Type | Description |
|---|---|---|
service_info |
Info | Static service metadata |
service_gauge |
Gauge | Point-in-time state (consecutive_failures, last_cycle_timestamp, progress) |
service_counter |
Counter | Cumulative totals (cycles_success, cycles_failed, errors by type) |
cycle_duration_seconds |
Histogram | Cycle latency with 10 buckets (1s to 1h) |
| Alert | Condition | Severity |
|---|---|---|
| ServiceDown | up == 0 for 5m |
critical |
| HighFailureRate | error rate > 0.1/s for 5m | warning |
| ConsecutiveFailures | 5+ consecutive cycle failures for 2m | critical |
| SlowCycles | p99 cycle duration > 300s for 5m | warning |
| DatabaseConnectionsHigh | > 80 active connections for 5m | warning |
| CacheHitRatioLow | buffer cache hit ratio < 95% for 10m | warning |
| RefresherViewsFailing | view refresh failures for 10m | warning |
Auto-provisioned dashboard with per-service panels: cycle duration, error counts, consecutive failures, and service-specific progress metrics.
info finder cycle_completed relay_count=100 duration=2.5
error validator retry_failed attempt=3 url="wss://relay.example.com"
JSON mode available for cloud aggregation:
{"timestamp": "2026-02-09T12:34:56+00:00", "level": "info", "service": "finder", "message": "cycle_completed", "relay_count": 100}| NIP | Usage |
|---|---|
| NIP-01 | Event model, relay communication |
| NIP-11 | Relay information document fetch and parse |
| NIP-42 | Relay authentication (Synchronizer auth, Validator detection) |
| NIP-66 | Relay monitoring and discovery (kinds 10166, 22456, 30166) |
| NIP-89 | Handler information (DVM announcement, kind 31990) |
| NIP-90 | Data Vending Machine (DVM job requests/results, kinds 5050/6050) |
| Kind | Direction | Purpose |
|---|---|---|
| 0 | Published | Monitor profile metadata |
| 5050 | Consumed | NIP-90 DVM job request |
| 6050 | Published | NIP-90 DVM job result |
| 10166 | Published | Monitor announcement (capabilities, networks, timeouts) |
| 22456 | Published | NIP-66 ephemeral relay test |
| 30166 | Published | Relay discovery (addressable, one per relay, health check tags) |
| 31990 | Published | NIP-89 handler information (DVM announcement) |
| Check | What It Measures | Networks |
|---|---|---|
| RTT | WebSocket open/read/write latency (ms), 3-phase with verification | All |
| SSL | Certificate validity, expiry, issuer, SANs, cipher, fingerprint | Clearnet |
| DNS | A/AAAA/CNAME/NS/PTR records, TTL | Clearnet |
| Geo | Country, city, coordinates, timezone, geohash (GeoLite2 City) | Clearnet |
| Net | IP address, ASN, organization, network ranges (GeoLite2 ASN) | Clearnet |
| HTTP | Server header, X-Powered-By (from WebSocket handshake) | All |
| Variable | Required | Description |
|---|---|---|
DB_ADMIN_PASSWORD |
Yes | PostgreSQL admin password |
DB_WRITER_PASSWORD |
Yes | Writer role password (Seeder, Finder, Validator, Monitor, Synchronizer) |
DB_REFRESHER_PASSWORD |
Yes | Refresher role password (matview ownership) |
DB_READER_PASSWORD |
Yes | Reader role password (Api, Dvm, postgres-exporter) |
NOSTR_PRIVATE_KEY |
For Monitor, Validator, Synchronizer, Dvm | Nostr private key (hex or nsec) for event signing and NIP-42 auth |
GRAFANA_PASSWORD |
For Grafana | Grafana admin password |
deployments/bigbrotr/config/
├── brotr.yaml # Pool, batch size, timeouts
└── services/
├── seeder.yaml # Seed file path, validate mode
├── finder.yaml # API sources (JMESPath), event scanning, concurrency
├── validator.yaml # Networks, cleanup, processing chunk size
├── monitor.yaml # Health checks, retry per type, publishing, GeoIP
├── synchronizer.yaml # Networks, filter, time range, per-relay overrides
├── refresher.yaml # View list, refresh interval
├── api.yaml # Host, port, pagination, CORS
└── dvm.yaml # NIP-90 kind, relay list, response format
All configs use Pydantic v2 validation with typed defaults and constraints.
git clone https://github.com/BigBrotr/bigbrotr.git && cd bigbrotr
curl -LsSf https://astral.sh/uv/install.sh | sh # install uv (one-time)
uv sync --group dev
pre-commit installmake lint # ruff check src/ tests/
make format # ruff format src/ tests/
make typecheck # mypy src/bigbrotr (strict mode)
make test # pytest unit tests (~2,737 tests)
make test-integration # pytest integration tests (~216 tests, requires Docker)
make test-fast # pytest -m "not slow"
make coverage # pytest --cov with HTML report (80% branch minimum)
make ci # all checks: lint + format-check + typecheck + test + sql-check + audit
make docs # build MkDocs documentation site
make docs-serve # serve docs locally with live reload
make build # build Python package (sdist + wheel)
make docker-build # build Docker image (DEPLOYMENT=bigbrotr)
make docker-up # start Docker stack
make docker-down # stop Docker stack
make clean # remove build artifacts and caches- ~2,737 unit tests + ~216 integration tests (testcontainers PostgreSQL)
asyncio_mode = "auto"— no@pytest.mark.asyncioneeded- Global timeout: 120s per test
- Shared fixtures via
tests/fixtures/relays.py(registered as pytest plugin) - Coverage threshold: 80% (branch coverage enabled)
| Stage | Tool | Purpose |
|---|---|---|
| Pre-commit | ruff, mypy, yamllint, detect-secrets, markdownlint, hadolint, sqlfluff, codespell | Code quality gates (23 hooks) |
| Unit Test | pytest (Python 3.11–3.14 matrix) | Unit tests + coverage |
| Integration Test | pytest + testcontainers | PostgreSQL integration tests |
| Build | Docker Buildx (matrix) | Multi-deployment image builds + Trivy scan |
| Security | uv-secure, Trivy, CodeQL | Dependency vulns, container scanning, static analysis |
| Release | PyPI (OIDC) + GHCR | Package + Docker image publishing, SBOM generation |
| Docs | MkDocs Material | Auto-generated API docs deployed to GitHub Pages |
| Dependencies | Dependabot | Weekly updates for uv, Docker, GitHub Actions |
bigbrotr/
├── src/bigbrotr/ # Main package
│ ├── __main__.py # CLI entry point (service registry)
│ ├── core/ # Infrastructure
│ │ ├── pool.py # asyncpg connection pool with retry/backoff
│ │ ├── brotr.py # DB facade (stored procedures, bulk inserts)
│ │ ├── base_service.py # Abstract service with run_forever loop
│ │ ├── logger.py # Structured key=value / JSON logging
│ │ ├── metrics.py # Prometheus metrics server
│ │ └── yaml.py # YAML config loader
│ ├── models/ # Pure frozen dataclasses (zero I/O)
│ │ ├── relay.py # URL validation (rfc3986), network detection
│ │ ├── event.py # Nostr event wrapper (nostr_sdk.Event)
│ │ ├── metadata.py # Content-addressed metadata (SHA-256)
│ │ ├── event_relay.py # Event-relay junction (cascade insert)
│ │ ├── relay_metadata.py # Relay-metadata junction (cascade insert)
│ │ ├── service_state.py # Operational state persistence
│ │ ├── constants.py # NetworkType, ServiceName, EventKind enums
│ │ └── _validation.py # Shared validation and sanitization
│ ├── nips/ # NIP protocol implementations (I/O)
│ │ ├── base.py # Base data, logs, metadata models
│ │ ├── parsing.py # Declarative field parsing (FieldSpec)
│ │ ├── event_builders.py # Kind 0/10166/30166 event construction
│ │ ├── nip11/ # Relay information document
│ │ └── nip66/ # Health checks: rtt, ssl, dns, geo, net, http
│ ├── utils/ # Network primitives
│ │ ├── protocol.py # Nostr client, relay connection, broadcasting
│ │ ├── transport.py # Insecure WebSocket transport, stderr filter
│ │ ├── streaming.py # Event streaming with binary-split windowing
│ │ ├── dns.py # Async hostname resolution (A/AAAA)
│ │ ├── keys.py # Nostr key loading from environment
│ │ ├── http.py # Bounded HTTP response reading
│ │ └── parsing.py # Tolerant model factory parsing
│ └── services/ # Business logic
│ ├── seeder/ # Seed file loading (one-shot)
│ ├── finder/ # Relay discovery (APIs + event scanning)
│ ├── validator/ # WebSocket protocol validation
│ ├── monitor/ # Health check orchestration + publishing
│ ├── synchronizer/ # Event collection (cursor-based)
│ ├── refresher/ # Materialized view refresh
│ ├── api/ # REST API (FastAPI, read-only)
│ ├── dvm/ # NIP-90 Data Vending Machine
│ └── common/ # Shared queries, configs, mixins
├── deployments/
│ ├── Dockerfile # Single parametric (ARG DEPLOYMENT)
│ ├── bigbrotr/ # Full archive deployment
│ │ ├── config/ # YAML configs (brotr + 8 services)
│ │ ├── postgres/init/ # SQL schema (10 files, 25 functions)
│ │ ├── monitoring/ # Prometheus + Alertmanager + Grafana
│ │ └── docker-compose.yaml # 15 containers, 2 networks
│ └── lilbrotr/ # Lightweight deployment
├── tests/
│ ├── fixtures/relays.py # Shared relay fixtures
│ ├── unit/ # ~2,737 tests (mirrors src/ structure)
│ └── integration/ # ~216 tests (testcontainers PostgreSQL)
├── docs/ # MkDocs Material documentation
├── Makefile # Development targets
└── pyproject.toml # All config: deps, ruff, mypy, pytest, coverage
| Container | Image | Purpose |
|---|---|---|
| postgres | postgres:18-alpine |
Primary storage |
| pgbouncer | edoburu/pgbouncer:v1.25.1-p0 |
Transaction-mode connection pooling |
| tor | osminogin/tor-simple:0.4.8.10 |
SOCKS5 proxy for .onion relays |
| seeder | bigbrotr (parametric) | Relay bootstrapping (one-shot) |
| finder | bigbrotr (parametric) | Relay discovery |
| validator | bigbrotr (parametric) | Candidate validation |
| monitor | bigbrotr (parametric) | Health monitoring + event publishing |
| synchronizer | bigbrotr (parametric) | Event archiving |
| refresher | bigbrotr (parametric) | Materialized view refresh |
| api | bigbrotr (parametric) | REST API (FastAPI) |
| dvm | bigbrotr (parametric) | NIP-90 Data Vending Machine |
| postgres-exporter | prometheuscommunity/postgres-exporter:v0.16.0 |
PostgreSQL metrics |
| prometheus | prom/prometheus:v2.51.0 |
Metrics collection (30d retention) |
| alertmanager | prom/alertmanager:v0.27.0 |
Alert routing and grouping |
| grafana | grafana/grafana:10.4.1 |
Dashboards |
data-network— postgres, pgbouncer, tor, all servicesmonitoring-network— prometheus, grafana, alertmanager, postgres-exporter, all services
- All ports bound to
127.0.0.1(no external exposure) - Non-root container execution (UID 1000)
tinias PID 1 for proper signal handling- SCRAM-SHA-256 authentication (PostgreSQL + PGBouncer)
- Healthchecks via
pg_isreadyand/metricsHTTP endpoint
| Category | Technologies |
|---|---|
| Language | Python 3.11+ (fully typed, strict mypy) |
| Database | PostgreSQL 18, asyncpg, PGBouncer |
| Async | asyncio, aiohttp, aiohttp-socks |
| Nostr | nostr-sdk (Rust FFI via UniFFI) |
| Web Framework | FastAPI, uvicorn |
| Validation | Pydantic v2, rfc3986 |
| Monitoring | Prometheus, Grafana, Alertmanager, structured logging |
| Networking | dnspython, geoip2, geohash2, tldextract, cryptography |
| Testing | pytest, pytest-asyncio, pytest-cov, testcontainers |
| Quality | ruff (lint+format), mypy (strict), pre-commit (23 hooks) |
| CI/CD | GitHub Actions, uv-secure, Trivy, CodeQL, Dependabot |
| Containers | Docker, Docker Compose, tini |
| Build | uv (dependency management + build) |
Full documentation is available at bigbrotr.github.io/bigbrotr.
| Section | Description |
|---|---|
| Getting Started | Installation, quick start tutorial, first deployment |
| User Guide | Architecture, services, configuration, database, monitoring |
| How-to Guides | Docker deploy, manual deploy, Tor setup, troubleshooting |
| Development | Setup, testing, contributing |
| API Reference | Auto-generated Python API docs |
| Changelog | Version history and migration guides |
See the Contributing Guide for detailed instructions.
- Fork and clone
uv sync --group devandpre-commit install- Write tests for new functionality
make ci— all checks must pass- Submit a pull request
Conventional commits: feat:, fix:, refactor:, docs:, test:, chore:
MIT — see LICENSE.