Skip to content

BigBrotr/bigbrotr

CI CodeQL Coverage Docs Python 3.11+ PostgreSQL 18 License

BigBrotr

Modular Nostr Network Observatory

Discovers relays across clearnet, Tor, I2P, and Lokinet. Validates connectivity, runs NIP-11 and NIP-66 health checks, archives events, materializes analytics views, and exposes data through a REST API and a NIP-90 Data Vending Machine.


What It Does

BigBrotr answers three questions about the Nostr network:

  1. What relays exist? — Seeder bootstraps from a seed file, Finder discovers new relays from event tag values and external APIs.
  2. How healthy are they? — Validator confirms WebSocket connectivity, Monitor runs 7 health checks (RTT, SSL, DNS, Geo, Net, HTTP, NIP-11) and publishes NIP-66 events.
  3. What events are they publishing? — Synchronizer connects to relays, streams events, and archives them with cursor-based resumption.

Eight independent async services share a PostgreSQL database. Each runs on its own schedule, can be started or stopped individually, and has no direct dependency on any other service.

                    ┌──────────────────────────────────────────────────────┐
                    │                    PostgreSQL Database               │
                    │                                                      │
                    │         relay ─── event_relay ─── event              │
                    │         metadata ─── relay_metadata                  │
                    │         service_state     11 materialized views      │
                    └──┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──┘
                       │      │      │      │      │      │      │      │
                       ▼      ▼      ▼      ▼      ▼      ▼      ▼      ▼
                    Seeder Finder Valid. Monitor Sync. Refresh. Api    Dvm
                       │      │      │      │      │      │      │      │
                       ▼      ▼      ▼      ▼      ▼      │      ▼      ▼
                    seed   HTTP   Relays Relays  Relays (no I/O) HTTP  Nostr
                    file   APIs   (WS)  (NIP-11, (fetch               clients
                                         NIP-66)  events)               │
                                           │                            ▼
                                           ▼                       Nostr Network
                                      Nostr Network               (kind 5050/6050)
                                    (kind 10166/30166)

Services

Service What it does External I/O
Seeder Loads relay URLs from a seed file (one-shot) Seed file
Finder Discovers relay URLs from event tag values and external APIs HTTP (nostr.watch)
Validator Tests candidates via WebSocket handshake, promotes valid relays WebSocket
Monitor Runs NIP-11 + 6 NIP-66 health checks, publishes kind 10166/30166 events HTTP, WS, DNS, SSL, GeoIP
Synchronizer Connects to relays, streams and archives signed events with cursor-based resumption WebSocket
Refresher Refreshes 11 materialized views in dependency order None
Api Read-only REST API with auto-generated paginated endpoints HTTP (FastAPI)
Dvm NIP-90 Data Vending Machine for database queries over Nostr WebSocket (Nostr)

All continuous services default to a 5-minute cycle interval (interval=300.0), configurable per deployment.

Services are loosely coupled through the database: Seeder and Finder populate candidates, Validator promotes them to relays, Monitor and Synchronizer operate on validated relays, Refresher materializes analytics. Stopping one does not break the others.


Architecture

Code Organization (Diamond DAG)

Imports flow strictly downward:

              services         src/bigbrotr/services/
             /   |   \
          core  nips  utils    src/bigbrotr/{core,nips,utils}/
             \   |   /
              models           src/bigbrotr/models/
  • models — Pure frozen dataclasses (Relay, Event, Metadata, ServiceState). Zero I/O, stdlib logging only.
  • core — Pool (asyncpg with retry), Brotr (DB facade), BaseService (lifecycle), Logger (structured kv/JSON), Metrics (Prometheus), YAML loader.
  • nips — NIP-11 relay info fetch/parse, NIP-66 health checks (RTT, SSL, DNS, Geo, Net, HTTP). Never raises — errors captured in structured logs.
  • utils — DNS resolution, Nostr key management, WebSocket/HTTP transport, SSL fallback, SOCKS5 proxy support, event streaming with binary-split windowing.
  • services — 8 independent services + shared queries, configs, mixins (ConcurrentStream, NetworkSemaphores, GeoReader, Clients, CatalogAccess).

Database Schema

┌─────────────────────┐         ┌──────────────────────────────────────┐
│      relay          │         │              event                   │
│─────────────────────│         │──────────────────────────────────────│
│ url          PK     │◄──┐ ┌──►│ id             PK  (BYTEA, 32B)      │
│ network      TEXT   │   │ │   │ pubkey         BYTEA (32B)           │
│ discovered_at BIGINT│   │ │   │ created_at     BIGINT                │
└─────────┬───────────┘   │ │   │ kind           INTEGER               │
          │               │ │   │ tags           JSONB                 │
          │               │ │   │ tagvalues      TEXT[]                │
          │               │ │   │ content        TEXT                  │
          │               │ │   │ sig            BYTEA (64B)           │
          │               │ │   └──────────────────────────────────────┘
          │               │ │
          │    ┌──────────┴─┴──────────────────┐
          │    │          event_relay          │
          │    │───────────────────────────────│
          ├───►│ relay_url    FK ──► relay.url |
          │    │ event_id     FK ──► event.id  |
          │    │ seen_at      BIGINT           |
          │    │ PK(event_id, relay_url)       |
          │    └───────────────────────────────┘
          │
          │    ┌───────────────────────────────────────────────┐
          │    │                   relay_metadata              │
          │    │───────────────────────────────────────────────│
          └───►│ relay_url    FK ──► relay.url                 |
               │ metadata_id  FK ──► metadata.id               |
               │ metadata_type FK ──► metadata.type            |
               │ generated_at BIGINT                           |
               │ PK(relay_url, generated_at, metadata_type)    |
               └──────────┬────────────────────────────────────┘
                          │
               ┌──────────┴────────────────────┐
               │          metadata             │
               │───────────────────────────────│
               │ id       PK  (BYTEA, SHA-256) |
               │ type     PK  (TEXT, 7 types)  |
               │ data     JSONB                |
               └───────────────────────────────┘


               ┌───────────────────────┐
               │    service_state      │
               │───────────────────────│
               │ service_name PK (TEXT)│
               │ state_type   PK (TEXT)│
               │ state_key    PK (TEXT)│
               │ state_value  JSONB    │
               └───────────────────────┘

Key relationships:

  • relay is the central entity. Cascade deletes propagate to event_relay and relay_metadata.
  • metadata is content-addressed: SHA-256 hash of canonical JSON + type as composite PK. Same data = same hash.
  • service_state is a generic key-value store used by Finder (cursors), Validator (candidates), Monitor (checkpoints), Synchronizer (cursors).
  • event.tagvalues is computed at insert time by event_insert() (from tags_to_tagvalues(tags)) and indexed with GIN for fast containment queries.

Service-Database Interaction Map

                 relay    event   event_   meta-   relay_    service_  materialized
                                  relay    data    metadata  state     views (11)
  ─────────────┬────────┬───────┬────────┬───────┬─────────┬─────────┬────────────
  Seeder       │  W(1)  │       │        │       │         │  W      │
  Finder       │  R     │       │  R     │       │         │  R/W    │
  Validator    │  W     │       │        │       │         │  R/W    │
  Monitor      │  R     │       │        │  W    │  W      │  R/W    │
  Synchronizer │  R     │  W    │  W     │       │         │  R/W    │
  Refresher    │        │       │        │       │         │         │  W
  Api          │  R     │  R    │  R     │  R    │  R      │  R      │  R
  Dvm          │  R     │  R    │  R     │  R    │  R      │  R      │  R
  ─────────────┴────────┴───────┴────────┴───────┴─────────┴─────────┴────────────

  R = reads    W = writes    (1) = only when to_validate=False

Quick Start

Prerequisites

  • Docker and Docker Compose
  • (Optional) Python 3.11+ and uv for local development

Deploy with Docker Compose

git clone https://github.com/BigBrotr/bigbrotr.git
cd bigbrotr/deployments/bigbrotr

# Configure secrets
cp .env.example .env
# Edit .env: set DB_ADMIN_PASSWORD, DB_WRITER_PASSWORD, DB_REFRESHER_PASSWORD, DB_READER_PASSWORD, NOSTR_PRIVATE_KEY, GRAFANA_PASSWORD

# Start everything
docker compose up -d

# Watch services start
docker compose logs -f seeder

This starts PostgreSQL 18, PGBouncer, Tor proxy, all 8 services, Prometheus, Alertmanager, and Grafana.

Endpoint URL
Grafana http://localhost:3000
Prometheus http://localhost:9090
Alertmanager http://localhost:9093
PostgreSQL localhost:5432
PGBouncer localhost:6432

Run a Single Service Locally

uv sync --group dev
cd deployments/bigbrotr

# One cycle
python -m bigbrotr seeder --once

# Continuous with debug logging
python -m bigbrotr finder --log-level DEBUG

Deployments

BigBrotr supports multiple deployment configurations from the same codebase via a single parametric Dockerfile (deployments/Dockerfile with ARG DEPLOYMENT).

BigBrotr (Full Archive)

Stores complete Nostr events (id, pubkey, created_at, kind, tags, content, sig).

cd deployments/bigbrotr && docker compose up -d

LilBrotr (Lightweight)

Same eight services and schema, but tags, content, and sig columns are nullable and never populated — approximately 60% disk savings while retaining all metadata and relay health data.

cd deployments/lilbrotr && docker compose up -d

Custom Deployment

cp -r deployments/bigbrotr deployments/myrelay
# Edit config, SQL schema, docker-compose.yaml
cd deployments/myrelay && docker compose up -d

Database

PostgreSQL 18 with PGBouncer (transaction-mode pooling) and asyncpg async driver. All mutations via stored functions with bulk array parameters.

Schema

Table Purpose
relay Validated relay URLs with network type and discovery timestamp
event Nostr events (BYTEA ids/pubkeys/sigs for space efficiency)
event_relay Junction: which events were seen at which relays (with seen_at)
metadata Content-addressed NIP-11/NIP-66 documents (SHA-256 dedup, composite PK (id, type))
relay_metadata Time-series snapshots linking relays to metadata records
service_state Per-service operational data (candidates, cursors, checkpoints)

Stored Functions (25)

  • 1 utility: tags_to_tagvalues (extracts key-prefixed single-char tag values for GIN indexing)
  • 10 CRUD: relay_insert, event_insert, metadata_insert, event_relay_insert, relay_metadata_insert, event_relay_insert_cascade, relay_metadata_insert_cascade, service_state_upsert, service_state_get, service_state_delete
  • 2 cleanup: orphan_event_delete, orphan_metadata_delete (batched)
  • 12 refresh: one per materialized view + all_statistics_refresh

All functions use SECURITY INVOKER, bulk array parameters, and ON CONFLICT DO NOTHING.

Materialized Views (11)

relay_metadata_latest, event_stats, relay_stats, kind_counts, kind_counts_by_relay, pubkey_counts, pubkey_counts_by_relay, network_stats, relay_software_counts, supported_nip_counts, event_daily_counts — all support REFRESH CONCURRENTLY via unique indexes.


Monitoring

Prometheus Metrics

Every service exposes /metrics on its configured port with four metric types:

Metric Type Description
service_info Info Static service metadata
service_gauge Gauge Point-in-time state (consecutive_failures, last_cycle_timestamp, progress)
service_counter Counter Cumulative totals (cycles_success, cycles_failed, errors by type)
cycle_duration_seconds Histogram Cycle latency with 10 buckets (1s to 1h)

Alert Rules (7)

Alert Condition Severity
ServiceDown up == 0 for 5m critical
HighFailureRate error rate > 0.1/s for 5m warning
ConsecutiveFailures 5+ consecutive cycle failures for 2m critical
SlowCycles p99 cycle duration > 300s for 5m warning
DatabaseConnectionsHigh > 80 active connections for 5m warning
CacheHitRatioLow buffer cache hit ratio < 95% for 10m warning
RefresherViewsFailing view refresh failures for 10m warning

Grafana Dashboard

Auto-provisioned dashboard with per-service panels: cycle duration, error counts, consecutive failures, and service-specific progress metrics.

Structured Logging

info finder cycle_completed relay_count=100 duration=2.5
error validator retry_failed attempt=3 url="wss://relay.example.com"

JSON mode available for cloud aggregation:

{"timestamp": "2026-02-09T12:34:56+00:00", "level": "info", "service": "finder", "message": "cycle_completed", "relay_count": 100}

Nostr Protocol Support

NIPs Implemented

NIP Usage
NIP-01 Event model, relay communication
NIP-11 Relay information document fetch and parse
NIP-42 Relay authentication (Synchronizer auth, Validator detection)
NIP-66 Relay monitoring and discovery (kinds 10166, 22456, 30166)
NIP-89 Handler information (DVM announcement, kind 31990)
NIP-90 Data Vending Machine (DVM job requests/results, kinds 5050/6050)

Event Kinds

Kind Direction Purpose
0 Published Monitor profile metadata
5050 Consumed NIP-90 DVM job request
6050 Published NIP-90 DVM job result
10166 Published Monitor announcement (capabilities, networks, timeouts)
22456 Published NIP-66 ephemeral relay test
30166 Published Relay discovery (addressable, one per relay, health check tags)
31990 Published NIP-89 handler information (DVM announcement)

NIP-66 Health Checks

Check What It Measures Networks
RTT WebSocket open/read/write latency (ms), 3-phase with verification All
SSL Certificate validity, expiry, issuer, SANs, cipher, fingerprint Clearnet
DNS A/AAAA/CNAME/NS/PTR records, TTL Clearnet
Geo Country, city, coordinates, timezone, geohash (GeoLite2 City) Clearnet
Net IP address, ASN, organization, network ranges (GeoLite2 ASN) Clearnet
HTTP Server header, X-Powered-By (from WebSocket handshake) All

Configuration

Environment Variables

Variable Required Description
DB_ADMIN_PASSWORD Yes PostgreSQL admin password
DB_WRITER_PASSWORD Yes Writer role password (Seeder, Finder, Validator, Monitor, Synchronizer)
DB_REFRESHER_PASSWORD Yes Refresher role password (matview ownership)
DB_READER_PASSWORD Yes Reader role password (Api, Dvm, postgres-exporter)
NOSTR_PRIVATE_KEY For Monitor, Validator, Synchronizer, Dvm Nostr private key (hex or nsec) for event signing and NIP-42 auth
GRAFANA_PASSWORD For Grafana Grafana admin password

Configuration Files

deployments/bigbrotr/config/
├── brotr.yaml                  # Pool, batch size, timeouts
└── services/
    ├── seeder.yaml             # Seed file path, validate mode
    ├── finder.yaml             # API sources (JMESPath), event scanning, concurrency
    ├── validator.yaml          # Networks, cleanup, processing chunk size
    ├── monitor.yaml            # Health checks, retry per type, publishing, GeoIP
    ├── synchronizer.yaml       # Networks, filter, time range, per-relay overrides
    ├── refresher.yaml          # View list, refresh interval
    ├── api.yaml                # Host, port, pagination, CORS
    └── dvm.yaml                # NIP-90 kind, relay list, response format

All configs use Pydantic v2 validation with typed defaults and constraints.


Development

Setup

git clone https://github.com/BigBrotr/bigbrotr.git && cd bigbrotr
curl -LsSf https://astral.sh/uv/install.sh | sh  # install uv (one-time)
uv sync --group dev
pre-commit install

Quality Checks

make lint             # ruff check src/ tests/
make format           # ruff format src/ tests/
make typecheck        # mypy src/bigbrotr (strict mode)
make test             # pytest unit tests (~2,737 tests)
make test-integration # pytest integration tests (~216 tests, requires Docker)
make test-fast        # pytest -m "not slow"
make coverage         # pytest --cov with HTML report (80% branch minimum)
make ci               # all checks: lint + format-check + typecheck + test + sql-check + audit
make docs             # build MkDocs documentation site
make docs-serve       # serve docs locally with live reload
make build            # build Python package (sdist + wheel)
make docker-build     # build Docker image (DEPLOYMENT=bigbrotr)
make docker-up        # start Docker stack
make docker-down      # stop Docker stack
make clean            # remove build artifacts and caches

Test Suite

  • ~2,737 unit tests + ~216 integration tests (testcontainers PostgreSQL)
  • asyncio_mode = "auto" — no @pytest.mark.asyncio needed
  • Global timeout: 120s per test
  • Shared fixtures via tests/fixtures/relays.py (registered as pytest plugin)
  • Coverage threshold: 80% (branch coverage enabled)

CI/CD Pipeline

Stage Tool Purpose
Pre-commit ruff, mypy, yamllint, detect-secrets, markdownlint, hadolint, sqlfluff, codespell Code quality gates (23 hooks)
Unit Test pytest (Python 3.11–3.14 matrix) Unit tests + coverage
Integration Test pytest + testcontainers PostgreSQL integration tests
Build Docker Buildx (matrix) Multi-deployment image builds + Trivy scan
Security uv-secure, Trivy, CodeQL Dependency vulns, container scanning, static analysis
Release PyPI (OIDC) + GHCR Package + Docker image publishing, SBOM generation
Docs MkDocs Material Auto-generated API docs deployed to GitHub Pages
Dependencies Dependabot Weekly updates for uv, Docker, GitHub Actions

Project Structure

bigbrotr/
├── src/bigbrotr/                    # Main package
│   ├── __main__.py                  # CLI entry point (service registry)
│   ├── core/                        # Infrastructure
│   │   ├── pool.py                  # asyncpg connection pool with retry/backoff
│   │   ├── brotr.py                 # DB facade (stored procedures, bulk inserts)
│   │   ├── base_service.py          # Abstract service with run_forever loop
│   │   ├── logger.py                # Structured key=value / JSON logging
│   │   ├── metrics.py               # Prometheus metrics server
│   │   └── yaml.py                  # YAML config loader
│   ├── models/                      # Pure frozen dataclasses (zero I/O)
│   │   ├── relay.py                 # URL validation (rfc3986), network detection
│   │   ├── event.py                 # Nostr event wrapper (nostr_sdk.Event)
│   │   ├── metadata.py              # Content-addressed metadata (SHA-256)
│   │   ├── event_relay.py           # Event-relay junction (cascade insert)
│   │   ├── relay_metadata.py        # Relay-metadata junction (cascade insert)
│   │   ├── service_state.py         # Operational state persistence
│   │   ├── constants.py             # NetworkType, ServiceName, EventKind enums
│   │   └── _validation.py           # Shared validation and sanitization
│   ├── nips/                        # NIP protocol implementations (I/O)
│   │   ├── base.py                  # Base data, logs, metadata models
│   │   ├── parsing.py               # Declarative field parsing (FieldSpec)
│   │   ├── event_builders.py        # Kind 0/10166/30166 event construction
│   │   ├── nip11/                   # Relay information document
│   │   └── nip66/                   # Health checks: rtt, ssl, dns, geo, net, http
│   ├── utils/                       # Network primitives
│   │   ├── protocol.py              # Nostr client, relay connection, broadcasting
│   │   ├── transport.py             # Insecure WebSocket transport, stderr filter
│   │   ├── streaming.py             # Event streaming with binary-split windowing
│   │   ├── dns.py                   # Async hostname resolution (A/AAAA)
│   │   ├── keys.py                  # Nostr key loading from environment
│   │   ├── http.py                  # Bounded HTTP response reading
│   │   └── parsing.py               # Tolerant model factory parsing
│   └── services/                    # Business logic
│       ├── seeder/                  # Seed file loading (one-shot)
│       ├── finder/                  # Relay discovery (APIs + event scanning)
│       ├── validator/               # WebSocket protocol validation
│       ├── monitor/                 # Health check orchestration + publishing
│       ├── synchronizer/            # Event collection (cursor-based)
│       ├── refresher/               # Materialized view refresh
│       ├── api/                     # REST API (FastAPI, read-only)
│       ├── dvm/                     # NIP-90 Data Vending Machine
│       └── common/                  # Shared queries, configs, mixins
├── deployments/
│   ├── Dockerfile                   # Single parametric (ARG DEPLOYMENT)
│   ├── bigbrotr/                    # Full archive deployment
│   │   ├── config/                  # YAML configs (brotr + 8 services)
│   │   ├── postgres/init/           # SQL schema (10 files, 25 functions)
│   │   ├── monitoring/              # Prometheus + Alertmanager + Grafana
│   │   └── docker-compose.yaml      # 15 containers, 2 networks
│   └── lilbrotr/                    # Lightweight deployment
├── tests/
│   ├── fixtures/relays.py           # Shared relay fixtures
│   ├── unit/                        # ~2,737 tests (mirrors src/ structure)
│   └── integration/                 # ~216 tests (testcontainers PostgreSQL)
├── docs/                            # MkDocs Material documentation
├── Makefile                         # Development targets
└── pyproject.toml                   # All config: deps, ruff, mypy, pytest, coverage

Docker Infrastructure

Container Stack

Container Image Purpose
postgres postgres:18-alpine Primary storage
pgbouncer edoburu/pgbouncer:v1.25.1-p0 Transaction-mode connection pooling
tor osminogin/tor-simple:0.4.8.10 SOCKS5 proxy for .onion relays
seeder bigbrotr (parametric) Relay bootstrapping (one-shot)
finder bigbrotr (parametric) Relay discovery
validator bigbrotr (parametric) Candidate validation
monitor bigbrotr (parametric) Health monitoring + event publishing
synchronizer bigbrotr (parametric) Event archiving
refresher bigbrotr (parametric) Materialized view refresh
api bigbrotr (parametric) REST API (FastAPI)
dvm bigbrotr (parametric) NIP-90 Data Vending Machine
postgres-exporter prometheuscommunity/postgres-exporter:v0.16.0 PostgreSQL metrics
prometheus prom/prometheus:v2.51.0 Metrics collection (30d retention)
alertmanager prom/alertmanager:v0.27.0 Alert routing and grouping
grafana grafana/grafana:10.4.1 Dashboards

Networks

  • data-network — postgres, pgbouncer, tor, all services
  • monitoring-network — prometheus, grafana, alertmanager, postgres-exporter, all services

Security

  • All ports bound to 127.0.0.1 (no external exposure)
  • Non-root container execution (UID 1000)
  • tini as PID 1 for proper signal handling
  • SCRAM-SHA-256 authentication (PostgreSQL + PGBouncer)
  • Healthchecks via pg_isready and /metrics HTTP endpoint

Technology Stack

Category Technologies
Language Python 3.11+ (fully typed, strict mypy)
Database PostgreSQL 18, asyncpg, PGBouncer
Async asyncio, aiohttp, aiohttp-socks
Nostr nostr-sdk (Rust FFI via UniFFI)
Web Framework FastAPI, uvicorn
Validation Pydantic v2, rfc3986
Monitoring Prometheus, Grafana, Alertmanager, structured logging
Networking dnspython, geoip2, geohash2, tldextract, cryptography
Testing pytest, pytest-asyncio, pytest-cov, testcontainers
Quality ruff (lint+format), mypy (strict), pre-commit (23 hooks)
CI/CD GitHub Actions, uv-secure, Trivy, CodeQL, Dependabot
Containers Docker, Docker Compose, tini
Build uv (dependency management + build)

Documentation

Full documentation is available at bigbrotr.github.io/bigbrotr.

Section Description
Getting Started Installation, quick start tutorial, first deployment
User Guide Architecture, services, configuration, database, monitoring
How-to Guides Docker deploy, manual deploy, Tor setup, troubleshooting
Development Setup, testing, contributing
API Reference Auto-generated Python API docs
Changelog Version history and migration guides

Contributing

See the Contributing Guide for detailed instructions.

  1. Fork and clone
  2. uv sync --group dev and pre-commit install
  3. Write tests for new functionality
  4. make ci — all checks must pass
  5. Submit a pull request

Conventional commits: feat:, fix:, refactor:, docs:, test:, chore:


License

MIT — see LICENSE.


Links

About

Modular Nostr network observatory — relay discovery, health monitoring, event archiving, analytics, and data access

Topics

Resources

License

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Packages

 
 
 

Contributors