Skip to content

Ingest target PostgreSQL logs as telemetry — errors, spills, autovacuum runs, and connection churn are invisible between counter samples #3601

Closed

Description

@erikdarlingdata

The PostgreSQL log is the engine's primary event record — errors, cancelled statements, temp-file spills, autovacuum runs, connection churn, and lock-wait reports all land there and nowhere else. Darling currently reads target logs for exactly two families: auto_explain plans (PgPlanCaptureCollector / RdsPlanIngestor) and deadlocks (PgDeadlocksCollector / RdsDeadlockIngestor). Everything else in the log is invisible, and the view-based collectors can only see what survives into a cumulative counter — a burst of FATAL connection failures, a 40-second statement cancelled by a client, or a temp-file storm between two samples leaves either a bare counter increment or nothing.

For the SQL Server DBA audience this is the sharpest remaining gap: their instinct of "check the error log" has no Darling answer today, and pganalyze's guide for that audience (https://pganalyze.com/blog/postgres-monitoring-for-sql-server-dbas) treats log workflows as a primary pillar.

The expensive part already ships. Both transports exist and run in production — pg_read_file tailing for self-hosted targets and the RDS/Aurora log API for managed ones — with cursoring, timezone handling (PgLogTimezoneUnsupportedException), and per-family parser precedent (PgDeadlockLogParser, PgPlanLogParser). What this issue proposes is generalizing that machinery into a log-event pipeline: a classifier that walks the same tailed stream and fans lines into event families (error/severity events first; sibling issues cover temp-file and autovacuum families), a hypertable for classified events with the store's usual retention/compression discipline, and a read surface (get_pg_log_events) with severity/family/time filters.

Scope note: statement literals in error context lines must get the same redaction discipline PgPlanLogParser applies to plan capture — the log pipeline must not become the place parameter values leak into the store.

Sibling increments filed separately (each can ride this pipeline independently, as deadlocks already prove): per-event temp spills, autovacuum run costs, and a logging-settings audit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions