Skip to content

cluster_status.blobReplicationFailures always reports 0 for copy-mode blob failures (observability gap) #502

Description

@kriszyp

Summary

During a full-copy resync with source-unavailable blobs, recordBlobReplicationFailure() fires (observed 14× in the repro) and the receiver logs advancing the resume cursor past it (7×), but cluster_status on both nodes reports blobReplicationFailures: 0. The log message itself cites cluster_status.blobReplicationFailures as the authoritative cumulative total — so an operator monitoring that field would never see blob-replication failures that ARE happening.

Likely cause

The auditStore lookup in clusterStatus.ts:47-52 finds nothing at query time (or reads a freshly-zeroed shared-buffer slot), so the metric-augmentation block is skipped. (harper-pro 282a0bc)

Severity

Low-medium — operator-facing observability gap, not data integrity. (The underlying advance-past behavior is separately tracked; this issue is specifically that the metric stays 0.)

Repro

integrationTests/cluster/qa-scratch/qa339-blob-resync-wedge.test.mjs (from QA-339).

— from Harper exploratory QA (KrAIs)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions