Skip to content

Commit 39b3f53

Browse files
committed
Alert NoOutputBytesProcessed should combine outputs
We were seeing issues where we got alerted due to some outputs not sending data for more than 15 minutes. For example, this is expected from journald logs on worker nodes in a cluster, thus this adjustment to the alert. We now aggregate output per pod and alert if no bytes are sent for 15 minutes. Signed-off-by: Victor Sollerhed <victor.sollerhed@pagero.com>
1 parent 00d6b84 commit 39b3f53

File tree

1 file changed

+2
-3
lines changed

1 file changed

+2
-3
lines changed

monitoring/alerts.yaml

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,10 @@ groups:
22
- name: fluent-bit
33
rules:
44
- alert: NoOutputBytesProcessed
5-
expr: rate(fluentbit_output_proc_bytes_total[5m]) == 0
5+
expr: sum(rate(fluentbit_output_proc_bytes_total[5m])) by (pod) == 0
66
annotations:
77
message: |
8-
Fluent Bit instance {{ $labels.instance }}'s output plugin {{ $labels.name }} has not processed any
9-
bytes for at least 15 minutes.
8+
Fluent Bit pod {{ $labels.pod }} has not processed any output bytes for at least 15 min.
109
summary: No Output Bytes Processed
1110
for: 15m
1211
labels:

0 commit comments

Comments
 (0)