Each warning is logged twice and each error four times, and there is no way to reduce log volume

[dberkerdem]

Summary

coroot-node-agent writes every warning to stderr twice and every error four times, and there is no supported way to reduce its log volume. On nodes with a lot of process churn the agent's own logs become the largest log producer on the host.

Both issues are in the same few lines of main.go, and both have a small fix.

1. Each message is written more than once

main.go sets up logging like this:

klog.LogToStderr(false)
klog.SetOutput(&RateLimitedLogOutput{limiter: rate.NewLimiter(rate.Limit(*flags.LogPerSecond), *flags.LogBurst)})

klog.SetOutput points every severity output at the same writer. klog's output() then writes a message to the output of its own severity and to the output of every lower severity, and separately copies anything at or above -stderrthreshold (default ERROR) straight to os.Stderr.

Since all the severity outputs are the same writer, one emission becomes:

Severity	stderr lines	Source
Info	1	INFO output
Warning	2	WARNING + INFO outputs
Error	4	direct stderr copy + ERROR + WARNING + INFO outputs

Reproduction

Self-contained, mirrors the agent's setup exactly:

package main

import (
	"flag"
	"io"
	"os"

	"k8s.io/klog/v2"
)

type passthrough struct{}

func (passthrough) Write(p []byte) (int, error) { return os.Stderr.Write(p) }

func main() {
	fs := flag.NewFlagSet("klog", flag.ContinueOnError)
	fs.SetOutput(io.Discard)
	klog.InitFlags(fs)

	klog.LogToStderr(false)
	klog.SetOutput(passthrough{})

	klog.Info("INFO_LINE")
	klog.Warning("WARN_LINE")
	klog.Error("ERROR_LINE")
	klog.Flush()
}

$ go run . 2>&1 >/dev/null | cut -c1 | sort | uniq -c
   4 E
   1 I
   2 W

Expected 1 E, 1 I, 1 W.

This is also visible in the wild — for example the duplicated lines in #214 carry identical microsecond timestamps, which reads like the agent looping when it is actually a single emission written twice.

A side effect is that the extra copies each consume a separate rate limiter token, so one warning costs 2 tokens and one error costs 3. And because the direct stderr copy at or above -stderrthreshold does not pass through RateLimitedLogOutput at all, --log-per-second can never throttle errors.

Fix

Enabling klog's -one_output and moving -stderrthreshold to FATAL makes each message be written exactly once:

   1 E
   1 I
   1 W

2. No way to reduce log volume

The bulk of the output on a busy node is per-process INFO from containers/registry.go — calculated container id ..., ignoring ..., skipping system service ... — emitted as processes come and go.

There is currently no way to turn this down:

• -v / -vmodule only gate klog.V(n) calls, and the agent has no klog.V( call sites on the Linux path, so verbosity controls nothing.
• -stderrthreshold controls which severities are additionally copied to stderr, not which are suppressed.
• klog.InitFlags is never called, so none of those flags are exposed on the command line anyway.
• --log-per-second is severity-blind: turning it down to suppress the INFO noise discards warnings and errors just as readily, and as noted above it cannot throttle errors at all.

Fix

A --log-level flag (LOG_LEVEL) that routes severities below the threshold to io.Discard via klog.SetOutputBySeverity, so they are dropped before reaching the rate limiter. Defaulting it to info keeps current behaviour.

Proposed change

I have both fixes ready and will open a PR shortly: -one_output + -stderrthreshold=FATAL for the duplication, and --log-level for the volume. Happy to split them or adjust the flag naming if you'd prefer a different shape.

Versions checked: v1.33.3 through v1.35.8 (current), k8s.io/klog/v2 v2.130.1. The four call sites and the logging setup are unchanged across that range.