Skip to content

Audit log pruning only scheduled on last worker thread; watchdog-restart cycles prevent cleanup #1164

Description

@kriszyp

Summary

Audit log cleanup is only scheduled on the last worker thread. In crash-loop scenarios where the watchdog repeatedly kills and restarts workers, the cleanup task is never reached, causing audit history to accumulate well beyond the configured retention window.

Observed behavior

12 days of audit history retained on a node configured for a 3-day retention window, during a period of repeated watchdog restarts.

Root cause

The last-worker scheduling pattern (if thread === lastThread) means the cleanup job depends on the longest-lived thread surviving long enough to execute. Under watchdog-restart cycles this condition is never met.

Impact

Unbounded audit log growth during crash-loop periods; can compound the OOM / disk pressure problems already present during recovery.

Suggested fix

Schedule audit pruning on a dedicated interval that survives worker thread recycling (e.g., use a cluster-scoped timer or schedule on thread 0 / main thread rather than the last worker).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions