Skip to content

Report external logging queue statistics to the container log - #1098

Open
quanah wants to merge 1 commit into
ctrliq:mainfrom
quanah:issue1097
Open

quanah wants to merge 1 commit into
ctrliq:mainfrom
quanah:issue1097

Conversation

@quanah

@quanah quanah commented Oct 8, 2026

Copy link
Copy Markdown
SUMMARY

Assisted-by: Claude Opus 5.5. Drafted with AI assistance; the commit carries the same trailer.

Option 2 from #1097: an opt-in setting that routes rsyslog's own queue counters into the container log, where any log collector already picks them up and a monitor can alert on discards.

  • New setting LOG_AGGREGATOR_ACTION_QUEUE_STATS_INTERVAL: seconds, default 0 (off).
  • Above 0, with external logging enabled ⇒ generated rsyslog.conf gains:
    module(load="impstats" interval="<N>" format="json" resetCounters="off" ruleset="ascender_queue_stats")
    ruleset(name="ascender_queue_stats") { action(type="omfile" file="/dev/stdout") }
    
  • Own ruleset ⇒ statistics never reach the external action, which stays in the default ruleset.
  • rsyslogd's stdout is already the container log (stdout_logfile=/dev/stdout in both supervisor configs) ⇒ no new files or volumes.
  • impstats ships in the base rsyslog package the image installs ⇒ no new dependency.
  • Default 0 ⇒ generated configuration byte-identical to today.
  • Independent of the conf.d include fix (The generated rsyslog.conf no longer reads /var/lib/ascender/rsyslog/conf.d #1091): it needs no operator drop-in.

Same change upstream: ansible/awx#16718 (for ansible/awx#16717).

related #1097

ISSUE TYPE
  • New or Enhanced Feature
COMPONENT NAME
  • API
ASCENDER VERSION
awx: 25.6.2-235-g640cf5f2
ADDITIONAL INFORMATION

Tests: test_rsyslog_conf_queue_stats in ascender/main/tests/unit/api/test_logger.py, parametrized over the registered default, 0, 60, and 60 with external logging disabled (no queue ⇒ no statistics). Run in python:3.14 with Ascender's requirements, against main 640cf5f2, with test_external_logging_durability.py alongside:

# with this change
34 passed

# main with only the test changes
1 failed, 33 passed    (the interval=60 case; default, 0 and disabled pass on main, as they must)

End to end, on rockylinux/rockylinux:9 with rsyslog-8.2510.0-2.el9_8.2: the rsyslog.conf the patched code generates for a TCP destination on a closed port, LOG_AGGREGATOR_ACTION_QUEUE_SIZE=100 and an interval of 1 s; 300 INFO messages sent through the Ascender socket:

$ rsyslogd -N1 -f rsyslog.conf
rsyslogd: End of config validation run. Bye.        (rc=0)

stdout: 64 lines, 64 impstats, 0 job events
rsyslogd-pstats: { "name": "action-1-builtin:omfwd queue", "origin": "core.queue", "size": 90, "enqueued": 300, "full": 0, "discarded.full": 0, "d
iscarded.nf": 300, "maxqsize": 90 }

enqueued 300 = only the job events reached the external queue; the statistics stayed in their own ruleset. Same result with the other external logging fixes applied underneath (conf.d include, keepalive, discard severity, RELP).

discarded.nf reads 300 with 90 messages still queued. The same test through AWX's datagram socket reads 210 (300 − 90), so the difference comes from rsyslog's batched imptcp path, not from this change; either way the counter is non-zero exactly when the queue discards.

ruff check and ruff format --check (ruff 0.16.10) are clean on the touched files.

rsyslog counts every message an action queue discards (discarded.nf
when the queue is nearly full, discarded.full when it is full), but
Ascender never loads impstats, so nothing reports those counters and a
deployment can lose external log events with no sign of it.

Register LOG_AGGREGATOR_ACTION_QUEUE_STATS_INTERVAL (seconds, default
0 = off). When it is above 0 and external logging is enabled, the
generated rsyslog.conf loads impstats at that interval, in JSON, bound
to its own ruleset whose only action writes to rsyslogd's stdout,
which is the container log. The separate ruleset keeps the statistics
out of the external action in the default ruleset. With the default,
the generated configuration is unchanged.

related ctrliq#1097

Assisted-by: Claude Opus 5.5
Signed-off-by: Quanah Gibson-Mount <quanah@dropbox.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant