Zabbix Process Execution Trace Analyzer

A Python tool to analyze Zabbix server logs and identify execution gaps and potential stalls by tracking time delays between consecutive log entries.

Features

• Parses Zabbix server log format
• Calculates time gaps between consecutive log entries for each process (by PID)
• Sorts gaps from highest to lowest duration
• Identifies potential stalls and performance issues
• Supports filtering by minimum gap duration
• Can read from file or stdin

Requirements

• Python 3.6 or higher
• No external dependencies (uses only standard library)

Usage

Basic Usage

# Analyze a log file
./zabbix_trace_analyzer.py zabbix.log

# Analyze the sample log
./zabbix_trace_analyzer.py sample_zabbix.log

Show Only Top N Gaps

# Show only top 10 gaps
./zabbix_trace_analyzer.py zabbix.log --limit 10

Filter by Minimum Gap Duration

# Show only gaps longer than 5 seconds
./zabbix_trace_analyzer.py zabbix.log --min-gap 5

# Show only gaps longer than 1 minute (60 seconds)
./zabbix_trace_analyzer.py zabbix.log --min-gap 60

Read from stdin

# Pipe log content
cat zabbix.log | ./zabbix_trace_analyzer.py

# Use with grep to filter specific processes
grep "zabbix_server\[20741\]" zabbix.log | ./zabbix_trace_analyzer.py

Specify Year

# Specify year for timestamp parsing (defaults to current year)
./zabbix_trace_analyzer.py zabbix.log --year 2024

Combined Options

# Show top 5 gaps that are at least 10 seconds long
./zabbix_trace_analyzer.py zabbix.log --limit 5 --min-gap 10

Output Format

The tool generates a report showing:

1. Summary Statistics

   • Total entries analyzed
   • Total gaps detected

2. Gap Details (sorted by duration, descending)

   • Gap number and formatted duration
   • Exact duration in seconds
   • Start and end timestamps
   • Log lines before and after the gap

Example Output

================================================================================
ZABBIX PROCESS EXECUTION GAP ANALYSIS
================================================================================

Total entries analyzed: 12
Total gaps detected: 4
Showing top 4 gaps:

================================================================================

Gap #1: 1m 16.0s
  Duration: 76.00 seconds
  Start:    2025-10-28 07:23:44
  End:      2025-10-28 07:25:00

  Before gap:
    Oct 28 07:23:44 zabbixServer2 zabbix_server[20741]: End of zbx_db_flush_trends()

  After gap:
    Oct 28 07:25:00 zabbixServer2 zabbix_server[20741]: zbx_setproctitle() title:'history syncer #70 [processed 905 values, 794 triggers in 96.222454 sec, syncing history]'

--------------------------------------------------------------------------------

Log Format

The tool expects Zabbix server logs in this format:

Oct 28 07:23:24 zabbixServer2 zabbix_server[20741]: In DCmass_prepare_history() history_num:905

Format breakdown:

• Oct 28 07:23:24 - Timestamp (without year)
• zabbixServer2 - Hostname
• zabbix_server - Process name
• [20741] - Process ID (PID)
• Message after colon

How It Works

1. Parse: Reads log file and extracts timestamp, hostname, process, PID, and message
2. Group: Groups log entries by PID
3. Calculate: Computes time difference between consecutive entries for the same PID
4. Sort: Orders gaps by duration (largest first)
5. Report: Displays formatted output with context

Use Cases

• Performance Analysis: Identify where Zabbix processes are spending the most time
• Stall Detection: Find unexpected delays in process execution
• Troubleshooting: Correlate gaps with system issues or resource constraints
• Capacity Planning: Understand typical execution patterns and bottlenecks

Files

• zabbix_trace_analyzer.py - Main analyzer script
• sample_zabbix.log - Sample log file for testing
• README.md - This file

Help

./zabbix_trace_analyzer.py --help