Add built-in performance monitoring and observability tools

[mre]

Summary

Add built-in performance monitoring capabilities to provide deep insights into job processing performance and system behavior.

Motivation

Production systems need comprehensive observability to identify bottlenecks, optimize performance, and troubleshoot issues. While we have good error handling, we lack detailed performance insights.

Proposed Features

Performance Tracking

• Job execution time histograms
• Queue wait time measurements
• Database query performance tracking
• Worker pool efficiency metrics

Built-in Observability

• Performance dashboard/summary functions
• Slow job detection and alerting
• Database connection health monitoring
• Memory usage tracking per worker

Integration Points

• Optional Prometheus metrics export
• Structured logging with performance data
• Health check endpoints for load balancers
• Integration with existing tracing/sentry setup

Performance Analysis Tools

• Job performance profiling by type
• Bottleneck identification (DB, CPU, I/O)
• Worker scaling recommendations
• Queue configuration optimization hints

Implementation Ideas

Core APIs

// Get performance summary
let perf = runner.get_performance_summary().await?;
println!(\"Average job time: {}ms\", perf.avg_job_time_ms);

// Enable detailed monitoring
.configure_queue(\"high_perf\", |queue| {
    queue
        .enable_performance_monitoring(true)
        .slow_job_threshold(Duration::from_secs(30))
        .prometheus_metrics(true)
})

Monitoring Features

• Real-time performance metrics
• Historical performance trends (via archive data)
• Automated performance regression detection
• Resource utilization tracking

Benefits

• Proactive performance optimization
• Faster troubleshooting of production issues
• Data-driven scaling decisions
• Better understanding of system behavior under load

Inspired By

HN discussion emphasizing the importance of monitoring processing time and worker utilization, plus the need for robust observability in production job processing systems.

[M3t0r]

Maybe add a native field for tracing contexts (trace ID + parent span)?

Also, there is a difference between monitoring

1. a job
2. a worker
3. the whole system

I think adding tracing support for job's would cover 1), and simple prometheus metric collection would cover 2). For 3) you'd have to write a dedicated exporter that can read the database and generate metrics from the metadata fields. A playground for 3) could be the TUI mentioned in #12 .