Add Prometheus Metrics Exporter for Celery

[berggren]

Summary:

This PR integrates a Prometheus metrics exporter to collect and expose performance data related to Celery tasks.

Technical Details:

• New Metrics Exporter: A new Prometheus exporter has been implemented in src/metrics/exporter.py. This exporter leverages the prometheus_client library to collect various metrics related to Celery tasks, such as task sent, received, started, succeeded, failed, rejected, revoked, and retried counts, as well as task runtime. The exporter exposes these metrics on port 8080, allowing Prometheus to scrape them.
• Metrics Implementation: The exporter uses counters and histograms to track task events and runtime. It extracts relevant information from Celery events, such as task name, hostname, and runtime, to provide detailed metrics. Labels are used effectively to categorize metrics by task name and hostname, enabling granular monitoring. Error handling for failed tasks is planned but not yet implemented.
• Celery Integration: The exporter.py script establishes a connection to the Celery broker and listens for task and worker events. It processes these events to update the Prometheus metrics accordingly.
• Dockerfile for Metrics Exporter: A new Dockerfile has been created in src/metrics/Dockerfile to build and run the metrics exporter as a standalone container. This allows for independent deployment and scaling of the metrics exporter.
• API Endpoint and Routing: A new API endpoint /metrics has been added and routed to the metrics exporter. This endpoint serves the Prometheus metrics, allowing them to be scraped by Prometheus.
• Dependency Management: The prometheus-client package has been added as a dependency in pyproject.toml and poetry.lock files.