update documentation and changes

CHANGES.md

@@ -15,7 +15,32 @@
 [Unreleased](https://github.com/bird-house/birdhouse-deploy/tree/master) (latest)
 ------------------------------------------------------------------------------------------------------------------
 
-[//]: # (list changes here, using '-' for each new entry, remove this when items are added)
+## Changes
+
+- Add the `prometheus-log-parser` optional component
+
+  This component parses log files from other components and converts their logs to prometheus
+  metrics that are then ingested by the monitoring Prometheus instance (the one created by the
+  `components/monitoring` component).
+
+  For more information on how this component reads log files and converts them to prometheus components see
+  the [log-parser](https://github.com/DACCS-Climate/log-parser/) documentation.
+
+  To configure this component:
+
+  * set the `PROMETHEUS_LOG_PARSER_POLL_DELAY` variable to a number of seconds to set how often the log parser
+    checks if new lines have been added to log files (default: 1)
+  * set the `PROMETHEUS_LOG_PARSER_TAIL` variable to `"true"` to only parse new lines in log files. If unset,
+    this will parse all existing lines in the log file as well (default: `"true"`)
+
+  To view all metrics exported by the log parser:
+
+  * Navigate to the `https://<BIRDHOUSE_FQDN>/prometheus/graph` search page
+  * Put `{job="log_parser"}` in the search bar and click the "Execute" button
+
+- Update the prometheus version to the current latest `v2.53.3`. This is required to support   
+  loading multiple prometheus scrape configuration files with the `scrape_config_files`
+  configuration option.
 
 [2.5.4](https://github.com/bird-house/birdhouse-deploy/tree/2.5.4) (2024-10-31)
 ------------------------------------------------------------------------------------------------------------------

birdhouse/env.local.example

@@ -632,6 +632,10 @@ export THREDDS_ADDITIONAL_CATALOG=''
 #export ALERTMANAGER_EXTRA_INHIBITION=""
 #export ALERTMANAGER_EXTRA_RECEIVERS=""
 
+# Below are for the prometheus-log-parser optional component
+#export PROMETHEUS_LOG_PARSER_POLL_DELAY=1 # time in seconds
+#export PROMETHEUS_LOG_PARSER_TAIL=t
+
 #############################################################################
 # Emu optional vars
 #############################################################################

birdhouse/optional-components/README.rst

@@ -443,3 +443,39 @@ How to enable X-Robots-Tag Header in ``env.local`` (a copy from `env.local.examp
 
     .. seealso::
         See the `env.local.example`_ file for more details about this ``BIRDHOUSE_PROXY_ROOT_LOCATION`` behaviour.
+
+.. _prometheus-log-parser
+
+Prometheus Log Parser
+---------------------
+
+Parses log files from other components and converts their logs to prometheus metrics that are then ingested by the
+monitoring Prometheus instance (the one created by the ``components/monitoring`` component).
+
+For more information on how this component reads log files and converts them to prometheus components see
+the log-parser_ documentation.
+
+To configure this component:
+
+    * set the ``PROMETHEUS_LOG_PARSER_POLL_DELAY`` variable to a number of seconds to set how often the log parser
+      checks if new lines have been added to log files (default: 1)
+    * set the ``PROMETHEUS_LOG_PARSER_TAIL`` variable to ``"true"`` to only parse new lines in log files. If unset,
+      this will parse all existing lines in the log file as well (default: ``"true"``)
+
+To view all metrics exported by the log parser:
+
+    * Navigate to the ``https://<BIRDHOUSE_FQDN>/prometheus/graph`` search page
+    * Put ``{job="log_parser"}`` in the search bar and click the "Execute" button
+
+For developers, to create a new parser that can be used to track log files:
+
+    1. create a python file that is can be mounted as a volume to the ``PROMETHEUS_LOG_PARSER_PARSERS_DIR``
+       directory on the ``prometheus-log-parser`` container.
+    2. mount any log files that you want to parse as a volume on the ``prometheus-log-parser`` container.
+    3. the python script must contain a variable named ``LOG_PARSER_CONFIG`` which is a
+       dictionary where keys are paths to log files and values are a list of "line parser" functions.
+       * a "line parser" is any function that takes a string as a single argument (a single line from a
+         log file). These functions are where you'd write the code that parses the line and converts it
+         into a prometheus metric.
+
+.. _log-parser: https://github.com/DACCS-Climate/log-parser/