RHDEVDOCS-3611 - configure-request-logging-for-thanos-querier

Author: bburt-rh

modules/monitoring-enabling-query-logging-for-thanos-querier.adoc

@@ -0,0 +1,90 @@
+// Module included in the following assemblies:
+//
+// * monitoring/configuring-the-monitoring-stack.adoc
+
+[id="enabling-query-logging-for-thanos-querier_{context}"]
+:_content-type: PROCEDURE
+= Enabling query logging for Thanos Querier 
+
+[role="_abstract"]
+For default platform monitoring in the `openshift-monitoring` project, you can enable the Cluster Monitoring Operator to log all queries run by Thanos Querier.
+
+[IMPORTANT]
+====
+Because log rotation is not supported, only enable this feature temporarily when you need to troubleshoot an issue. After you finish troubleshooting, disable query logging by reverting the changes you made to the `ConfigMap` object to enable the feature.
+====
+
+.Prerequisites
+
+* You have installed the OpenShift CLI (`oc`).
+* You have access to the cluster as a user with the `cluster-admin` role.
+* You have created the `cluster-monitoring-config` `ConfigMap` object.
+
+.Procedure
+
+You can enable query logging for Thanos Querier in the `openshift-monitoring` project:
+
+. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` project:
++
+[source,terminal]
+----
+$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
+----
++
+. Add a `thanosQuerier` section under `data/config.yaml` and add values as shown in the following example:
++
+[source,yaml]
+----
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cluster-monitoring-config
+  namespace: openshift-monitoring
+data:
+  config.yaml: |
+    thanosQuerier:
+      enableRequestLogging: <value> <1>
+      logLevel: <value> <2>
+      
+
+----
+<1> Set the value to `true` to enable logging and `false` to disable logging. The default value is `false`.
+<2> Set the value to `debug`, `info`, `warn`, or `error`. If no value exists for `logLevel`, the log level defaults to `error`.
++
+. Save the file to apply the changes.
++
+[WARNING]
+====
+When you save changes to a monitoring config map, pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.
+====
+
+.Verification
+
+. Verify that the Thanos Querier pods are running. The following sample command lists the status of pods in the `openshift-monitoring` project:
++
+[source,terminal]
+----
+$ oc -n openshift-monitoring get pods
+----
++
+. Run a test query using the following sample commands as a model:
++
+[source,terminal]
+----
+$ token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
+$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?query=cluster_version'
+----
+. Run the following command to read the query log:
++
+[source,terminal]
+----
+$ oc -n openshift-monitoring logs <thanos_querier_pod_name> -c thanos-query
+----
++
+[NOTE]
+====
+Because the `thanos-querier` pods are highly available (HA) pods, you might be able to see logs in only one pod.
+====
++
+. After you examine the logged query information, disable query logging by changing the `enableRequestLogging` value to `false` in the config map.
+

monitoring/configuring-the-monitoring-stack.adoc

@@ -139,7 +139,6 @@ include::modules/monitoring-attaching-additional-labels-to-your-time-series-and-
 
 * See xref:../monitoring/configuring-the-monitoring-stack.adoc#preparing-to-configure-the-monitoring-stack[Preparing to configure the monitoring stack] for steps to create monitoring config maps
 * xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects]
-* See xref:../monitoring/configuring-the-monitoring-stack.adoc#preparing-to-configure-the-monitoring-stack[Preparing to configure the monitoring stack] for steps to create monitoring config maps
 
 // Setting log levels for monitoring components
 include::modules/monitoring-setting-log-levels-for-monitoring-components.adoc[leveloffset=+1]
@@ -153,6 +152,11 @@ include::modules/monitoring-setting-query-log-file-for-prometheus.adoc[leveloffs
 * See xref:../monitoring/configuring-the-monitoring-stack.adoc#preparing-to-configure-the-monitoring-stack[Preparing to configure the monitoring stack] for steps to create monitoring config maps
 * See xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects] for steps to enable user-defined monitoring.
 
+// Enabling query logging for Thanos Querier
+include::modules/monitoring-enabling-query-logging-for-thanos-querier.adoc[leveloffset=+1]
+
+* See xref:../monitoring/configuring-the-monitoring-stack.adoc#preparing-to-configure-the-monitoring-stack[Preparing to configure the monitoring stack] for steps to create monitoring config maps.
+
 // Disabling the default Grafana deployment
 include::modules/monitoring-disabling-grafana.adoc[leveloffset=+1]