openshift · briandooley · Jul 29, 2026 · Jul 28, 2026
diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml
@@ -167,15 +167,14 @@ Topics:
 #  File: logging-node-selectors
 #- Name: Using tolerations to control logging pod placement
 #  File: logging-taints-tolerations
-#---
-#Name: Troubleshooting logging
-#Dir: troubleshooting
-#Topics:
-#- Name: Viewing Logging status
-#  File: cluster-logging-cluster-status
-#- Name: Troubleshooting log forwarding
-#  File: log-forwarding-troubleshooting
-#- Name: Troubleshooting logging alerts
-#  File: troubleshooting-logging-alerts
-#- Name: Viewing Elasticsearch log store status
-#  File: cluster-logging-log-store-status
+---
+Name: Troubleshoot
+Dir: troubleshooting
+Distros: openshift-logging
+Topics:
+- Name: Troubleshoot the installation
+  File: troubleshoot-installation
+- Name: Troubleshoot log forwarding
+  File: troubleshoot-log-forwarding
+- Name: Troubleshoot logging alerts
+  File: troubleshoot-logging-alerts
diff --git a/configuring/configuring-the-log-store.adoc b/configuring/configuring-the-log-store.adoc
@@ -58,7 +58,7 @@ include::modules/loki-zone-aware-replication.adoc[leveloffset=+2]
 
 include::modules/loki-zone-fail-recovery.adoc[leveloffset=+2]
 
-include::modules/loki-rate-limit-errors.adoc[leveloffset=+2]
+include::modules/troubleshoot-loki-rate-limit-errors.adoc[leveloffset=+2]
 
 [id="loki-network-policies-for-added-security_{context}"]
 == Loki network policies for added security

diff --git a/configuring/tuning-the-log-store.adoc b/configuring/tuning-the-log-store.adoc
@@ -34,7 +34,7 @@ include::modules/loki-zone-aware-replication.adoc[leveloffset=+2]
 
 include::modules/loki-zone-fail-recovery.adoc[leveloffset=+2]
 
-include::modules/loki-rate-limit-errors.adoc[leveloffset=+2]
+include::modules/troubleshoot-loki-rate-limit-errors.adoc[leveloffset=+2]
 
 //Loki network policies for added security
 include::modules/about-loki-network-security.adoc[leveloffset=+1]

diff --git a/installing/installing-the-red-hat-openshift-logging-operator.adoc b/installing/installing-the-red-hat-openshift-logging-operator.adoc
@@ -23,13 +23,13 @@ include::modules/enabling-the-logs-tab.adoc[leveloffset=+1]
 
 include::modules/verifying-logging-installation.adoc[leveloffset=+1]
 
-include::modules/troubleshooting-logging-installation.adoc[leveloffset=+1]
-
 [role="_additional-resources"]
 .Additional resources
 
 * xref:../installing/installing-the-loki-operator.adoc#installing-the-loki-operator[Installing the {loki-op}]
 
+* xref:../troubleshooting/troubleshoot-installation.adoc#troubleshoot-installation[Troubleshoot the installation]
+
 * link:https://docs.redhat.com/en/documentation/red_hat_openshift_logging/{logging-major-version}/html/configuring_logging/configuring-log-forwarding[Configuring log forwarding]
 
 * link:https://docs.redhat.com/en/documentation/red_hat_openshift_cluster_observability_operator/{coo-version}/html/installing_red_hat_openshift_cluster_observability_operator/installing-the-cluster-observability-operator[Installing the {coo-full}]

diff --git a/modules/troubleshoot-installation-symptoms.adoc b/modules/troubleshoot-installation-symptoms.adoc
@@ -0,0 +1,173 @@
+// Module included in the following assemblies:
+//
+// * troubleshooting/troubleshoot-installation.adoc
+
+:_mod-docs-content-type: REFERENCE
+[id="troubleshoot-installation-symptoms_{context}"]
+[id="troubleshooting-installation-symptoms_{context}"]
+= Troubleshoot installation symptoms
+
+[role="_abstract"]
+Use this reference to diagnose common installation issues. Many installation failures occur when cluster prerequisites are not met before operator installation.
+
+ResolutionFailed on the Loki Operator subscription::
++
+*Symptom*: The Loki Operator subscription shows `ResolutionFailed` status.
++
+*Cause*: Incorrect channel name or source catalog.
++
+*Resolution*:
++
+--
+. Verify available channels by running the following command:
++
+[source,terminal]
+----
+$ oc get packagemanifest loki-operator -n openshift-marketplace
+----
+
+. Ensure the subscription uses `source: redhat-operators` (not the community catalog).
+--
+
+LokiStack pods stuck in Pending state::
++
+*Symptom*: After creating the `LokiStack` custom resource, pods remain in `Pending` state and never start.
++
+*Cause*: No `StorageClass` exists, or the specified `StorageClass` does not support dynamic provisioning.
++
+*Common on*: {sno} deployments and edge clusters.
++
+*Resolution*:
++
+--
+. Verify that a `StorageClass` exists by running the following command:
++
+[source,terminal]
+----
+$ oc get storageclass
+----
++
+If no `StorageClass` is listed, you must configure storage before `LokiStack` can deploy.
+
+. For {sno} or edge deployments without dynamic storage, see "Configuring local storage for single-node and edge deployments".
+
+. For multi-node clusters, see "Verifying StorageClass availability".
+--
+
+LokiStack shows Ready but logs are not stored::
++
+*Symptom*: The `LokiStack` custom resource shows `Ready: True`, operators are installed, but logs disappear and are not stored or queryable.
++
+*Cause*: Object storage is not configured or the object storage secret is incorrect.
++
+*Resolution*:
++
+--
+`LokiStack` requires TWO separate types of storage:
+
+* **Block storage (StorageClass)**: For internal PVCs
+* **Object storage (S3-compatible)**: For actual log data
+
+See "Configuring object storage for LokiStack" to configure the S3-compatible backend and create the storage secret.
+--
+
+Logs are silently dropped with no error::
++
+*Symptom*: The `LokiStack` shows `Ready: True`, operators are installed, log collector pods are running, but logs are silently dropped and never appear in queries.
++
+*Cause*: Missing TLS CA bundle for object storage, or incorrect object storage permissions.
++
+*Resolution*:
++
+--
+. If using self-signed certificates or private CA for object storage, see "Configuring object storage for LokiStack" to configure the CA bundle.
+
+. Verify that the object storage credentials have all required S3 permissions. See "Object storage permissions requirements".
+--
+
+Certificate verify failed error in collector logs::
++
+*Symptom*: The `ClusterLogForwarder` custom resource shows `Ready: True`, but no logs appear in `LokiStack` or other HTTP-based sinks. The collector logs show the following error:
++
+[source,terminal]
+----
+error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2123:: self-signed certificate in certificate chain
+----
++
+*Cause*: The `tls.ca` block is missing or incorrect in the `ClusterLogForwarder` output configuration.
++
+*Resolution*:
++
+--
+Add the `tls.ca` block to your `ClusterLogForwarder` output:
+
+[source,yaml]
+----
+spec:
+  outputs:
+  - name: default-lokistack
+    type: lokiStack
+    tls:
+      ca:
+        configMapName: logging-loki-gateway-ca-bundle
+        key: service-ca.crt
+----
+
+The Loki Operator creates the `logging-loki-gateway-ca-bundle` `ConfigMap` automatically. For self-signed certificates on external outputs, reference your custom CA `ConfigMap`.
+--
+
+429 Too Many Requests in collector logs::
++
+*Symptom*: The collector logs show `429 Too Many Requests` responses and the collector retries requests.
++
+*Cause*: The collector is sending logs faster than the `LokiStack` or destination sink can process them.
++
+*Common during*:
++
+--
+* Initial deployment when the collector flushes a backlog of accumulated logs
+* After restoring connectivity to a `LokiStack` that was temporarily unavailable
+--
++
+*Resolution*:
++
+--
+This condition resolves automatically. The collector retries with exponential back-off. On a `1x.demo` `LokiStack`, the backlog typically clears within several minutes. Larger `LokiStack` sizes clear backlogs faster.
+--
+
+No Logs tab under Observe::
++
+*Symptom*: After installing operators and creating `LokiStack`, the *Logs* tab does not appear under *Observe* in the web console.
++
+*Cause*: The {coo-full} is not installed, or the `UIPlugin` custom resource was not created.
++
+*Resolution*:
++
+--
+Both steps are required to enable the web console logs interface:
+
+. See "Installing the Cluster Observability Operator".
+
+. After the operator is installed, create the `UIPlugin` custom resource as described in the installation procedure.
+--
+
+ClusterLogForwarder Ready but no logs appear::
++
+*Symptom*: The `ClusterLogForwarder` status shows `Ready: True` but no logs appear in `LokiStack` or destination sinks.
++
+*Cause*: The `ClusterLogForwarder` validation passes even when the collector cannot connect to `LokiStack`. The most common cause is missing TLS CA configuration.
++
+*Resolution*:
++
+--
+. Check the collector pod logs by running the following command:
++
+[source,terminal]
+----
+$ oc logs -n openshift-logging -l app.kubernetes.io/component=collector --tail=50
+----
+
+. Look for TLS certificate errors or connection failures.
+
+. If you see certificate errors, add the `tls.ca` block to your `ClusterLogForwarder` output as described in "Certificate verify failed error in collector logs."
+--
diff --git a/modules/troubleshoot-loki-rate-limit-errors.adoc b/modules/troubleshoot-loki-rate-limit-errors.adoc
@@ -0,0 +1,87 @@
+// Module is included in the following assemblies:
+//
+// * troubleshooting/troubleshoot-log-forwarding.adoc
+// * configuring/configuring-the-log-store.adoc
+// * configuring/tuning-the-log-store.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="troubleshoot-loki-rate-limit-errors_{context}"]
+[id="loki-rate-limit-errors_{context}"]
+= Troubleshoot Loki rate limit errors
+
+[role="_abstract"]
+Resolve Loki rate limit errors when the Log Forwarder API sends large message blocks that exceed ingestion limits.
+
+If the Log Forwarder API forwards a large block of messages that exceeds the rate limit to Loki, Loki generates rate limit (`429`) errors.
+
+These errors can occur during normal operation. For example, when adding the {logging} to a cluster that already has some logs, rate limit errors might occur while the {logging} tries to process all of the existing log entries. In this case, if the rate of addition of new logs is less than the total rate limit, the historical data is eventually ingested, and the rate limit errors are resolved without requiring user intervention.
+
+In cases where the rate limit errors continue to occur, you can fix the issue by modifying the `LokiStack` custom resource (CR).
+
+[IMPORTANT]
+====
+The `LokiStack` CR is not available on Grafana-hosted Loki. These instructions do not apply to Grafana-hosted Loki servers.
+====
+
+.Prerequisites
+
+* The Log Forwarder API is configured to forward logs to Loki.
+
+* Your system sends a block of messages that is larger than 2 MB to Loki. For example:
++
+[source,text]
+----
+"values":[["1630410392689800468","{\"kind\":\"Event\",\"apiVersion\":\
+.......
+......
+......
+......
+\"received_at\":\"2021-08-31T11:46:32.800278+00:00\",\"version\":\"1.7.4 1.6.0\"}},\"@timestamp\":\"2021-08-31T11:46:32.799692+00:00\",\"viaq_index_name\":\"audit-write\",\"viaq_msg_id\":\"MzFjYjJkZjItNjY0MC00YWU4LWIwMTEtNGNmM2E5ZmViMGU4\",\"log_type\":\"audit\"}"]]}]}
+----
+
+* After you enter `oc logs -n openshift-logging -l component=collector`, the collector logs in your cluster show a line containing one of the following error messages:
++
+[source,text]
+----
+429 Too Many Requests Ingestion rate limit exceeded
+----
++
+.Example Vector error message
+[source,text]
+----
+2023-08-25T16:08:49.301780Z  WARN sink{component_kind="sink" component_id=default_loki_infra component_type=loki component_name=default_loki_infra}: vector::sinks::util::retries: Retrying after error. error=Server responded with an error: 429 Too Many Requests internal_log_rate_limit=true
+----
++
+The error is also visible on the receiving end. For example, in the `LokiStack` ingester pod:
++
+.Example Loki ingester error message
+[source,text]
+----
+level=warn ts=2023-08-30T14:57:34.155592243Z caller=grpc_logging.go:43 duration=1.434942ms method=/logproto.Pusher/Push err="rpc error: code = Code(429) desc = entry with timestamp 2023-08-30 14:57:32.012778399 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream
+----
+
+.Procedure
+
+* Update the `ingestionBurstSize` and `ingestionRate` fields in the `LokiStack` CR:
++
+[source,yaml]
+----
+apiVersion: loki.grafana.com/v1
+kind: LokiStack
+metadata:
+  name: logging-loki
+  namespace: openshift-logging
+spec:
+  limits:
+    global:
+      ingestion:
+        ingestionBurstSize: 16
+        ingestionRate: 8
+# ...
+----
++
+spec.limits.global.ingestion.ingestionBurstSize;;
+  The `ingestionBurstSize` field defines the maximum local rate-limited sample size per distributor replica in MB. This value is a hard limit. Set this value to at least the maximum logs size expected in a single push request. Single requests that are larger than the `ingestionBurstSize` value are not permitted.
++
+spec.limits.global.ingestion.ingestionRate;;
+  The `ingestionRate` field is a soft limit on the maximum amount of ingested samples per second in MB. Rate limit errors occur if the rate of logs exceeds the limit, but the collector retries sending the logs. If the total average is lower than the limit, the system recovers and errors are resolved without user intervention.