Merge branch 'main' into 2024.02.02_OTEL

.github/workflows/checks.yml

@@ -1,5 +1,8 @@
 name: Checks
-on: [push]
+on:
+  pull_request:
+    branches:    
+      - main
 jobs:
   checks:
     runs-on: ubuntu-latest
@@ -10,6 +13,11 @@ jobs:
     steps:
       - uses: actions/checkout@v4
       - run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
+      - name: golangci-lint
+        uses: golangci/golangci-lint-action@08e2f20817b15149a52b5b3ebe7de50aff2ba8c5
+        with:
+          version: v1.55.1
+          only-new-issues: true
       - run: make lint
       - run: make check-doc
       - run: make check-mod

CHANGELOG.md

@@ -53,6 +53,7 @@
 * [11679](https://github.com/grafana/loki/pull/11679) **dannykopping** Cache: extending #11535 to align custom ingester query split with cache keys for correct caching of results.
 * [11143](https://github.com/grafana/loki/pull/11143) **sandeepsukhani** otel: Add support for per tenant configuration for mapping otlp data to loki format
 * [11499](https://github.com/grafana/loki/pull/11284) **jmichalek132** Config: Adds `frontend.log-query-request-headers` to enable logging of request headers in query logs.
+* [11817](https://github.com/grafana/loki/pull/11817) **ashwanthgoli** Ruler: Add support for filtering results of `/prometheus/api/v1/rules` endpoint by rule_name, rule_group, file and type.
 
 ##### Fixes
 * [11074](https://github.com/grafana/loki/pull/11074) **hainenber** Fix panic in lambda-promtail due to mishandling of empty DROP_LABELS env var.

docs/sources/configure/_index.md

@@ -3143,14 +3143,22 @@ shard_streams:
 
 # OTLP log ingestion configurations
 otlp_config:
+  # Configuration for resource attributes to store them as index labels or
+  # Structured Metadata or drop them altogether
   resource_attributes:
-    [ignore_defaults: <boolean>]
+    # Configure whether to ignore the default list of resource attributes to be
+    # stored as index labels and only use the given resource attributes config
+    [ignore_defaults: <boolean> | default = false]
 
-    [attributes: <list of AttributesConfigs>]
+    [attributes_config: <list of attributes_configs>]
 
-  [scope_attributes: <list of AttributesConfigs>]
+  # Configuration for scope attributes to store them as Structured Metadata or
+  # drop them altogether
+  [scope_attributes: <list of attributes_configs>]
 
-  [log_attributes: <list of AttributesConfigs>]
+  # Configuration for log attributes to store them as Structured Metadata or
+  # drop them altogether
+  [log_attributes: <list of attributes_configs>]
 ```
 
 ### frontend_worker
@@ -5292,6 +5300,24 @@ Named store from this example can be used by setting object_store to store-1 in
 [cos: <map of string to cos_storage_config>]
 ```
 
+### attributes_config
+
+Define actions for matching OpenTelemetry (OTEL) attributes.
+
+```yaml
+# Configures action to take on matching attributes. It allows one of
+# [structured_metadata, drop] for all attribute types. It additionally allows
+# index_label action for resource attributes
+[action: <string> | default = ""]
+
+# List of attributes to configure how to store them or drop them altogether
+[attributes: <list of strings>]
+
+# Regex to choose attributes to configure how to store them or drop them
+# altogether
+[regex: <Regexp>]
+```
+
 ## Runtime Configuration file
 
 Loki has a concept of "runtime config" file, which is simply a file that is reloaded while Loki is running. It is used by some Loki components to allow operator to change some aspects of Loki configuration without restarting it. File is specified by using `-runtime-config.file=<filename>` flag and reload period (which defaults to 10 seconds) can be changed by `-runtime-config.reload-period=<duration>` flag. Previously this mechanism was only used by limits overrides, and flags were called `-limits.per-user-override-config=<filename>` and `-limits.per-user-override-period=10s` respectively. These are still used, if `-runtime-config.file=<filename>` is not specified.

docs/sources/reference/api.md

@@ -1178,11 +1178,15 @@ Deletes all the rule groups in a namespace (including the namespace itself). Thi
 ### List rules
 
 ```
-GET /prometheus/api/v1/rules
+GET /prometheus/api/v1/rules?type={alert|record}&file={}&rule_group={}&rule_name={}
 ```
 
 Prometheus-compatible rules endpoint to list alerting and recording rules that are currently loaded.
 
+The `type` parameter is optional. If set, only the specified type of rule is returned.
+
+The `file`, `rule_group` and `rule_name` parameters are optional, and can accept multiple values. If set, the response content is filtered accordingly.
+
 For more information, refer to the [Prometheus rules](https://prometheus.io/docs/prometheus/latest/querying/api/#rules) documentation.
 
 ### List alerts

docs/sources/send-data/otel/_index.md

@@ -65,7 +65,7 @@ service:
 
 ## Format considerations
 
-Since the OpenTelemetry protocol  differs from the Loki storage model, here is how data in the OpenTelemetry format will be mapped to the Loki data model during ingestion:
+Since the OpenTelemetry protocol differs from the Loki storage model, here is how data in the OpenTelemetry format will be mapped by default to the Loki data model during ingestion, which can be changed as explained later:
 
 - Index labels: Resource attributes map well to index labels in Loki, since both usually identify the source of the logs. Because Loki has a limit of 30 index labels, we have selected the following resource attributes to be stored as index labels, while the remaining attributes are stored as [Structured Metadata]({{< relref "../../get-started/labels/structured-metadata" >}}) with each log entry:
   - cloud.availability_zone
@@ -112,3 +112,102 @@ Things to note before ingesting OpenTelemetry logs to Loki:
 - Stringification of non-string Attribute values
 
   While converting Attribute values in OTLP to Index label values or Structured Metadata, any non-string values are converted to string using [AsString method from the OTEL collector lib](https://github.com/open-telemetry/opentelemetry-collector/blob/ab3d6c5b64701e690aaa340b0a63f443ff22c1f0/pdata/pcommon/value.go#L353).
+
+### Changing the default mapping of OTLP to Loki Format
+
+Loki supports [per tenant]({{< relref "../../configure#limits_config" >}}) OTLP config which lets you change the default mapping of OTLP to Loki format for each tenant.
+It currently only supports changing the storage of Attributes. Here is how the config looks like:
+
+```yaml
+# OTLP log ingestion configurations
+otlp_config:
+  # Configuration for Resource Attributes to store them as index labels or
+  # Structured Metadata or drop them altogether
+  resource_attributes:
+    # Configure whether to ignore the default list of Resource Attributes to be
+    # stored as Index Labels and only use the given Resource Attributes config
+    [ignore_defaults: <boolean>]
+
+    [attributes_config: <list of attributes_configs>]
+
+  # Configuration for Scope Attributes to store them as Structured Metadata or
+  # drop them altogether
+  [scope_attributes: <list of attributes_configs>]
+
+  # Configuration for Log Attributes to store them as Structured Metadata or
+  # drop them altogether
+  [log_attributes: <list of attributes_configs>]
+
+attributes_config:
+  # Configures action to take on matching Attributes. It allows one of
+  # [structured_metadata, drop] for all Attribute types. It additionally allows
+  # index_label action for Resource Attributes
+  [action: <string> | default = ""]
+
+  # List of attributes to configure how to store them or drop them altogether
+  [attributes: <list of strings>]
+
+  # Regex to choose attributes to configure how to store them or drop them
+  # altogether
+  [regex: <Regexp>]
+```
+
+Here are some example configs to change the default mapping of OTLP to Loki format:
+
+#### Example 1:
+
+```yaml
+otlp_config:
+  resource_attributes:
+    attributes_config:
+      - action: index_label
+        attributes:
+          - service.group
+```
+
+With the example config, here is how various kinds of Attributes would be stored:
+* Store all 17 Resource Attributes mentioned earlier and `service.group` Resource Attribute as index labels.
+* Store remaining Resource Attributes as Structured Metadata.
+* Store all the Scope and Log Attributes as Structured Metadata.
+
+#### Example 2:
+
+```yaml
+otlp_config:
+  resource_attributes:
+    ignore_defaults: true
+    attributes_config:
+      - action: index_label
+        regex: service.group
+```
+
+With the example config, here is how various kinds of Attributes would be stored:
+* **Only** store `service.group` Resource Attribute as index labels.
+* Store remaining Resource Attributes as Structured Metadata.
+* Store all the Scope and Log Attributes as Structured Metadata.
+
+#### Example 2:
+
+```yaml
+otlp_config:
+  resource_attributes:
+    attributes_config:
+      - action: index_label
+        regex: service.group
+  scope_attributes:
+    - action: drop
+      attributes:
+        - method.name
+  log_attributes:
+    - action: structured_metadata
+      attributes:
+        - user.id
+    - action: drop
+      regex: .*
+```
+
+With the example config, here is how various kinds of Attributes would be stored:
+* Store all 17 Resource Attributes mentioned earlier and `service.group` Resource Attribute as index labels.
+* Store remaining Resource Attributes as Structured Metadata.
+* Drop Scope Attribute named `method.name` and store all other Scope Attributes as Structured Metadata.
+* Store Log Attribute named `user.id` as Structured Metadata and drop all other Log Attributes.

pkg/loghttp/push/otlp_config.go

@@ -56,9 +56,9 @@ var DefaultOTLPConfig = OTLPConfig{
 }
 
 type OTLPConfig struct {
-	ResourceAttributes ResourceAttributesConfig `yaml:"resource_attributes,omitempty"`
-	ScopeAttributes    []AttributesConfig       `yaml:"scope_attributes,omitempty"`
-	LogAttributes      []AttributesConfig       `yaml:"log_attributes,omitempty"`
+	ResourceAttributes ResourceAttributesConfig `yaml:"resource_attributes,omitempty" doc:"description=Configuration for resource attributes to store them as index labels or Structured Metadata or drop them altogether"`
+	ScopeAttributes    []AttributesConfig       `yaml:"scope_attributes,omitempty" doc:"description=Configuration for scope attributes to store them as Structured Metadata or drop them altogether"`
+	LogAttributes      []AttributesConfig       `yaml:"log_attributes,omitempty" doc:"description=Configuration for log attributes to store them as Structured Metadata or drop them altogether"`
 }
 
 func (c *OTLPConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
@@ -115,9 +115,9 @@ func (c *OTLPConfig) Validate() error {
 }
 
 type AttributesConfig struct {
-	Action     Action         `yaml:"action,omitempty"`
-	Attributes []string       `yaml:"attributes,omitempty"`
-	Regex      relabel.Regexp `yaml:"regex,omitempty"`
+	Action     Action         `yaml:"action,omitempty" doc:"description=Configures action to take on matching attributes. It allows one of [structured_metadata, drop] for all attribute types. It additionally allows index_label action for resource attributes"`
+	Attributes []string       `yaml:"attributes,omitempty" doc:"description=List of attributes to configure how to store them or drop them altogether"`
+	Regex      relabel.Regexp `yaml:"regex,omitempty" doc:"description=Regex to choose attributes to configure how to store them or drop them altogether"`
 }
 
 func (c *AttributesConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
@@ -146,8 +146,8 @@ func (c *AttributesConfig) UnmarshalYAML(unmarshal func(interface{}) error) erro
 }
 
 type ResourceAttributesConfig struct {
-	IgnoreDefaults   bool               `yaml:"ignore_defaults,omitempty"`
-	AttributesConfig []AttributesConfig `yaml:"attributes,omitempty"`
+	IgnoreDefaults   bool               `yaml:"ignore_defaults,omitempty" doc:"default=false|description=Configure whether to ignore the default list of resource attributes to be stored as index labels and only use the given resource attributes config"`
+	AttributesConfig []AttributesConfig `yaml:"attributes_config,omitempty"`
 }
 
 func (c *ResourceAttributesConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {

pkg/loghttp/push/otlp_config_test.go

@@ -19,7 +19,7 @@ func TestUnmarshalOTLPConfig(t *testing.T) {
 			name: "only resource_attributes set",
 			yamlConfig: []byte(`
 resource_attributes:
-  attributes:
+  attributes_config:
     - action: index_label
       regex: foo`),
 			expectedCfg: OTLPConfig{
@@ -39,7 +39,7 @@ resource_attributes:
 			yamlConfig: []byte(`
 resource_attributes:
   ignore_defaults: true
-  attributes:
+  attributes_config:
     - action: index_label
       regex: foo`),
 			expectedCfg: OTLPConfig{
@@ -82,7 +82,7 @@ scope_attributes:
 			name: "all 3 set",
 			yamlConfig: []byte(`
 resource_attributes:
-  attributes:
+  attributes_config:
     - action: index_label
       regex: foo
 scope_attributes: