Skip to content

Commit

Permalink
Merge branch 'main' into aws-helm-distributed
Browse files Browse the repository at this point in the history
  • Loading branch information
@Jayclifford345
Jayclifford345 authored Oct 24, 2024
2 parents fbfb275 + cff9f43 commit 01ca977
Show file tree
Hide file tree
Showing 48 changed files with 3,915 additions and 3,193 deletions.
6 changes: 2 additions & 4 deletions docs/sources/release-notes/v3-1.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,19 +79,18 @@ Out of an abundance of caution, we advise that users with Loki or Grafana Enterp

## Bug fixes

## 3.1.2 (2024-10-17)
### 3.1.2 (2024-10-17)

- **config:** Copy Alibaba and IBM object storage configuration from common ([#14316](https://github.com/grafana/loki/issues/14316)) ([7184d45](https://github.com/grafana/loki/commit/7184d45d8e080874feea8bfd223dedf5f20d3836)).
- **docker-compose:** upgrade loki and grafana production image tags to 3.1.1 ([#14025](https://github.com/grafana/loki/issues/14025)) ([36fe29e](https://github.com/grafana/loki/commit/36fe29eb334d8300265ca437c0acb423a01c5041)).
- **logql:** updated JSONExpressionParser not to unescape extracted values if it is JSON object. (backport release-3.1.x) ([#14503](https://github.com/grafana/loki/issues/14503)) ([759f9c8](https://github.com/grafana/loki/commit/759f9c8525227bb1272771a40429d12e015874d9)).
- **promtail:** Revert build image to Debian Bullseye to fix libc version issue in Promtail ([#14387](https://github.com/grafana/loki/issues/14387)) ([05b6a65](https://github.com/grafana/loki/commit/05b6a65f8bf00b880f17465553b1adaf0cf56d60)).
- **storage:** have GetObject check for canceled context (backport release-3.1.x) ([#14421](https://github.com/grafana/loki/issues/14421)) ([f3d69ff](https://github.com/grafana/loki/commit/f3d69ffa960c91c0239436a32bb0aa578c0f022a)).

## 3.1.1 (2024-08-08)
### 3.1.1 (2024-08-08)

- **deps:** Bumped dependencies versions to resolve CVEs ([#13789](https://github.com/grafana/loki/issues/13789)) ([34206cd](https://github.com/grafana/loki/commit/34206cd2d6290566034710ae6c2d08af8804bc91)).


### 3.1.0 (2024-07-02)

- **blooms:** Fix duplicate enqueue item problem in bloom download queue when do sync download ([#13114](https://github.com/grafana/loki/issues/13114)) ([f98ff7f](https://github.com/grafana/loki/commit/f98ff7f58400b5f5a425fae003fb959bfb8c6454)).
Expand Down Expand Up @@ -193,4 +192,3 @@ With this fix, if the `max_line_size` config in Promtail has a value, the docker
- Use to the proper config names in warning messages ([#12114](https://github.com/grafana/loki/issues/12114)) ([4a05964](https://github.com/grafana/loki/commit/4a05964d5520d46d149f2a4e4709eee36c7fb418)).
- **workflows:** Use an intermediate env variable in GitHub workflow ([#12905](https://github.com/grafana/loki/issues/12905)) ([772616c](https://github.com/grafana/loki/commit/772616cd8f5cbac70374dd4a53f1714fb49a7a3b)).
- **workflows:** Don't run metric collector on forks ([#12687](https://github.com/grafana/loki/issues/12687)) ([7253444](https://github.com/grafana/loki/commit/72534449a07cd9f410973f2d01772024e8e4b7ba)).

9 changes: 9 additions & 0 deletions docs/sources/setup/install/helm/reference.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4233,6 +4233,15 @@ false
<td><pre lang="json">
{}
</pre>
</td>
</tr>
<tr>
<td>gateway.nginxConfig.clientMaxBodySize</td>
<td>string</td>
<td>Allows customizing the `client_max_body_size` directive</td>
<td><pre lang="json">
"4M"
</pre>
</td>
</tr>
<tr>
Expand Down
13 changes: 5 additions & 8 deletions docs/sources/shared/configuration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3765,7 +3765,7 @@ shard_streams:
[bloom_creation_enabled: <boolean> | default = false]

# Experimental. Bloom planning strategy to use in bloom creation. Can be one of:
# 'split_keyspace_by_factor'
# 'split_keyspace_by_factor', 'split_by_series_chunks_size'
# CLI flag: -bloom-build.planning-strategy
[bloom_planning_strategy: <string> | default = "split_keyspace_by_factor"]

Expand All @@ -3775,6 +3775,10 @@ shard_streams:
# CLI flag: -bloom-build.split-keyspace-by
[bloom_split_series_keyspace_by: <int> | default = 256]

# Experimental. Target chunk size in bytes for bloom tasks. Default is 20GB.
# CLI flag: -bloom-build.split-target-series-chunk-size
[bloom_task_target_series_chunk_size: <int> | default = 20GB]

# Experimental. Compression algorithm for bloom block pages.
# CLI flag: -bloom-build.block-encoding
[bloom_block_encoding: <string> | default = "none"]
Expand Down Expand Up @@ -5646,13 +5650,6 @@ congestion_control:
# CLI flag: -store.max-parallel-get-chunk
[max_parallel_get_chunk: <int> | default = 150]

# Enables the use of thanos-io/objstore clients for connecting to object
# storage. When set to true, the configuration inside
# `storage_config.object_store` or `common.storage.object_store` block takes
# effect.
# CLI flag: -use-thanos-objstore
[use_thanos_objstore: <boolean> | default = false]

# The maximum number of chunks to fetch per batch.
# CLI flag: -store.max-chunk-batch-size
[max_chunk_batch_size: <int> | default = 50]
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ require (
github.com/thanos-io/objstore v0.0.0-20241015070247-5f04b8b0b52a
github.com/twmb/franz-go v1.17.1
github.com/twmb/franz-go/pkg/kadm v1.13.0
github.com/twmb/franz-go/pkg/kfake v0.0.0-20240821035758-b77dd13e2bfa
github.com/twmb/franz-go/pkg/kfake v0.0.0-20241015013301-cea7aa5d8037
github.com/twmb/franz-go/pkg/kmsg v1.8.0
github.com/twmb/franz-go/plugin/kotel v1.5.0
github.com/twmb/franz-go/plugin/kprom v1.1.0
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1831,8 +1831,8 @@ github.com/twmb/franz-go v1.17.1 h1:0LwPsbbJeJ9R91DPUHSEd4su82WJWcTY1Zzbgbg4CeQ=
github.com/twmb/franz-go v1.17.1/go.mod h1:NreRdJ2F7dziDY/m6VyspWd6sNxHKXdMZI42UfQ3GXM=
github.com/twmb/franz-go/pkg/kadm v1.13.0 h1:bJq4C2ZikUE2jh/wl9MtMTQ/kpmnBgVFh8XMQBEC+60=
github.com/twmb/franz-go/pkg/kadm v1.13.0/go.mod h1:VMvpfjz/szpH9WB+vGM+rteTzVv0djyHFimci9qm2C0=
github.com/twmb/franz-go/pkg/kfake v0.0.0-20240821035758-b77dd13e2bfa h1:OmQ4DJhqeOPdIH60Psut1vYU8A6LGyxJbF09w5RAa2w=
github.com/twmb/franz-go/pkg/kfake v0.0.0-20240821035758-b77dd13e2bfa/go.mod h1:nkBI/wGFp7t1NJnnCeJdS4sX5atPAqwCPpDXKuI7SC8=
github.com/twmb/franz-go/pkg/kfake v0.0.0-20241015013301-cea7aa5d8037 h1:M4Zj79q1OdZusy/Q8TOTttvx/oHkDVY7sc0xDyRnwWs=
github.com/twmb/franz-go/pkg/kfake v0.0.0-20241015013301-cea7aa5d8037/go.mod h1:nkBI/wGFp7t1NJnnCeJdS4sX5atPAqwCPpDXKuI7SC8=
github.com/twmb/franz-go/pkg/kmsg v1.8.0 h1:lAQB9Z3aMrIP9qF9288XcFf/ccaSxEitNA1CDTEIeTA=
github.com/twmb/franz-go/pkg/kmsg v1.8.0/go.mod h1:HzYEb8G3uu5XevZbtU0dVbkphaKTHk0X68N5ka4q6mU=
github.com/twmb/franz-go/plugin/kotel v1.5.0 h1:TiPfGUbQK384OO7ZYGdo7JuPCbJn+/8njQ/D9Je9CDE=
Expand Down
2 changes: 2 additions & 0 deletions integration/bloom_building_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,8 @@ func TestBloomBuilding(t *testing.T) {
"-bloom-build.planner.interval=15s",
"-bloom-build.planner.min-table-offset=0", // Disable table offset so we process today's data.
"-bloom.cache-list-ops=0", // Disable cache list operations to avoid caching issues.
"-bloom-build.planning-strategy=split_by_series_chunks_size",
"-bloom-build.split-target-series-chunk-size=1KB",
)
require.NoError(t, clu.Run())

Expand Down
193 changes: 72 additions & 121 deletions operator/api/loki/v1/lokistack_types.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,34 @@ type OpenshiftTenantSpec struct {
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Admin Groups"
AdminGroups []string `json:"adminGroups"`

// OTLP contains settings for ingesting data using OTLP in the OpenShift tenancy mode.
//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="OpenTelemetry Protocol"
OTLP *OpenshiftOTLPConfig `json:"otlp,omitempty"`
}

// OpenshiftOTLPConfig defines configuration specific to users using OTLP together with an OpenShift tenancy mode.
type OpenshiftOTLPConfig struct {
// DisableRecommendedAttributes can be used to reduce the number of attributes used for stream labels and structured
// metadata.
//
// Enabling this setting removes the "recommended attributes" from the generated Loki configuration. This will cause
// meta information to not be available as stream labels or structured metadata, potentially making queries more
// expensive and less performant.
//
// Note that there is a set of "required attributes", needed for OpenShift Logging to work properly. Those will be
// added to the configuration, even if this field is set to true.
//
// This option is supposed to be combined with a custom label configuration customizing the labels for the specific
// usecase.
//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Disable recommended OTLP attributes"
DisableRecommendedAttributes bool `json:"disableRecommendedAttributes,omitempty"`
}

// LokiComponentSpec defines the requirements to configure scheduling
Expand Down Expand Up @@ -791,138 +819,70 @@ type IngestionLimitSpec struct {
PerStreamRateLimitBurst int32 `json:"perStreamRateLimitBurst,omitempty"`
}

// OTLPAttributeAction defines the action to executed when indexing
// OTLP resource attributes. Resource attributes can be either added
// to the index, the chunk structured metadata or entirely dropped.
type OTLPAttributeAction string

const (
// OTLPAttributeActionIndexLabel stores a resource attribute as a label, which is part of the index identifying streams.
OTLPAttributeActionIndexLabel OTLPAttributeAction = "indexLabel"
// OTLPAttributeActionStructuredMetadata stores an attribute as structured metadata with each log entry.
OTLPAttributeActionStructuredMetadata OTLPAttributeAction = "structuredMetadata"
// OTLPAttributeActionDrop removes the matching attributes from the log entry.
OTLPAttributeActionDrop OTLPAttributeAction = "drop"
)

// OTLPAttributesSpec contains the configuration for a set of attributes
// to store them as index labels or structured metadata or drop them altogether.
type OTLPAttributesSpec struct {
// Action defines the indexing action for the selected attributes. They
// can be either added to structured metadata or drop altogether.
//
// +required
// +kubebuilder:validation:Required
// +kubebuilder:validation:Enum=structured_metadata;drop
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Action"
Action OTLPAttributeAction `json:"action"`

// Attributes allows choosing the attributes by listing their names.
// OTLPSpec defines which resource, scope and log attributes should be used as stream labels or
// stored as structured metadata.
type OTLPSpec struct {
// StreamLabels configures which resource attributes are converted to Loki stream labels.
//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Attribute Names"
Attributes []string `json:"attributes,omitempty"`
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Stream Labels"
StreamLabels *OTLPStreamLabelSpec `json:"streamLabels,omitempty"`

// Regex allows choosing the attributes by matching a regular expression.
// StructuredMetadata configures which attributes are saved in structured metadata.
//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Regular Expression"
Regex string `json:"regex,omitempty"`
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Structured Metadata"
StructuredMetadata *OTLPMetadataSpec `json:"structuredMetadata,omitempty"`
}

// OTLPResourceAttributesConfigSpec contains the configuration for a set of resource attributes
// to store them as index labels or structured metadata or drop them altogether.
type OTLPResourceAttributesConfigSpec struct {
// Action defines the indexing action for the selected resoure attributes. They
// can be either indexed as labels, added to structured metadata or drop altogether.
//
// +required
// +kubebuilder:validation:Required
// +kubebuilder:validation:Enum=index_label;structured_metadata;drop
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Action"
Action OTLPAttributeAction `json:"action"`

// Attributes is the list of attributes to configure indexing or drop them
// altogether.
//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Attribute Names"
Attributes []string `json:"attributes,omitempty"`

// Regex allows choosing the attributes by matching a regular expression.
type OTLPStreamLabelSpec struct {
// ResourceAttributes lists the names of the resource attributes that should be converted into Loki stream labels.
//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Regular Expression"
Regex string `json:"regex,omitempty"`
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Resource Attributes"
ResourceAttributes []OTLPAttributeReference `json:"resourceAttributes,omitempty"`
}

// OTLPResourceAttributesSpec contains the configuration for resource attributes
// to store them as index labels or structured metadata or drop them altogether.
type OTLPResourceAttributesSpec struct {
// IgnoreDefaults controls whether to ignore the global configuration for resource attributes
// indexed as labels.
//
// If IgnoreDefaults is true, then this spec needs to contain at least one mapping to a index label.
type OTLPMetadataSpec struct {
// ResourceAttributes lists the names of resource attributes that should be included in structured metadata.
//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,xDescriptors="urn:alm:descriptor:com.tectonic.ui:booleanSwitch",displayName="Ignore Global Defaults"
IgnoreDefaults bool `json:"ignoreDefaults,omitempty"`
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Resource Attributes"
ResourceAttributes []OTLPAttributeReference `json:"resourceAttributes,omitempty"`

// Attributes contains the configuration for resource attributes
// to store them as index labels or structured metadata or drop them altogether.
// ScopeAttributes lists the names of scope attributes that should be included in structured metadata.
//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Attributes"
Attributes []OTLPResourceAttributesConfigSpec `json:"attributes,omitempty"`
}
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Scope Attributes"
ScopeAttributes []OTLPAttributeReference `json:"scopeAttributes,omitempty"`

// GlobalOTLPSpec defines which resource, scope and log attributes to
// be stored as index or structured metadata or drop altogether for all
// tenants.
type GlobalOTLPSpec struct {
// IndexedResourceAttributes contains the global configuration for resource attributes
// to store them as index labels.
// LogAttributes lists the names of log attributes that should be included in structured metadata.
//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Indexed Resource Attributes"
IndexedResourceAttributes []string `json:"indexedResourceAttributes,omitempty"`

OTLPSpec `json:",omitempty"`
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Log Attributes"
LogAttributes []OTLPAttributeReference `json:"logAttributes,omitempty"`
}

// OTLPSpec defines which resource, scope and log attributes to
// be stored as index or structured metadata or drop altogether
type OTLPSpec struct {
// ResourceAttributes contains the configuration for resource attributes
// to store them as index labels or structured metadata or drop them altogether.
type OTLPAttributeReference struct {
// Name contains either a verbatim name of an attribute or a regular expression matching many attributes.
//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Resource Attributes"
ResourceAttributes *OTLPResourceAttributesSpec `json:"resourceAttributes,omitempty"`

// ScopeAttributes contains the configuration for scope attributes
// to store them as structured metadata or drop them altogether.
//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Scope Attributes"
ScopeAttributes []OTLPAttributesSpec `json:"scopeAttributes,omitempty"`
// +required
// +kubebuilder:validation:Required
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Name"
Name string `json:"name"`

// LogAttributes contains the configuration for log attributes
// to store them as structured metadata or drop them altogether.
// If Regex is true, then Name is treated as a regular expression instead of as a verbatim attribute name.
//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Log Attributes"
LogAttributes []OTLPAttributesSpec `json:"logAttributes,omitempty"`
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Treat name as regular expression"
Regex bool `json:"regex,omitempty"`
}

// RetentionStreamSpec defines a log stream with separate retention time.
Expand Down Expand Up @@ -978,13 +938,14 @@ type LimitsTemplateSpec struct {
// +kubebuilder:validation:Optional
QueryLimits *QueryLimitSpec `json:"queries,omitempty"`

// OTLP to configure which resource, scope and log attributes
// to store as labels or structured metadata or drop them altogether
// for all tenants.
// OTLP to configure which resource, scope and log attributes are stored as stream labels or structured metadata.
//
// Tenancy modes can provide a default OTLP configuration, when no custom OTLP configuration is set or even
// enforce the use of some required attributes.
//
// +optional
// +kubebuilder:validation:Optional
OTLP *GlobalOTLPSpec `json:"otlp,omitempty"`
OTLP *OTLPSpec `json:"otlp,omitempty"`

// Retention defines how long logs are kept in storage.
//
Expand All @@ -993,7 +954,7 @@ type LimitsTemplateSpec struct {
Retention *RetentionLimitSpec `json:"retention,omitempty"`
}

// LimitsTemplateSpec defines the limits applied at ingestion or query path.
// PerTenantLimitsTemplateSpec defines the limits applied at ingestion or query path.
type PerTenantLimitsTemplateSpec struct {
// IngestionLimits defines the limits applied on ingested log streams.
//
Expand All @@ -1007,9 +968,12 @@ type PerTenantLimitsTemplateSpec struct {
// +kubebuilder:validation:Optional
QueryLimits *PerTenantQueryLimitSpec `json:"queries,omitempty"`

// OTLP to configure which resource, scope and log attributes
// to store as labels or structured metadata or drop them altogether
// for a single tenants.
// OTLP to configure which resource, scope and log attributes are stored as stream labels or structured metadata.
//
// Tenancy modes can provide a default OTLP configuration, when no custom OTLP configuration is set or even
// enforce the use of some required attributes.
//
// The per-tenant configuration for OTLP attributes will be merged with the global configuration.
//
// +optional
// +kubebuilder:validation:Optional
Expand Down Expand Up @@ -1463,16 +1427,3 @@ func (t BlockedQueryTypes) String() string {

return strings.Join(res, ",")
}

func (a OTLPAttributeAction) Value() string {
switch a {
case OTLPAttributeActionIndexLabel:
return "index_label"
case OTLPAttributeActionStructuredMetadata:
return "structured_metadata"
case OTLPAttributeActionDrop:
return "drop"
default:
return string(a)
}
}
Loading

0 comments on commit 01ca977

Please sign in to comment.