Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Common storage backend docs #2347

Merged
merged 10 commits into from
Aug 8, 2022
30 changes: 30 additions & 0 deletions docs/sources/operators-guide/configure/about-configurations.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,36 @@ A given configuration loads at startup and cannot be modified at runtime. Howeve

To see the current configuration state of any component, use the [`/config`]({{< relref "../reference-http-api/index.md#configuration" >}}) or [`/runtime_config`]({{< relref "../reference-http-api/index.md#runtime-configuration" >}}) HTTP API endpoint.

## Common configurations

Some configurations, such as object storage backend, are repeated for multiple components.
To avoid the repetition in the configuration file, use the [`common`]({{< relref "../configure/reference-configuration-parameters/index.md#common" >}}) configuration section or `-common.*` CLI flags.
colega marked this conversation as resolved.
Show resolved Hide resolved
Common configurations are first applied to all the specific configurations, which allows them to be overridden later by specific values.
colega marked this conversation as resolved.
Show resolved Hide resolved

For example, the following configuration uses the same Amazon S3 object storage bucket called `mimir`. The common storage is located in the `us-east` region for both ruler and alertmanager stores, and the blocks storage uses the `mimir-blocks` bucket from the same region:
colega marked this conversation as resolved.
Show resolved Hide resolved

```yaml
common:
storage:
backend: s3
s3:
region: us-east
bucket_name: mimir

blocks_storage:
s3:
bucket_name: mimir-blocks
```

For a reference of this configuration, see [object storage configuration reference ]({{< relref "configure-object-storage-backend.md" >}}).
colega marked this conversation as resolved.
Show resolved Hide resolved

The precedence of the common configuration is as follows, where each configuration overrides the previous one:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very clear, nice job! :)


- YAML common values
- YAML specific values
- CLI common flags
- CLI specific flags

## Operational considerations

Use a single configuration file, and either pass it to all replicas of Grafana Mimir (when running multiple single-process Mimir replicas) or to all components of Grafana Mimir (when running Grafana Mimir as microservices). When running Grafana Mimir on Kubernetes, you can achieve this by storing the configuration file in a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) and mounting it in each Grafana Mimir container.
colega marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
title: "Configure Grafana Mimir object storage backend"
menuTitle: "Configure object storage"
description: "Learn how to configure Grafana Mimir to use different object storage backend implementations."
weight: 70
---

# Configure Grafana Mimir object storage backend

Grafana Mimir can use different object storage services to persist blocks containing the metrics data, as well as recording rules and alertmanager state.
The supported backends are:

- [Amazon S3](https://aws.amazon.com/s3/) (and compatible implementations like [MinIO](https://min.io/))
- [Google Cloud Storage](https://cloud.google.com/storage)
- [Azure Blob Storage](https://azure.microsoft.com/es-es/services/storage/blobs/)
- [Swift (OpenStack Object Storage)](https://wiki.openstack.org/wiki/Swift)

Additionally, a file system emulated [`filesystem`]({{< relref "../configure/reference-configuration-parameters/index.md#filesystem_storage_backend" >}}) object storage implementation can be also used for testing purposes (it's not recommended for production workloads).
colega marked this conversation as resolved.
Show resolved Hide resolved

[Ruler and alertmanager support a `local` implementation]({{< relref "../architecture/components/ruler/index.md#local-storage" >}}), which is similar to `filesystem` in the way that it uses the local file system, but it is a read-only data source and can be used to provision state into those components.
colega marked this conversation as resolved.
Show resolved Hide resolved

## Common configuration

The [common configuration]({{< relref "about-configurations.md#common-configurations" >}}) can be used to avoid repetition by filling the [`common`]({{< relref "../configure/reference-configuration-parameters/index.md#common" >}}) configuration block or by providing the `-common.storage.*` CLI flags.
colega marked this conversation as resolved.
Show resolved Hide resolved

Note that blocks storage can't be located in the same path of the same bucket as the ruler and alertmanager stores, so when using the common configuration, [`blocks_storage`]({{< relref "../configure/reference-configuration-parameters/index.md#blocks_storage" >}}) should either:
colega marked this conversation as resolved.
Show resolved Hide resolved

- use a different bucket, overriding the common bucket name
- use a storage prefix
colega marked this conversation as resolved.
Show resolved Hide resolved

Grafana Mimir will fail to start if blocks storage is configured to use the same bucket and storage prefix as alertmanager or ruler store.
colega marked this conversation as resolved.
Show resolved Hide resolved

A valid configuration for the object storage (taken from the ["Play with Grafana Mimir" tutorial](https://grafana.com/tutorials/play-with-grafana-mimir/)) would be:
colega marked this conversation as resolved.
Show resolved Hide resolved

```yaml
common:
storage:
backend: s3
s3:
endpoint: minio:9000
access_key_id: mimir
secret_access_key: supersecret
insecure: true
bucket_name: mimir

blocks_storage:
storage_prefix: blocks
```

> **Note**: If you're using a mix of YAML files and CLI flags, pay attention to the [precedence logic]({{< relref "about-configurations.md#common-configurations" >}}) of those.
colega marked this conversation as resolved.
Show resolved Hide resolved