Skip to content

Commit

Permalink
Add comment about Thanos, external labels and downsampling. (grafana#…
Browse files Browse the repository at this point in the history
…2122)

* Add comment about Thanos, external labels and downsampling.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* Explain how Mimir uses external labels from Thanos.
Removed section on metaconvert.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* Apply Nick's suggestions.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* Apply suggestions from review.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* Be explicit about what user should do in different scenarios.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* make doc

* Apply review feedback.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* Don't mention label used by compactor at all.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* Add notes about rewriting blocks and offline deduplication.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
  • Loading branch information
pstibrany authored Jul 4, 2022
1 parent cff5d73 commit 78f18f5
Showing 1 changed file with 28 additions and 38 deletions.
66 changes: 28 additions & 38 deletions docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ This document guides an operator through the process of migrating a deployment o

Grafana Mimir stores series in TSDB blocks uploaded in an object storage bucket.
These blocks are the same as those used by Prometheus and Thanos.
Each project stores blocks in different places and uses different block metadata files.
Each project stores blocks in different places and uses slightly different block metadata files.

## Configuring remote write to Grafana Mimir

Expand Down Expand Up @@ -47,53 +47,43 @@ aws s3 cp <DIRECTORY> s3://<TENANT>/<DIRECTORY>
gsutil -m cp -r <DIRECTORY> gs://<TENANT>/<DIRECTORY>
```

## Migrating the block `meta.json` metadata using `metaconvert`
## Block metadata

Every block has a `meta.json` metadata file used by Grafana Mimir, Prometheus, and Thanos to identify the block contents.
Each block has a `meta.json` metadata file that is used by Grafana Mimir, Prometheus, and Thanos to identify the block contents.
Each project has its own metadata conventions.
The `metaconvert` tool migrates the `meta.json` metadata from one project to another.

### Downloading `metaconvert`
In the Grafana Mimir 2.1 (or earlier) release, the ingesters added an external label to the `meta.json` file to identify the tenant that owns the block.

- If you are using Docker, run the following command:
In the Grafana Mimir 2.2 (or later) release, blocks no longer have a label that identifies the tenant.

```bash
docker pull grafana/metaconvert:latest
```

- If you are using a release binary, download the appropriate release asset for your operating system and architecture and make it executable. For Linux with the AMD64 architecture, run the following command:

```bash
curl -LO https://github.com/grafana/mimir/releases/latest/download/metaconvert-linux-amd64
chmod +x metaconvert-linux-amd64
```

- If you are using Go, run the following command:

```bash
go install github.com/grafana/mimir/cmd/metaconvert@latest
```

### Running `metaconvert`
> **Note**: Blocks from Prometheus do not have any external labels stored in them; only blocks from Thanos use labels.
> **Warning:** The `metaconvert` tool modifies objects in place.
> Ensure you enable bucket versioning or have backups before running the tool.
## Considerations on Thanos specific features

To run `metaconvert`, provide it with the bucket configuration. Use `metaconvert -h` to list available parameters.
Thanos requires that Prometheus is configured with external labels.
When the Thanos sidecar uploads blocks, it includes the external labels from Prometheus in the `meta.json` file inside the block.
When you query the block, Thanos injects Prometheus’ external labels in the series returned in the query result.
Thanos also uses labels for the deduplication of replicated data.

1. Complete one of the following steps to run `metaconvert` in a dry-run mode, which lists blocks for migration.
If you want to use existing blocks from Thanos by Grafana Mimir, there are some considerations:

- If you are using Docker, run the following command:
**Grafana Mimir doesn't inject external labels into query results.**
This means that blocks that were originally created by Thanos will not include their external labels in the results when queried by Grafana Mimir.
If you need to have external labels in your query results, this is currently not possible to achieve in Grafana Mimir.

```bash
docker run grafana/metaconvert -backend=gcs -gcs.bucket-name=bucket -tenant=anonymous -dry-run
```
**Grafana Mimir will not respect deduplication labels configured in Thanos when querying the blocks.**
For the best query performance please only upload Thanos blocks from single Prometheus replica from each HA pair.
If you upload blocks from both replicas, the query results returned by Mimir will include samples from both replicas.

- If you are using a local binary, run the following command:
> **Note**: Thanos provides `thanos tools bucket rewrite` tool for manipulating blocks in the bucket.
> It may be possible to use this tool to embed external labels into blocks.
> Please refer to [`thanos tools bucket rewrite` documentation](https://thanos.io/tip/components/tools.md/#bucket-rewrite) for more details.
```bash
./metaconvert -backend=filesystem -filesystem.dir=/bucket -tenant=anonymous -dry-run
```
**Grafana Mimir does not support Thanos’ _downsampling_ feature.**
To guarantee query results correctness please only upload original (raw) Thanos blocks into Mimir's storage.
If you also upload blocks with downsampled data (ie. blocks with non-zero `Resolution` field in `meta.json` file), Grafana Mimir will merge raw samples and downsampled samples together at the query time.
This may cause that incorrect results are returned for the query.

1. Remove the `-dry-run` flag to apply the migration.
1. Verify the migration by re-running the tool with `-dry-run` and confirming that no changes are required.
> **Note**: It is possible to run compaction and deduplicate blocks by Thanos first by using `thanos compact` command
> with `--compact.enable-vertical-compaction --deduplication.func=penalty --deduplication.replica-label=<LABEL>` flags.
> Please refer to [Vertical Compaction Use Cases](https://thanos.io/tip/components/compact.md/#vertical-compaction-use-cases) for more details on offline deduplication.

0 comments on commit 78f18f5

Please sign in to comment.