Skip to content
This repository was archived by the owner on Jul 19, 2023. It is now read-only.
This repository was archived by the owner on Jul 19, 2023. It is now read-only.

Query from object store #717

@cyriltovena

Description

@cyriltovena

Currently we can only query data from ingesters, to query data from the storage we want to introduce a store-gateway that syncs blocks and cache respective metadata such as symbols, labels, etc..

### Tasks
- [x] Hook into the Pyroscope components.go
- [x] Add defaults config
- [x] Implements tenant listing
- [x] Implements block listing per tenant.
- [x] Speed block initial sync.
- [x] Partially load and reload blocks to disk
- [x] Investigate why opening block fails when using a remote bucket
- [x] Create query iterators.
- [x] Implement the Pyroscope Read API on the gateway side
- [x] Discover and use store gateway in the queriers.
- [x] Update helm chart
- [x] Parallel rollout (already the default)
- [x] <https://github.com/grafana/phlare/issues/731>
- [x] All bucket providers should have this fix https://github.com/simonswine/objstore/commit/890519321a99558ed34df71b72e5f3d313c403fd
- [x] Implement a component `store-gateway` with ring exposing read subset of the `IngesterService` interface for blocks from object store. I think this is best done by looking at https://github.com/grafana/mimir/blob/main/pkg/storegateway/gateway.go
- [x] Move POC block discovery into store-gatwaye code https://github.com/grafana/phlare/pull/636/files#diff-8d0d4c7633f85ca22a5b86e9557fcf05131f2a5f1620c70a5d53649513c28aa0R98
- [x] https://github.com/grafana/phlare/issues/734
- [x] Implement prefix match for object storage provider `GCS` https://github.com/simonswine/objstore/commit/890519321a99558ed34df71b72e5f3d313c403fd
- [x] Implement prefix match for object storage provider `S3`
- [x] Implement prefix match for object storage provider `Azure`
- [x] Add memcache to helm micro-services deployment (similar to how minio operates)
- [x] Contribute iter changes upstream to [github.com/thanos-io/objstore](https://github.com/thanos-io/objstore/tree/main/providers)
- [ ] https://github.com/grafana/phlare/issues/735
- [ ] https://github.com/grafana/phlare/issues/732
- [ ] https://github.com/grafana/pyroscope/issues/2052
- [ ] grafana/pyroscope#2045

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions