Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(blooms): compute chunks once #12664

Merged
merged 27 commits into from
Apr 29, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
4807f48
remove unnecessary fallback
owen-d Mar 27, 2024
48402d2
[wip] wiring up support to pass store chunks to queriers
owen-d Mar 28, 2024
bb283f1
Merge remote-tracking branch 'upstream/main' into blooms/compute-chun…
owen-d Mar 29, 2024
7a05032
[wip] threading through store overrides for chunkrefs
owen-d Mar 29, 2024
97bc139
multi-tenant querier partitions store overrides by tenant id
owen-d Mar 29, 2024
3858893
metrics & ifc alignment
owen-d Mar 31, 2024
ea68788
Merge remote-tracking branch 'upstream/main' into blooms/compute-chun…
owen-d Apr 17, 2024
46d85f9
remove unused fn
owen-d Apr 17, 2024
db62dd8
send chunks in shards resp
owen-d Apr 17, 2024
ad855e6
type alignment
owen-d Apr 17, 2024
93a9ce4
Merge remote-tracking branch 'upstream/main' into blooms/compute-chun…
owen-d Apr 17, 2024
6ce86b4
type alignment
owen-d Apr 17, 2024
294261d
ShardsResponse.Merge extension
owen-d Apr 17, 2024
d7f2af9
fix unrelated codec test err msg
owen-d Apr 17, 2024
e8f58f5
tidy
owen-d Apr 17, 2024
4639cfd
binding shard to chunk refs
owen-d Apr 18, 2024
a764c11
simplify+pointer for shard chunks
owen-d Apr 18, 2024
3ba9330
fix signature
owen-d Apr 18, 2024
b2990bf
precomputed chunk logging
owen-d Apr 19, 2024
141c4f7
log matchers & always use mutex while accumulating chunks to shards
owen-d Apr 19, 2024
8db855d
more logging
owen-d Apr 19, 2024
8bdb823
better logging for gateway.go
owen-d Apr 22, 2024
56eabcc
independent handling for precomputed chunks vs bloom enablement optio…
owen-d Apr 23, 2024
33c8e82
Merge remote-tracking branch 'upstream/main' into blooms/compute-chun…
owen-d Apr 23, 2024
a3bd99c
make doc
owen-d Apr 23, 2024
82923e7
pr feedback
owen-d Apr 29, 2024
7b2f72e
pr feedback: only dispatch to bloom querier when line filters exist
owen-d Apr 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[wip] wiring up support to pass store chunks to queriers
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
  • Loading branch information
owen-d committed Mar 28, 2024
commit 48402d2be7521a3b7f0c197189e8c957777a2a48
1 change: 1 addition & 0 deletions pkg/logcli/client/file.go
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ func (f *FileClient) Query(q string, limit int, t time.Time, direction logproto.
direction,
uint32(limit),
nil,
nil,
)
if err != nil {
return nil, fmt.Errorf("failed to parse query: %w", err)
Expand Down
2 changes: 2 additions & 0 deletions pkg/logcli/query/query.go
Original file line number Diff line number Diff line change
Expand Up @@ -490,6 +490,7 @@ func (q *Query) DoLocalQuery(out output.LogOutput, statistics bool, orgID string
q.resultsDirection(),
uint32(q.Limit),
nil,
nil,
)
if err != nil {
return err
Expand All @@ -506,6 +507,7 @@ func (q *Query) DoLocalQuery(out output.LogOutput, statistics bool, orgID string
q.resultsDirection(),
uint32(q.Limit),
nil,
nil,
)
if err != nil {
return err
Expand Down
2 changes: 1 addition & 1 deletion pkg/logcli/query/query_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -426,7 +426,7 @@ func (t *testQueryClient) Query(_ string, _ int, _ time.Time, _ logproto.Directi
func (t *testQueryClient) QueryRange(queryStr string, limit int, from, through time.Time, direction logproto.Direction, step, interval time.Duration, _ bool) (*loghttp.QueryResponse, error) {
ctx := user.InjectOrgID(context.Background(), "fake")

params, err := logql.NewLiteralParams(queryStr, from, through, step, interval, direction, uint32(limit), nil)
params, err := logql.NewLiteralParams(queryStr, from, through, step, interval, direction, uint32(limit), nil, nil)
if err != nil {
return nil, err
}
Expand Down
185 changes: 136 additions & 49 deletions pkg/logproto/indexgateway.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 7 additions & 1 deletion pkg/logproto/indexgateway.proto
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ message ShardsRequest {
(gogoproto.jsontag) = "through"
];
string query = 3 [(gogoproto.jsontag) = "query"];
uint64 target_bytes_per_shard = 4 [(gogoproto.jsontag) = "targetBytesPerShard"];
uint64 targetBytesPerShard = 4 [(gogoproto.jsontag) = "targetBytesPerShard"];
}

message ShardsResponse {
Expand All @@ -57,6 +57,12 @@ message ShardsResponse {
(gogoproto.nullable) = false,
(gogoproto.jsontag) = "statistics"
];
// If present, includes the chunk refs for each shard,
// therefore the length will be equal to the length of the `shards` field.
repeated logproto.ChunkRefGroup chunkGroups = 3 [
(gogoproto.nullable) = false,
(gogoproto.jsontag) = "chunkGroups"
];
}

message Shard {
Expand Down
Loading