Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(blooms): bloom integration in query planning #12208

Merged
merged 72 commits into from
Mar 26, 2024
Merged

feat(blooms): bloom integration in query planning #12208

merged 72 commits into from
Mar 26, 2024

Commits on Mar 14, 2024

  1. streaming GetShards on idx-gw + some refactoring 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    b831d07 View commit details
    Browse the repository at this point in the history

  2. moves FingerprintBounds to logproto pkg to avoid circular imports 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    91ee79e View commit details
    Browse the repository at this point in the history

  3. proto alignment, wiring GetShards() through storage 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    2db2852 View commit details
    Browse the repository at this point in the history

  4. tsdb index ForSeries() can signal a stop to iteration 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    6801fbb View commit details
    Browse the repository at this point in the history

  5. ForSeries in tsdb.Index ifc 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    c2200a0 View commit details
    Browse the repository at this point in the history

  6. pr feedback 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    db1c830 View commit details
    Browse the repository at this point in the history

  7. removes pointers in protos 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    693c48b View commit details
    Browse the repository at this point in the history

  8. gateway GetShards impl 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    808d88b View commit details
    Browse the repository at this point in the history

  9. [wip] integrating bound ranges into GetShards() 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    391093f View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2024

  1. generic result accumulator 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    42dccdd View commit details
    Browse the repository at this point in the history

  2. indexclient getshards impl 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    cc105e0 View commit details
    Browse the repository at this point in the history

  3. comment 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    b355b9f View commit details
    Browse the repository at this point in the history

  4. thread plan+predicate 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    14e4815 View commit details
    Browse the repository at this point in the history

  5. [wip] HasForSeries wiring 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    c42344e View commit details
    Browse the repository at this point in the history

  6. finish HasForSeries impl, move ifc to index pkg 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    3052e2f View commit details
    Browse the repository at this point in the history

  7. moves HasForSeries to StatsReader 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    548fe30 View commit details
    Browse the repository at this point in the history

  8. new sharding pkg 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    272e06f View commit details
    Browse the repository at this point in the history

  9. removes bounds from shardsrequest 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    cfb38a5 View commit details
    Browse the repository at this point in the history

  10. HasForSeries accepts timerange + composite store impl 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    8988232 View commit details
    Browse the repository at this point in the history

  11. resultAccumulator doesnt run merge for single list 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    300d710 View commit details
    Browse the repository at this point in the history

  12. sorts MultiIndex.GetChunkRefs during merge 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    1ea82b1 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2024

  1. naive implementation of bloom-accelerated sharding calculation in gat… 

    …eway

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    6bad531 View commit details
    Browse the repository at this point in the history

  2. extracts & tests accumulating chunks to shards 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    09f1618 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2024

  1. moves common sharding logic to shared pkg, lets idx-gw fallback to St… 

    …ats when resolving shards

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    6b8feac View commit details
    Browse the repository at this point in the history

  2. GatewayClient.poolDoWithStrategy to avoid cascading requests during r… 

    …ollout

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    3629b7b View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2024

  1. shardsrequest compat.go 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    a6e0506 View commit details
    Browse the repository at this point in the history

  2. shardsresonse extensions.go 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    1961eb8 View commit details
    Browse the repository at this point in the history

  3. removes plan from ShardsRequest in favor of a query string which can … 

    …be more effectively controlled

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    27599b7 View commit details
    Browse the repository at this point in the history

  4. queryrange+codec shardrequesting support + some cleanup to use query … 

    …string instead of plan

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    fdbe1fa View commit details
    Browse the repository at this point in the history

  5. ShardResolver.GetShardingRanges support 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    42fe9cf View commit details
    Browse the repository at this point in the history

  6. querier, http wiring for shards request 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    3704abc View commit details
    Browse the repository at this point in the history

  7. json tags for shard protos 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    8854fb4 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2024

  1. [unrelated] fixes double-stats-requesting on quantiles 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    32d400d View commit details
    Browse the repository at this point in the history

  2. multiple logql shard reprs 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    f8ff1fb View commit details
    Browse the repository at this point in the history

  3. logql sharding strategy 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    8118844 View commit details
    Browse the repository at this point in the history

  4. test alignment 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    21a2edd View commit details
    Browse the repository at this point in the history

  5. shard parsing returns verison 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    b9f6121 View commit details
    Browse the repository at this point in the history

  6. integrating new shards into label injection 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    29cb72a View commit details
    Browse the repository at this point in the history

  7. aligns inclusivity expectations between sharding impls for Fingerprin… 

    …tFilter interface

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    7bb381f View commit details
    Browse the repository at this point in the history

  8. integrates shard-bounds into ingesters 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    b5f2501 View commit details
    Browse the repository at this point in the history

  9. tsdb sharding strategy 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    e54db18 View commit details
    Browse the repository at this point in the history

  10. ingester fixes 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    48258a0 View commit details
    Browse the repository at this point in the history

  11. fix test + docs 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    e7b4930 View commit details
    Browse the repository at this point in the history

  12. Merge remote-tracking branch 'upstream/main' into blooms/query-planni… 

    …ng-i

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    f476128 View commit details
    Browse the repository at this point in the history

  13. fix ifc alignment 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    f4b4e86 View commit details
    Browse the repository at this point in the history

  14. gen files 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    189b4e3 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. linting, circular imports, default impls for mocks 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    91277c8 View commit details
    Browse the repository at this point in the history

  2. preserve old indexreaderwriter behavior 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    3f14f1a View commit details
    Browse the repository at this point in the history

  3. goimports -s 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    eda6ef8 View commit details
    Browse the repository at this point in the history

  4. legacy index reader getshards 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    1f22cee View commit details
    Browse the repository at this point in the history

  5. adds pre+postfilter chunkref histograms to idx-gws bloom-accelerated … 

    …routes

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    f24c768 View commit details
    Browse the repository at this point in the history

  6. corrects shard routing in middleware 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    9049d97 View commit details
    Browse the repository at this point in the history

  7. proto response wrapping for new shards types 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    5d0d1a0 View commit details
    Browse the repository at this point in the history

  8. promauto for indexgateway metrics 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    66e5998 View commit details
    Browse the repository at this point in the history

  9. proto roundtripping + lint 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    5300648 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2024

  1. s/table-compaction-period/table-offset/g 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    5f82560 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into blooms/query-planni… 

    …ng-i

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    60b305b View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'upstream/main' into blooms/query-planni… 

    …ng-i

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    37edc8b View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. index statistics 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    b0350ae View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into blooms/query-planni… 

    …ng-i

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    5b70e96 View commit details
    Browse the repository at this point in the history

  3. stats cleanup and proper propagation 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    db506cb View commit details
    Browse the repository at this point in the history

  4. routing stats through store 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    5ca8d2f View commit details
    Browse the repository at this point in the history

  5. gatewayclient propagates stats 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    d43b3f7 View commit details
    Browse the repository at this point in the history

  6. logging fixes 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    52df554 View commit details
    Browse the repository at this point in the history

  7. Merge remote-tracking branch 'upstream/main' into blooms/query-planni… 

    …ng-i

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    ce33787 View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'upstream/main' into blooms/query-planni… 

    …ng-i

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    3e96e75 View commit details
    Browse the repository at this point in the history

  9. skip (unhealthy?) bloom-gws with no token range 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    1e77cee View commit details
    Browse the repository at this point in the history

  10. correctly merges stats in collector middleware for bloom data, some a… 

    …ddtl logging

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    fed5262 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2024

  1. v1 metrics wired up through read path 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 23, 2024
    Configuration menu
    Copy the full SHA
    2805339 View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2024

  1. optional pool support for bloom pages 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    11d8b42 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into blooms/query-planni… 

    …ng-i

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    ade14a9 View commit details
    Browse the repository at this point in the history

  3. test signature 

    Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
    @owen-d
    owen-d committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    463dafd View commit details
    Browse the repository at this point in the history