Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make boltdb-shipper use the generic indexshipper for managing index in object storage #6226

Merged

Conversation

sandeepsukhani
Copy link
Contributor

@sandeepsukhani sandeepsukhani commented May 23, 2022

What this PR does / why we need it:
Make boltdb-shipper use the generic indexshipper for managing the index on object storage.
I have also moved index gateway client and its config to indexshipper since it is supposed to be used with tsdb as well.

Special notes for your reviewer:
Since this PR was getting too big, I will open a follow-up PR for the following changes:

  1. Move code handling writes in pkg/storage/stores/shipper/index to pkg/storage/stores/shipper/index/writes package.
  2. See if we can write index gateway benchmark test in a reliable way which is commented out to avoid circular dependency.
  3. Add boltdb-shipper metrics to indexshipper and make it reusable for tsdb.
  4. Rename pkg/storage/stores/shipper to pkg/storage/stores/boltdbshipper.

Checklist

  • Tests updated

@sandeepsukhani sandeepsukhani force-pushed the boltdb-shipper-use-generic-shipper branch 2 times, most recently from b16502f to a849898 Compare May 24, 2022 12:55
@sandeepsukhani sandeepsukhani force-pushed the boltdb-shipper-use-generic-shipper branch from a849898 to 39f2c6a Compare May 24, 2022 13:18
@grafanabot
Copy link
Collaborator

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
-            storage	-0.2%
+           chunkenc	0%
+              logql	0%
+               loki	0%

@sandeepsukhani sandeepsukhani marked this pull request as ready for review May 24, 2022 13:39
@sandeepsukhani sandeepsukhani requested a review from a team as a code owner May 24, 2022 13:39
@sandeepsukhani sandeepsukhani changed the title first-cut changes for making boltdb-shipper use the generic indexshipper make boltdb-shipper use the generic indexshipper for managing index in object storage May 24, 2022
@grafanabot
Copy link
Collaborator

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
-            storage	-0.2%
+           chunkenc	0%
-              logql	-0.5%
+               loki	0%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants