Skip to content

Fix updateCachedShippedBlocks - Thanos Update #4806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

alanprot
Copy link
Member

@alanprot alanprot commented Jul 28, 2022

What this PR does:
Fixing updateCachedShippedBlocks function.

With the thanos update, we started to see the following error:

level=error ts=2022-07-28T02:35:59.814001241Z caller=ingester_v2.go:1753 org_id=blah msg="failed to update cached shipped blocks after shipper initialisation" err="failed to read /data/tsdb/blah/thanos.shipper.json: open /data/tsdb/blah/thanos.shipper.json: no such file or directory"

As thanos changed the error type:

thanos-io/thanos#5174

It does not seems to be a big issue (or not a issue at all) as when the file does not exists there is nothing to update anyway - but the log line is ugly!

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@alanprot alanprot force-pushed the fix/updateCachedShippedBlocks branch from 4ef04f4 to 8afe1f3 Compare July 28, 2022 03:18
Signed-off-by: Alan Protasio <approtas@amazon.com>
@alanprot alanprot force-pushed the fix/updateCachedShippedBlocks branch from 8afe1f3 to ca37194 Compare July 28, 2022 03:19
@alanprot alanprot marked this pull request as ready for review July 28, 2022 03:20
@alanprot alanprot merged commit 427679e into cortexproject:master Jul 28, 2022
@alanprot alanprot deleted the fix/updateCachedShippedBlocks branch July 28, 2022 03:54
alexqyle pushed a commit to alexqyle/cortex that referenced this pull request Aug 2, 2022
Signed-off-by: Alan Protasio <approtas@amazon.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
alanprot added a commit that referenced this pull request Sep 3, 2022
* Introduced lock file to shuffle sharding grouper

Signed-off-by: Alex Le <leqiyue@amazon.com>

* let redis cache logs log with context (#4785)

* let redis cache logs log with context

Signed-off-by: Mengmeng Yang <mengmengyang616@gmail.com>

* fix import

Signed-off-by: Mengmeng Yang <mengmengyang616@gmail.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>

* DoBatch preference to 4xx if error (#4783)

* DoBatch preference to 4xx if error

Signed-off-by: Daniel Blando <ddeluigg@amazon.com>

* Fix comment

Signed-off-by: Daniel Blando <ddeluigg@amazon.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>

* Updated CHANGELOG and ordered imports

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Fixed lint and removed groupCallLimit

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Changed lock file to json format and make sure planner would not pick up group that is locked by other compactor

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Fix updateCachedShippedBlocks - new thanos (#4806)

Signed-off-by: Alan Protasio <approtas@amazon.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>

* Join memberlist on starting with no retry (#4804)

Signed-off-by: Daniel Blando <ddeluigg@amazon.com>

* Fix alertmanager log message (#4801)

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>

* Grafana Cloud uses Mimir now, so remove Grafana Cloud as hosted service in documents (#4809)

* Grafana Cloud uses Mimir, for of Cortex, now

Signed-off-by: Alvin Lin <alvinlin123@gmail.com>

* Improve doc

Signed-off-by: Alvin Lin <alvinlin@amazon.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>

* Created block_locker to handle all block lock file operations. Added block lock metrics.

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Moved lock file heart beat into planner and refined planner logic to make sure blocks are locked by current compactor

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Updated documents

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Return concurrency number of group. Use ticker for lock file heart beat

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Renamed lock file to be visit marker file

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Fixed unit test

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Make sure visited block can be picked by compactor visited it

Signed-off-by: Alex Le <leqiyue@amazon.com>

Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: Mengmeng Yang <mengmengyang616@gmail.com>
Signed-off-by: Daniel Blando <ddeluigg@amazon.com>
Signed-off-by: Alan Protasio <approtas@amazon.com>
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
Signed-off-by: Alvin Lin <alvinlin@amazon.com>
Signed-off-by: Alex Le <emoc1989@gmail.com>
Co-authored-by: Mengmeng Yang <mengmengyang616@gmail.com>
Co-authored-by: Daniel Blando <ddeluigg@amazon.com>
Co-authored-by: Alan Protasio <approtas@amazon.com>
Co-authored-by: Xiaochao Dong <the.xcdong@gmail.com>
Co-authored-by: Alvin Lin <alvinlin@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants