Thanos Side Endpoints Not Showing on Querier

[kimcagayat]

Thanos, Prometheus and Golang version used:

Thanos:
Version 0.12.2
Revision 52e10c6
Branch HEAD
BuildUser circleci@241aa351893e
BuildDate 20200430-16:37:24
GoVersion go1.13.1

Prometheus:
Version 2.25.0
Revision a6be548dbc17780d562a39c0e4bd0bd4c00ad6e2
Branch HEAD
BuildUser root@615f028225c9
BuildDate 20210217-14:17:24
GoVersion go1.15.8

Object Storage Provider:
Azure Storage
What happened:
Thanos Side Endpoints Not Showing on Querier. Seeing below errors from one of the end points that is not showing:

level=warn ts=2022-08-12T02:01:45.672722834Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 172.29.180.4:10901: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=172.29.180.4:10901
level=warn ts=2022-08-12T02:01:50.674126075Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 172.29.180.4:10901: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=172.29.180.4:10901
level=warn ts=2022-08-12T02:01:55.675674611Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 172.29.180.4:10901: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=172.29.180.4:10901
level=warn ts=2022-08-12T02:02:00.67664108Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 172.29.180.4:10901: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=172.29.180.4:10901
level=warn ts=2022-08-12T02:02:05.685128107Z caller=storeset.go:440 component=storeset msg="update of store node failed" err="getting metadata: fetching store info from 172.29.180.4:10901: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=172.29.180.4:10901

What you expected to happen:
Thanos sidecar showul be showing on querier. Only some are showing. Not all.

[matej-g]

Are you using 0.12.2? That seems like a rather outdated version. Besides that, this seems to be a connectivity issue where querier cannot reach the store or the connection times out?

[kimcagayat]

hi @matej-g, yes but I already upgraded the thanos-sidecar, store, compactor and querier to Image: quay.io/thanos/thanos:v0.27.0 just now. Except on thanos bucket because I am receiving a "crashloopback" error so I revert the bucket to v0.12.2. Do you know if I did it correctly on the image upgrade? I just used kubectl edit deploy/statefulset command to update the thanos images.

[matej-g]

Hm so I'm not sure on you're deployment / roll out strategy, normally bumping the image should be fine. By Thanos bucket, do you mean the Thanos tool or something else?

Either way, I'm wondering if you'll be able to still replicate the error after you'll successfully roll out the new version.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.