Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sidecar: Do not crash when Object Storage is not accessible #7585

Open
ahurtaud opened this issue Aug 2, 2024 · 2 comments
Open

sidecar: Do not crash when Object Storage is not accessible #7585

ahurtaud opened this issue Aug 2, 2024 · 2 comments
Labels
bug component: sidecar

Comments

@ahurtaud
Copy link
Contributor

ahurtaud commented Aug 2, 2024 •
edited
Loading

Is your proposal related to a problem?

Also related to objstore project.

We had a network outage accessing our storage endpoint. (DNS failure)
when sidecar restarted it then go into crashloop with :

ts=2024-08-02T08:22:02.642324362Z caller=main.go:145 level=error err="
Get \"https://<redacted>.privatelink.blob.core.windows.net/<container>?restype=container\": dial tcp: lookup <redacted>.privatelink.blob.core.windows.net on xx.xx.xx.xx:53: no such host\ncreate AZURE client\ngithub.com/thanos-io/objstore/client.NewBucket
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20240309075357-e8336a5fd5f3/client/factory.go:90\nmain.runSidecar
	/app/cmd/thanos/sidecar.go:327\nmain.registerSidecar.func1
	/app/cmd/thanos/sidecar.go:104\nmain.main
	/app/cmd/thanos/main.go:143\nruntime.main
	/usr/local/go/src/runtime/proc.go:267\nruntime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650\npreparing sidecar command failed\nmain.main
	/app/cmd/thanos/main.go:145\nruntime.main
	/usr/local/go/src/runtime/proc.go:267\nruntime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650"

While we consider objectstorage for long term metrics only, we would like sidecar to continue to serve prometheus read path and not crash.

Describe the solution you'd like

Could this error become a warning. And we would alert on a failing metrics or so instead of crashing.

Additional context

Thanos v0.35.0
ObjStore Azure
@yeya24
Copy link
Contributor

yeya24 commented Aug 3, 2024

I think it is a valid issue. Help wanted.

amaury-d added a commit to amaury-d/thanos that referenced this issue Sep 6, 2024
@amaury-d
fix(sidecar): Allow sidecar to not crash on bucket initialization tha…
ddb694d 
…nos-io#7585

The goal is to allow sidecar to start to continue to serve prometheus read path if objstore is not available at startup.
Bucket creation will be attempted again on next upload.

This commit brings a new counter to alert in case of buck initialization crash:
`thanos_sidecar_upload_failures_total{reason="bucket_initialization"}`

Signed-off-by: Amaury Decrême <amaury.decreme@gmail.com>
amaury-d added a commit to amaury-d/thanos that referenced this issue Sep 6, 2024
@amaury-d
fix(sidecar): Allow sidecar to not crash on bucket initialization tha…
c1b9766 
…nos-io#7585

The goal is to allow sidecar to start to continue to serve prometheus read path if objstore is not available at startup.
Bucket creation will be attempted again on next upload.

This commit brings a new counter to alert in case of buck initialization crash:
`thanos_sidecar_upload_failures_total{reason="bucket_initialization"}`

Signed-off-by: Amaury Decrême <amaury.decreme@gmail.com>
amaury-d added a commit to amaury-d/thanos that referenced this issue Sep 9, 2024
@amaury-d
fix(sidecar): Allow sidecar to not crash on bucket initialization tha…
6b84452 
…nos-io#7585

The goal is to allow sidecar to start to continue to serve prometheus read path if objstore is not available at startup.
Bucket creation will be attempted again on next upload.

This commit brings a new counter to alert in case of buck initialization crash:
`thanos_sidecar_upload_failures_total{reason="bucket_initialization"}`

Signed-off-by: Amaury Decrême <amaury.decreme@gmail.com>
amaury-d added a commit to amaury-d/thanos that referenced this issue Sep 9, 2024
@amaury-d
fix(sidecar): Allow sidecar to not crash on bucket initialization tha…
002a7fd 
…nos-io#7585

The goal is to allow sidecar to start to continue to serve prometheus read path if objstore is not available at startup.
Bucket creation will be attempted again on next upload.

This commit brings a new counter to alert in case of buck initialization crash:
`thanos_sidecar_upload_failures_total{reason="bucket_initialization"}`

Signed-off-by: Amaury Decrême <amaury.decreme@gmail.com>
amaury-d added a commit to amaury-d/thanos that referenced this issue Sep 10, 2024
@amaury-d
fix(sidecar): Allow sidecar to not crash on bucket initialization tha…
6ec7fdc 
…nos-io#7585

This commit allows sidecar to continue to serve prometheus read path if objstore is not available at startup.
Bucket creation will be attempted again on next upload.

This commit brings a new metric to alert in case of bucket initialization crash: thanos_sidecar_shipper_up

Signed-off-by: Amaury Decrême <amaury.decreme@gmail.com>
amaury-d added a commit to amaury-d/thanos that referenced this issue Sep 10, 2024
@amaury-d
fix(sidecar): Allow sidecar to not crash on bucket initialization tha…
10349bf 
…nos-io#7585

This commit allows sidecar to continue to serve prometheus read path if objstore is not available at startup.
Bucket creation will be attempted again on next upload.

This commit brings a new metric to alert in case of bucket initialization crash: thanos_sidecar_shipper_up

Signed-off-by: Amaury Decrême <amaury.decreme@gmail.com>
@amaury-d
Copy link

amaury-d commented Sep 19, 2024 •
edited
Loading

After a discussion with @MichaHoffmann, we came to realise that sidecar crashing can be useful for some users that rely on it to "detect" when something is wrong (like an uninitialised S3 bucket).

While it was suggested to add a metric to alert on the situation, such situations go could unnoticed.

I suggest to let sidecar crash by default and add an option to allow sidecar to continue to serve prometheus read path even if the objstore is not working.

amaury-d added a commit to amaury-d/thanos that referenced this issue Sep 24, 2024
@amaury-d
fix(sidecar): Add option "--shipper.retry-init" to sidecar thanos-io#…
d5b1fa3 
…7585

Signed-off-by: Amaury Decrême <amaury.decreme@gmail.com>
amaury-d added a commit to amaury-d/thanos that referenced this issue Sep 24, 2024
@amaury-d
fix(sidecar): Allow sidecar to not crash on shipper init thanos-io#7585
5f774ea 
This commit allows sidecar to continue to serve prometheus read path if objstore is not available at startup.
Bucket creation will be attempted again on next upload.

This option is disabled by default and can be enabled by passing
argument "--shipper.retry-init".

This commit also brings a new metric to alert in case of bucket initialization crash: thanos_sidecar_shipper_up

Signed-off-by: Amaury Decrême <amaury.decreme@gmail.com>
amaury-d added a commit to amaury-d/thanos that referenced this issue Sep 26, 2024
@amaury-d
fix(sidecar): Allow sidecar to not crash on shipper init thanos-io#7585
42b6661 
This commit allows sidecar to continue to serve prometheus read path if objstore is not available at startup.
Bucket creation will be attempted again on next upload.

This option is disabled by default and can be enabled by passing
argument "--shipper.retry-init".

This commit also brings a new metric to alert in case of bucket initialization crash: thanos_sidecar_shipper_up

Signed-off-by: Amaury Decrême <amaury.decreme@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug component: sidecar
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@amaury-d @ahurtaud @yeya24