[Store] slice bounds out of range [:1029] with capacity 1024

[wiardvanrij]

: load chunks: populate chunk: allocate chunk bytes: pool exhausted"
panic: runtime error: slice bounds out of range [:1029] with capacity 1024

goroutine 29754135 [running]:
github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).loadChunks(0xc01aacecc0, {0x2b75f90, 0xc069448640}, {0xc08007e000, 0x1209, 0xc2077d7eb0?}, {0xc09bc44798, 0x1, 0x2}, 0x1be, ...)
    /app/pkg/store/bucket.go:3379 +0x11e5
github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).load.func4()
    /app/pkg/store/bucket.go:3307 +0xff
golang.org/x/sync/errgroup.(*Group).Go.func1()
    /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
    /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:72 +0xa5

Version: v0.32.2

I did check latest releases as I'm not running the latest, but didn't saw any reference to this. Hence the report.

[hczhu-db]

I encountered the same panic and found that the chunk pool returns a slice shorter than required.

panic: runtime error: slice bounds out of range [:2297] with capacity 2129

goroutine 4034779 [running]:
github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).loadChunks(0xc0914ad920, {0x35f4b28, 0xc091ff2a00}, {0xc093280000, 0x317, 0x317?}, {0xc0914b5968, 0x2, 0x2}, 0x29, ...)
        /thanos/thanos/pkg/store/bucket.go:3542 +0x1118
github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).load.func4()
        /thanos/thanos/pkg/store/bucket.go:3476 +0x11b
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 4034046
        /go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

[yeya24]

@hczhu-db What version of Thanos are you running?
I think we used to see this issue. However, I think after several rounds of fixes around some race conditions when fetching chunks, I haven't seen this issue for a while.

[hczhu-db]

@hczhu-db What version of Thanos are you running? I think we used to see this issue. However, I think after several rounds of fixes around some race conditions when fetching chunks, I haven't seen this issue for a while.

We are running 0.34.1 with our own changes on top of it. Those changes are mainly for observability (like new metrics and logs) and Compactor skipping overlapping block files.

@jnyi

[hczhu-db]

I also got the following error interleaving with slice bounds out of range error. More weirdly, the same range query can work or trigger either error if it's run multiple times.

MinTime: 1708874139848 MaxTime: 1711214139848: rpc error: code = Unknown desc = receive series from 01HSED6EP817R1VPXQCS73ZGDA: load chunks: populate chunk: allocate chunk bytes: pool exhausted