Skip to content

Data race in native histogram #1605

New issue
New issue
Closed
#1608
Closed
Data race in native histogram#1605
#1608
Assignees
krajorama
Labels
bug
@krajorama

Description

@krajorama
krajorama
opened on Aug 31, 2024

Version github.com/prometheus/client_golang v1.20.0

Capacity for exemplars is called without lock here:

client_golang/prometheus/histogram.go

Line 1685 in dbf72fc

if cap(n.exemplars) == 0 {

Which causes the race with

client_golang/prometheus/histogram.go

Line 1704 in dbf72fc

n.exemplars = append(n.exemplars[:nIdx], append([]*dto.Exemplar{e}, n.exemplars[nIdx:]...)...)

See logs from Mimir PR CI run:

22:12:09 query-frontend: ==================
22:12:09 query-frontend: WARNING: DATA RACE
22:12:09 query-frontend: Read at 0x00c001bba3d8 by goroutine 78:
22:12:09 query-frontend: github.com/prometheus/client_golang/prometheus.(*nativeExemplars).addExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:1685 +0x57
22:12:09 query-frontend: github.com/prometheus/client_golang/prometheus.(*histogram).updateExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:1140 +0x116
22:12:09 query-frontend: github.com/prometheus/client_golang/prometheus.(*histogram).ObserveWithExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:770 +0x69
22:12:09 query-frontend: github.com/grafana/dskit/instrument.ObserveWithExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/instrument/instrument.go:76 +0x1a2
22:12:09 query-frontend: github.com/grafana/dskit/middleware.observe()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:34 +0x205
22:12:09 query-frontend: github.com/grafana/dskit/middleware.StreamServerInstrumentInterceptor.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:58 +0x18b
22:12:09 query-frontend: google.golang.org/grpc.getChainStreamHandler.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1525 +0x118
22:12:09 query-frontend: github.com/opentracing-contrib/go-grpc.OpenTracingStreamServerInterceptor.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/opentracing-contrib/go-grpc/server.go:114 +0x4ae
22:12:09 query-frontend: google.golang.org/grpc.getChainStreamHandler.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1525 +0x118
22:12:09 query-frontend: github.com/grafana/dskit/middleware.GRPCServerLog.StreamServerInterceptor()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_logging.go:87 +0xbd
22:12:09 query-frontend: github.com/grafana/dskit/middleware.GRPCServerLog.StreamServerInterceptor-fm()
22:12:09 query-frontend: <autogenerated>:1 +0xbd
22:12:09 query-frontend: google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1516 +0xdc
22:12:09 query-frontend: google.golang.org/grpc.(*Server).processStreamingRPC()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1680 +0x1f8e
22:12:09 query-frontend: google.golang.org/grpc.(*Server).handleStream()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1794 +0x17c4
22:12:09 query-frontend: google.golang.org/grpc.(*Server).serveStreams.func2.1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1029 +0x158
22:12:09 query-frontend: google.golang.org/grpc.(*Server).serverWorker()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:637 +0x32
22:12:09 query-frontend: google.golang.org/grpc.(*Server).initServerWorkers.gowrap1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:650 +0x33
22:12:09 query-frontend: Previous write at 0x00c001bba3d8 by goroutine 73:
22:12:09 query-frontend: github.com/prometheus/client_golang/prometheus.(*nativeExemplars).addExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:1704 +0x54b
22:12:09 query-frontend: github.com/prometheus/client_golang/prometheus.(*histogram).updateExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:1140 +0x116
22:12:09 query-frontend: github.com/prometheus/client_golang/prometheus.(*histogram).ObserveWithExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:770 +0x69
22:12:09 query-frontend: github.com/grafana/dskit/instrument.ObserveWithExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/instrument/instrument.go:76 +0x1a2
22:12:09 query-frontend: github.com/grafana/dskit/middleware.observe()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:34 +0x205
22:12:09 query-frontend: github.com/grafana/dskit/middleware.StreamServerInstrumentInterceptor.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:58 +0x18b
22:12:09 query-frontend: google.golang.org/grpc.getChainStreamHandler.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1525 +0x118
22:12:09 query-frontend: github.com/opentracing-contrib/go-grpc.OpenTracingStreamServerInterceptor.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/opentracing-contrib/go-grpc/server.go:114 +0x4ae
22:12:09 query-frontend: google.golang.org/grpc.getChainStreamHandler.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1525 +0x118
22:12:09 query-frontend: github.com/grafana/dskit/middleware.GRPCServerLog.StreamServerInterceptor()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_logging.go:87 +0xbd
22:12:09 query-frontend: github.com/grafana/dskit/middleware.GRPCServerLog.StreamServerInterceptor-fm()
22:12:09 query-frontend: <autogenerated>:1 +0xbd
22:12:09 query-frontend: google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1516 +0xdc
22:12:09 query-frontend: google.golang.org/grpc.(*Server).processStreamingRPC()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1680 +0x1f8e
22:12:09 query-frontend: google.golang.org/grpc.(*Server).handleStream()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1794 +0x17c4
22:12:09 query-frontend: google.golang.org/grpc.(*Server).serveStreams.func2.1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1029 +0x158
22:12:09 query-frontend: google.golang.org/grpc.(*Server).serverWorker()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:637 +0x32
22:12:09 query-frontend: google.golang.org/grpc.(*Server).initServerWorkers.gowrap1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:650 +0x33
22:12:09 query-frontend: Goroutine 78 (running) created at:
22:12:09 query-frontend: google.golang.org/grpc.(*Server).initServerWorkers()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:650 +0x219
22:12:09 query-frontend: google.golang.org/grpc.NewServer()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:682 +0xdda
22:12:09 query-frontend: github.com/grafana/dskit/server.newServer()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/server/server.go:453 +0x3a9e
22:12:09 query-frontend: github.com/grafana/dskit/server.New()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/server/server.go:243 +0x97
22:12:09 query-frontend: github.com/grafana/mimir/pkg/mimir.(*Mimir).initServer()
22:12:09 query-frontend: /__w/mimir/mimir/pkg/mimir/modules.go:325 +0x3c4
22:12:09 query-frontend: github.com/grafana/mimir/pkg/mimir.(*Mimir).initServer-fm()
22:12:09 query-frontend: <autogenerated>:1 +0x33
22:12:09 query-frontend: github.com/grafana/dskit/modules.(*Manager).initModule()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:136 +0x35a
22:12:09 query-frontend: github.com/grafana/dskit/modules.(*Manager).InitModuleServices()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108 +0x14a
22:12:09 query-frontend: github.com/grafana/mimir/pkg/mimir.(*Mimir).Run()
22:12:09 query-frontend: /__w/mimir/mimir/pkg/mimir/mimir.go:851 +0x72c
22:12:09 query-frontend: main.main()
22:12:09 query-frontend: /__w/mimir/mimir/cmd/mimir/main.go:227 +0x1dd6
22:12:09 query-frontend: Goroutine 73 (running) created at:
22:12:09 query-frontend: google.golang.org/grpc.(*Server).initServerWorkers()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:650 +0x219
22:12:09 query-frontend: google.golang.org/grpc.NewServer()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:682 +0xdda
22:12:09 query-frontend: github.com/grafana/dskit/server.newServer()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/server/server.go:453 +0x3a9e
22:12:09 query-frontend: github.com/grafana/dskit/server.New()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/server/server.go:243 +0x97
22:12:09 query-frontend: github.com/grafana/mimir/pkg/mimir.(*Mimir).initServer()
22:12:09 query-frontend: /__w/mimir/mimir/pkg/mimir/modules.go:325 +0x3c4
22:12:09 query-frontend: github.com/grafana/mimir/pkg/mimir.(*Mimir).initServer-fm()
22:12:09 query-frontend: <autogenerated>:1 +0x33
22:12:09 query-frontend: github.com/grafana/dskit/modules.(*Manager).initModule()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:136 +0x35a
22:12:09 query-frontend: github.com/grafana/dskit/modules.(*Manager).InitModuleServices()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108 +0x14a
22:12:09 query-frontend: github.com/grafana/mimir/pkg/mimir.(*Mimir).Run()
22:12:09 query-frontend: /__w/mimir/mimir/pkg/mimir/mimir.go:851 +0x72c
22:12:09 query-frontend: main.main()
22:12:09 query-frontend: /__w/mimir/mimir/cmd/mimir/main.go:227 +0x1dd6
22:12:09 query-frontend: ==================

Metadata

Metadata

Assignees

Labels

bug

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions