kv/kvserver: TestReplicaCircuitBreaker_Partial_Retry failed

[cockroach-teamcity]

kv/kvserver.TestReplicaCircuitBreaker_Partial_Retry failed on release-24.3.26-rc @ 77aa3fcad169b09166b9105d52c05ab0122988d1:

        	Error Trace:	pkg/kv/kvserver/client_replica_circuit_breaker_test.go:837
        	            				pkg/kv/kvserver/client_replica_circuit_breaker_test.go:900
        	Error:      	Received unexpected error:
        	            	replica unavailable: (n3,s3):3 unable to serve request to r70:/{Table/Max-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=5, sticky=9223372036.854775807,2147483647]: closed timestamp: 1770269062.130348275,0 (2026-02-05 05:24:22); raft status: {"id":"3","term":7,"vote":"3","commit":25,"lead":"3","leadEpoch":"0","raftState":"StateLeader","applied":25,"progress":{"1":{"match":25,"next":26,"state":"StateReplicate"},"2":{"match":25,"next":26,"state":"StateReplicate"},"3":{"match":25,"next":26,"state":"StateReplicate"}},"leadtransferee":"0"}: injected error
        	            	(1) forced error mark
        	            	  | "originated at breaker breaker"
        	            	  | github.com/cockroachdb/cockroach/pkg/util/circuit/*circuit.breakerErrorMark::
        	            	Wraps: (2) forced error mark
        	            	  | "breaker open"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (3) replica unavailable: (n3,s3):3 unable to serve request to r70:/{Table/Max-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=5, sticky=9223372036.854775807,2147483647]: closed timestamp: 1770269062.130348275,0 (2026-02-05 05:24:22); raft status: {"id":"3","term":7,"vote":"3","commit":25,"lead":"3","leadEpoch":"0","raftState":"StateLeader","applied":25,"progress":{"1":{"match":25,"next":26,"state":"StateReplicate"},"2":{"match":25,"next":26,"state":"StateReplicate"},"3":{"match":25,"next":26,"state":"StateReplicate"}},"leadtransferee":"0"}: injected error
        	            	Wraps: (4)
        	            	  | (opaque error wrapper)
        	            	  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
        	            	  | reportable 0:
        	            	  |
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.replicaUnavailableError
        	            	  | 	pkg/kv/kvserver/replica_circuit_breaker.go:268
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).replicaUnavailableError
        	            	  | 	pkg/kv/kvserver/replica_circuit_breaker.go:277
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaCircuitBreaker).tripSync
        	            	  | 	pkg/kv/kvserver/replica_circuit_breaker.go:97
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).TripBreaker
        	            	  | 	pkg/kv/kvserver/helpers_test.go:602
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestReplicaCircuitBreaker_Partial_Retry
        	            	  | 	pkg/kv/kvserver/client_replica_circuit_breaker_test.go:868
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1689
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (5) closed timestamp: 1770269062.130348275,0 (2026-02-05 05:24:22); raft status: {"id":"3","term":7,"vote":"3","commit":25,"lead":"3","leadEpoch":"0","raftState":"StateLeader","applied":25,"progress":{"1":{"match":25,"next":26,"state":"StateReplicate"},"2":{"match":25,"next":26,"state":"StateReplicate"},"3":{"match":25,"next":26,"state":"StateReplicate"}},"leadtransferee":"0"}
        	            	Wraps: (6)
        	            	  | (opaque error wrapper)
        	            	  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
        	            	  | reportable 0:
        	            	  |
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).TripBreaker
        	            	  | 	pkg/kv/kvserver/helpers_test.go:602
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestReplicaCircuitBreaker_Partial_Retry
        	            	  | 	pkg/kv/kvserver/client_replica_circuit_breaker_test.go:868
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1689
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (7) injected error
        	            	Error types: (1) *markers.withMark (2) *markers.withMark (3) *kvpb.ReplicaUnavailableError (4) *errbase.opaqueWrapper (5) *errutil.withPrefix (6) *errbase.opaqueWrapper (7) *errutil.leafError
        	Test:       	TestReplicaCircuitBreaker_Partial_Retry
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestReplicaCircuitBreaker_Partial_Retry2420737363
--- FAIL: TestReplicaCircuitBreaker_Partial_Retry (5.41s)

Parameters:

• attempt=1
• run=5
• shard=26

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-59516