Uptick of FLOW_CONTROL_ERROR errors in CI after updating to 1.66.0

[stanhu]

What version of gRPC are you using?

1.66.0

What version of Go are you using (go version)?

go version go1.22.6 linux/amd64

What operating system (Linux, Windows, …) and version?

Linux Docker container based on Debian bookworm

What did you do?

We recently updated gprc-go from 1.65.0 to 1.66.0. Our Ruby client runs the grpc gem 1.63.0.

What did you expect to see?

No issues.

What did you see instead?

I'm not sure how to reproduce this yet. A number of our tests in CI failed intermittently with this error:

     GRPC::ResourceExhausted:
       8:stream terminated by RST_STREAM with error code: FLOW_CONTROL_ERROR. debug_error_string:{UNKNOWN:Error received from peer unix:tmp/tests/gitaly/praefect.socket {grpc_message:"stream terminated by RST_STREAM with error code: FLOW_CONTROL_ERROR", grpc_status:8, created_time:"2024-09-09T11:44:10.228551833+00:00"}}

Full backtrace:

  1) MergeRequestDiff validations 
     Failure/Error: stub(service, storage).__send__(rpc, request, kwargs) # rubocop:disable GitlabSecurity/PublicSend
     GRPC::ResourceExhausted:
       8:stream terminated by RST_STREAM with error code: FLOW_CONTROL_ERROR. debug_error_string:{UNKNOWN:Error received from peer unix:tmp/tests/gitaly/praefect.socket {grpc_message:"stream terminated by RST_STREAM with error code: FLOW_CONTROL_ERROR", grpc_status:8, created_time:"2024-09-09T11:44:10.228551833+00:00"}}
     # ./lib/gitlab/gitaly_client.rb:292:in `execute'
     # ./lib/gitlab/gitaly_client/call.rb:18:in `block in call'
     # ./lib/gitlab/gitaly_client/call.rb:60:in `recording_request'
     # ./lib/gitlab/gitaly_client/call.rb:17:in `call'
     # ./lib/gitlab/gitaly_client.rb:281:in `call'
     # ./lib/gitlab/gitaly_client/with_feature_flag_actors.rb:31:in `block in gitaly_client_call'
     # ./lib/gitlab/gitaly_client.rb:631:in `with_feature_flag_actors'
     # ./lib/gitlab/gitaly_client/with_feature_flag_actors.rb:25:in `gitaly_client_call'
     # ./lib/gitlab/gitaly_client/repository_service.rb:383:in `gitaly_repo_stream_request'
     # ./lib/gitlab/gitaly_client/repository_service.rb:210:in `create_from_bundle'
     # ./lib/gitlab/git/repository.rb:954:in `create_from_bundle'
     # ./spec/factories/projects.rb:505:in `block (4 levels) in <top (required)>'
     # ./spec/models/merge_request_diff_spec.rb:10:in `block (2 levels) in <top (required)>'
     # ./spec/models/merge_request_diff_spec.rb:13:in `block (3 levels) in <top (required)>'
     # ./spec/models/merge_request_diff_spec.rb:15:in `block (3 levels) in <top (required)>'
     # ./spec/spec_helper.rb:471:in `block (3 levels) in <top (required)>'
     # ./lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'
     # ./spec/spec_helper.rb:470:in `block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:465:in `block (3 levels) in <top (required)>'
     # ./spec/support/sidekiq_middleware.rb:9:in `with_sidekiq_server_middleware'
     # ./spec/spec_helper.rb:456:in `block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:452:in `block (3 levels) in <top (required)>'
     # ./lib/gitlab/application_context.rb:92:in `with_raw_context'
     # ./spec/spec_helper.rb:452:in `block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:268:in `block (2 levels) in <top (required)>'
     # ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <top (required)>'
     # ./spec/support/fast_quarantine.rb:22:in `block (2 levels) in <top (required)>'
     # ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <top (required)>'
     # ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
     # ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <top (required)>'

[arjan-bal]

Hi stanhu@, it difficult to find a root cause with the limited information. I have a few questions to understand the issue better:

1. Did rolling back the server to gRPC Go 1.65 resolve the issue?
2. What are the kind of tests are you running? What is the network setup? For example, are client and server on the same machine/k8s pod/k8s cluster?
3. How much did the error rate deviate from usual? Was it a one time occurrence or was the uptick constant?
4. Did you see any increase in CPU/memory usage or performance degradation of the server that could indicate it was under high load?

[arjan-bal]

Also, if you can provide a way to repro the issue, investigation would become much easier.

[stanhu]

I'm going to close this out until we have more info here. It's possible our tests aren't cleaning up properly.

[stanhu]

To circle back here, I think #7667 might have solved this problem.

[ukai]

grpc-go 1.67.1 (both client, server) still getting the error...
https://ci.chromium.org/ui/p/chromium/builders/ci/Win%20Builder%20(dbg)/187249/overview

[arjan-bal]

@ukai is it possible that either the client or the server is still using a gRPC version with the bug? I don't understand the test environment, but maybe the proxy or server is still using an older version?
Effected versions: 1.66.0. 1.66.1, 1.66.2, 1.67.0.