Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RedisClient::ConnectionError Broken pipe #116

Closed
wnm opened this issue May 4, 2023 · 4 comments
Closed

RedisClient::ConnectionError Broken pipe #116

wnm opened this issue May 4, 2023 · 4 comments

Comments

@wnm
Copy link

wnm commented May 4, 2023

We exchanged our old redis setup:

@pool = ConnectionPool.new(size: 5, timeout: 5) { Redis.new(url: Settings.redis.url, ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }) }

with redis-client this morning:

redis_config = RedisClient.config(url: Settings.redis.url, ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE })
    @pool = redis_config.new_pool(timeout: 5, size: 5)

Most things work fine, but we got a few RedisClient::ConnectionError Broken pipe errors since upgrading. Any pointers on how to debug this, and what the reason could be?
@byroot
Copy link
Member

byroot commented May 4, 2023

Could you share the full backtrace for these errors?

@wnm
Copy link
Author

wnm commented May 4, 2023 

 File /app/app/jobs/email/imap/concerns/lock_manager.rb:64 in lock
File /app/app/jobs/email/imap/concerns/lock_manager.rb:29 in block (2 levels) in <module:LockManager>
Hide 96 non-project frames
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:467 in instance_exec
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:467 in block in make_lambda
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:199 in block (2 levels) in halting
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:687 in block (2 levels) in default_terminator
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:686 in catch
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:686 in block in default_terminator
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:200 in block in halting
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:595 in block in invoke_before
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:595 in each
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:595 in invoke_before
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:116 in block in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/instrumentation/active_job.rb:67 in perform
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/instrumentation/active_job.rb:23 in block (4 levels) in <main>
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in instance_exec
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in block in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/i18n-1.13.0/lib/i18n.rb:322 in with_locale
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/translation.rb:9 in block (2 levels) in <module:Translation>
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in instance_exec
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in block in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/core_ext/time/zones.rb:66 in use_zone
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/timezones.rb:9 in block (2 levels) in <module:Timezones>
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in instance_exec
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in block in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:138 in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/execution.rb:58 in _perform_job
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/instrumentation.rb:20 in _perform_job
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/execution.rb:46 in perform_now
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/instrumentation.rb:14 in block in perform_now
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/instrumentation.rb:25 in block in instrument
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/notifications.rb:206 in block in instrument
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/notifications/instrumenter.rb:24 in instrument
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/notifications.rb:206 in instrument
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/instrumentation.rb:35 in instrument
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/instrumentation.rb:14 in perform_now
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/logging.rb:18 in block in perform_now
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/tagged_logging.rb:99 in block in tagged
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/tagged_logging.rb:37 in tagged
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/tagged_logging.rb:99 in tagged
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/logging.rb:25 in tag_logger
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/logging.rb:18 in perform_now
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/execution.rb:24 in block in execute
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:118 in block in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/railtie.rb:54 in block (4 levels) in <class:Railtie>
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/execution_wrapper.rb:92 in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/reloader.rb:72 in block in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/execution_wrapper.rb:88 in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/reloader.rb:71 in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/railtie.rb:53 in block (3 levels) in <class:Railtie>
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in instance_exec
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in block in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:138 in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/execution.rb:22 in execute
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/queue_adapters/sidekiq_adapter.rb:42 in perform
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:209 in execute_job
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:177 in block (2 levels) in process
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:180 in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:183 in block in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/rollbar-3.4.0/lib/rollbar/plugins/sidekiq/plugin.rb:11 in call
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:182 in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:183 in block in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/instrumentation/sidekiq/server.rb:25 in block in call
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:384 in perform_action_with_newrelic_trace
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/instrumentation/sidekiq/server.rb:20 in call
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:182 in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:183 in block in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/metrics/tracking.rb:26 in track
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/metrics/tracking.rb:122 in call
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:182 in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:173 in invoke
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:176 in block in process
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:140 in block (6 levels) in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/job_retry.rb:114 in local
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:139 in block (5 levels) in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/rails.rb:16 in block in call
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/execution_wrapper.rb:92 in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/reloader.rb:72 in block in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/execution_wrapper.rb:92 in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/reloader.rb:71 in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/rails.rb:15 in call
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:135 in block (4 levels) in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:270 in stats
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:130 in block (3 levels) in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/job_logger.rb:13 in call
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:129 in block (2 levels) in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/job_retry.rb:81 in global
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:128 in block in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/job_logger.rb:39 in prepare
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:127 in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:175 in process
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:82 in process_one
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:72 in run
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/component.rb:10 in watchdog
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/component.rb:19 in block in safe_thread
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/tracer.rb:434 in block (2 levels) in thread_block_with_current_transaction
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/tracer.rb:356 in capture_segment_error
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/tracer.rb:433 in block in thread_block_with_current_transaction
RedisClient::ConnectionError: Broken pipe (Most recent call first)
Hide 28 non-project frames
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/ruby_connection.rb:76 in rescue in write
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/ruby_connection.rb:73 in write
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/connection_mixin.rb:30 in call
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:211 in block (2 levels) in call
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/middlewares.rb:16 in call
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:210 in block in call
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:626 in ensure_connected
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:209 in call
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:172 in block (2 levels) in lock
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/pooled.rb:30 in block in with
File /app/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.0/lib/connection_pool.rb:107 in block (2 levels) in with
File /app/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.0/lib/connection_pool.rb:106 in handle_interrupt
File /app/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.0/lib/connection_pool.rb:106 in block in with
File /app/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.0/lib/connection_pool.rb:103 in handle_interrupt
File /app/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.0/lib/connection_pool.rb:103 in with
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/pooled.rb:26 in with
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:171 in block in lock
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:216 in recover_from_script_flush
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:170 in lock
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:266 in block (2 levels) in lock_instances
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:266 in select
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:266 in block in lock_instances
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:319 in timed
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:265 in lock_instances
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:240 in block in try_lock_instances
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:236 in times
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:236 in try_lock_instances
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:69 in lock
File /app/app/jobs/email/imap/concerns/lock_manager.rb:64 in lock
File /app/app/jobs/email/imap/concerns/lock_manager.rb:29 in block (2 levels) in <module:LockManager>

@byroot
Copy link
Member

byroot commented May 4, 2023

Ok, I suspect the connection is lost for some reason, but that's probably not new.

See https://github.com/redis-rb/redis-client#reconnection

redis-rb does retry by default, and that's probably what was hiding these events to you. You can pass reconnect_attempts: 1 to restore the old behavior.

@byroot byroot closed this as completed May 4, 2023
@wnm
Copy link
Author

wnm commented May 4, 2023

ah fantastic, thanks so much for the quick reply!!! ❤️

@byroot byroot mentioned this issue May 22, 2023
Closed
@leequarella leequarella mentioned this issue May 22, 2023
Open
@jorge-d jorge-d mentioned this issue Jan 5, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@byroot @wnm