Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sporadic RedisClient::ConnectionError: Broken pipe #69

Open
davidgm0 opened this issue Dec 6, 2023 · 6 comments
Open

Sporadic RedisClient::ConnectionError: Broken pipe #69

davidgm0 opened this issue Dec 6, 2023 · 6 comments

Comments

@davidgm0
Copy link

davidgm0 commented Dec 6, 2023

I have not been able to reproduce this, but I'm seeing this error daily when a job gets enqueued from a model callback:

RedisClient::ConnectionError: Broken pipe
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client/ruby_connection.rb:76:in `rescue in write'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client/ruby_connection.rb:73:in `write'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client/connection_mixin.rb:30:in `call'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:256:in `block (2 levels) in call'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client/middlewares.rb:16:in `call'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:255:in `block in call'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:675:in `ensure_connected'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:254:in `call'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:13:in `block (2 levels) in delete_lock'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:200:in `block (2 levels) in synchronize'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:219:in `with'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:200:in `block in synchronize'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:200:in `synchronize'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:200:in `synchronize'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:35:in `synced_redis_connection'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:12:in `block in delete_lock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:11:in `each'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:11:in `delete_lock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/strategies/base.rb:30:in `unlock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/strategies/until_executed.rb:12:in `after_perform'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/active_job_patch.rb:52:in `block (2 levels) in <module:ActiveJobPatch>'

Another similar error I see is

Redlock::LockAcquisitionError: failed to acquire lock on 'Too many Redis errors prevented lock acquisition:
RedisClient::ConnectionError: Broken pipe'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:323:in `lock_instances'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:275:in `block in try_lock_instances'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:271:in `times'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:271:in `try_lock_instances'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:79:in `lock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/strategies/base.rb:24:in `lock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/strategies/base.rb:59:in `before_enqueue'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/active_job_patch.rb:50:in `block (2 levels) in <module:ActiveJobPatch>'
/usr/local/bundle/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:467:in `instance_exec'

This does not happen if I downgrade to 0.2.5 and redlock 1.x. I'm using rails 7.0.8

@micahtessler
Copy link

I have the same error if I use activejob-uniqueness v0.3.1 and redlock 2.0.6
The error goes away if I downgrade redlock with
gem "redlock", "=2.0.4"

It's intermittant, not every lock check throws the error.

@micahtessler
Copy link

I have the same error if I use activejob-uniqueness v0.3.1 and redlock 2.0.6 The error goes away if I downgrade redlock with gem "redlock", "=2.0.4"

It's intermittant, not every lock check throws the error.

Now it's happening with redlock 2.04 as well. Downgrading redlock did not fix the issue for me...

@micahtessler
Copy link

I have the same error if I use activejob-uniqueness v0.3.1 and redlock 2.0.6 The error goes away if I downgrade redlock with gem "redlock", "=2.0.4"
It's intermittant, not every lock check throws the error.

Now it's happening with redlock 2.04 as well. Downgrading redlock did not fix the issue for me...

To stop the error, I downgraded to

gem "activejob-uniqueness", "=0.2.5"
gem "redlock", "=1.3.2"

@jorge-d
Copy link

jorge-d commented Jan 4, 2024

Hey there, I did encounter the same issue using rails 7.0.8.

After investigation I stumbled upon this fix that was shipped recently in Redlock 2.0.6.

Updating to Rails 7.1 seems to fix the issue. I suggest that you add an activejob >= 7.1 dependency for the next gem versions.

Best,

EDIT: Ok it turns out it reduced the number of errors but we still have intermittent Broken Pipes, I believe it is related to the discussion in this issue: redis-rb/redis-client#119

EDIT 2: It's been a few hours now that I think I've fixed it by setting reconnect_attempts: 1 in my redlock server initialization, as suggested in this answer
Here's the interesting part in my config/initializers/active_job_uniqueness.rb:

  config.redlock_servers = [{
    url: REDIS_URL,
    db: REDIS_WORKER_DB,
    reconnect_attempts: 1,
    ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }
  }]

@jeffbax
Copy link

jeffbax commented Jan 25, 2024

Hey there, I did encounter the same issue using rails 7.0.8.

After investigation I stumbled upon this fix that was shipped recently in Redlock 2.0.6.

Updating to Rails 7.1 seems to fix the issue. I suggest that you add an activejob >= 7.1 dependency for the next gem versions.

Best,

EDIT: Ok it turns out it reduced the number of errors but we still have intermittent Broken Pipes, I believe it is related to the discussion in this issue: redis-rb/redis-client#119

EDIT 2: It's been a few hours now that I think I've fixed it by setting reconnect_attempts: 1 in my redlock server initialization, as suggested in this answer Here's the interesting part in my config/initializers/active_job_uniqueness.rb:

  config.redlock_servers = [{
    url: REDIS_URL,
    db: REDIS_WORKER_DB,
    reconnect_attempts: 1,
    ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }
  }]

I just noticed a similar error on Heroku running Rails 6.1, redis-rb 4 (due to ActionCable) on Redis 7 itself in a review app testing the upgrades, will try this but it has not been super smooth sailing to update so far 🥲

@jeffbax
Copy link

jeffbax commented Mar 14, 2024

FWIW I have not run into this since doing the full upgrade in our app (although do have reconnect_attempts: 1 specified)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants