[Sidekiq 6.5.1] SSL_read: unexpected eof while reading

[jeremiemv]

Ruby version: 3.1.2
Rails version: 7.0.3
Sidekiq: 6.5.1

redis gem: 4.6.0

Running on Heroku
Happening in

• staging, common runtime, heroku-22
• production, shield private space, heroku-20

Heroku Redis: 6.2.3, premium-0

We have dedicated redis instances for sidekiq.

After upgrading to 6.5.1 from 6.4.2, we are having intermittent redis connection issues, in both staging and production.

OpenSSL::SSL::SSLError: SSL_read: unexpected eof while reading

/app/vendor/ruby-3.1.2/lib/ruby/3.1.0/openssl/buffering.rb line 214 in sysread_nonblock
/app/vendor/ruby-3.1.2/lib/ruby/3.1.0/openssl/buffering.rb line 214 in read_nonblock
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb line 55 in block in _read_from_socket
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb line 54 in loop
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb line 54 in _read_from_socket
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb line 47 in gets
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb line 382 in read
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 311 in block in read
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 299 in io
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 310 in read
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 161 in block in call
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 279 in block (2 levels) in process
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 420 in ensure_connected
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 269 in block in process
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 356 in logging
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 268 in process
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 161 in call
/app/vendor/bundle/ruby/3.1.0/gems/scout_apm-5.2.0/lib/scout_apm/instruments/redis.rb line 32 in block in call_with_scout_instruments
/app/vendor/bundle/ruby/3.1.0/gems/scout_apm-5.2.0/lib/scout_apm/tracer.rb line 34 in instrument
/app/vendor/bundle/ruby/3.1.0/gems/scout_apm-5.2.0/lib/scout_apm/tracer.rb line 44 in instrument
/app/vendor/bundle/ruby/3.1.0/gems/scout_apm-5.2.0/lib/scout_apm/instruments/redis.rb line 31 in call_with_scout_instruments
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis.rb line 263 in block in send_command
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis.rb line 262 in synchronize
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis.rb line 262 in send_command
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/commands/scripting.rb line 110 in _eval
/app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/commands/scripting.rb line 97 in evalsha
/app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 55 in zpopbyscore
/app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 35 in block (2 levels) in enqueue_jobs
/app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 30 in each
/app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 30 in block in enqueue_jobs
/app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq.rb line 156 in block in redis
/app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb line 63 in block (2 levels) in with
/app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb line 62 in handle_interrupt
/app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb line 62 in block in with
/app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb line 59 in handle_interrupt
/app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb line 59 in with
/app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq.rb line 153 in redis
/app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 29 in enqueue_jobs
/app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 109 in enqueue
/app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 101 in block in start
/app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/component.rb line 8 in watchdog
/app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/component.rb line 17 in block in safe_thread

initializer

require "sidekiq/web"

Sidekiq.configure_server do |config|
  config.redis = {
    url: ENV["SIDEKIQ_REDIS_URL"] || ENV["REDIS_URL"] || "redis://localhost:6379/1",
    ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }
  }
end

Sidekiq.configure_client do |config|
  config.redis = {
    url: ENV["SIDEKIQ_REDIS_URL"] || ENV["REDIS_URL"] || "redis://localhost:6379/1",
    ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }
  }
end

sidekiq.yml

# Sample configuration file for Sidekiq.
# Options here can still be overridden by cmd line args.
# Place this file at config/sidekiq.yml and Sidekiq will
# pick it up automatically.
---
:verbose: false
:concurrency: 5
:timeout: 25
:max_retries: 14

# Sidekiq will run this file through ERB when reading it so you can
# even put in dynamic logic, like a host-specific queue.
# http://www.mikeperham.com/2013/11/13/advanced-sidekiq-host-specific-queues/
:queues:
  - critical
  - default
  - mailers
  - low

# you can override concurrency based on environment
production:
  :concurrency: <%= ENV.fetch("SIDEKIQ_CONCURRENCY") { 5 } %>

development:
  :max_retries: 0

[mperham]

SSL issues are lower level than Sidekiq. Check the redis gem or driver for issues.

…

On Fri, Jun 24, 2022 at 19:34 Jérémie Meyer de Ville < ***@***.***> wrote: Ruby version: 3.1.2 Rails version: 7.0.3 Sidekiq: 6.5.1 redis gem: 4.6.0 Running on Heroku Happening in - staging, common runtime, heroku-22 - production, shield private space, heroku-20 Heroku Redis: 6.2.3 After upgrading to 6.5.1 from 6.4.0, we are having intermittent redis connection issues, in both staging and production *OpenSSL::SSL::SSLError: SSL_read: unexpected eof while reading* /app/vendor/ruby-3.1.2/lib/ruby/3.1.0/openssl/buffering.rb line 214 in sysread_nonblock /app/vendor/ruby-3.1.2/lib/ruby/3.1.0/openssl/buffering.rb line 214 in read_nonblock /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb line 55 in block in _read_from_socket /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb line 54 in loop /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb line 54 in _read_from_socket /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb line 47 in gets /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb line 382 in read /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 311 in block in read /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 299 in io /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 310 in read /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 161 in block in call /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 279 in block (2 levels) in process /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 420 in ensure_connected /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 269 in block in process /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 356 in logging /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 268 in process /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb line 161 in call /app/vendor/bundle/ruby/3.1.0/gems/scout_apm-5.2.0/lib/scout_apm/instruments/redis.rb line 32 in block in call_with_scout_instruments /app/vendor/bundle/ruby/3.1.0/gems/scout_apm-5.2.0/lib/scout_apm/tracer.rb line 34 in instrument /app/vendor/bundle/ruby/3.1.0/gems/scout_apm-5.2.0/lib/scout_apm/tracer.rb line 44 in instrument /app/vendor/bundle/ruby/3.1.0/gems/scout_apm-5.2.0/lib/scout_apm/instruments/redis.rb line 31 in call_with_scout_instruments /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis.rb line 263 in block in send_command /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis.rb line 262 in synchronize /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis.rb line 262 in send_command /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/commands/scripting.rb line 110 in _eval /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/commands/scripting.rb line 97 in evalsha /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 55 in zpopbyscore /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 35 in block (2 levels) in enqueue_jobs /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 30 in each /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 30 in block in enqueue_jobs /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq.rb line 156 in block in redis /app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb line 63 in block (2 levels) in with /app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb line 62 in handle_interrupt /app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb line 62 in block in with /app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb line 59 in handle_interrupt /app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb line 59 in with /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq.rb line 153 in redis /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 29 in enqueue_jobs /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 109 in enqueue /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb line 101 in block in start /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/component.rb line 8 in watchdog /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.5.1/lib/sidekiq/component.rb line 17 in block in safe_thread *initializer* require "sidekiq/web" Sidekiq.configure_server do |config| config.redis = { url: ENV["SIDEKIQ_REDIS_URL"] || ENV["REDIS_URL"] || "redis://localhost:6379/1", ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } } end Sidekiq.configure_client do |config| config.redis = { url: ENV["SIDEKIQ_REDIS_URL"] || ENV["REDIS_URL"] || "redis://localhost:6379/1", ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } } end *sidekiq.yml* # Sample configuration file for Sidekiq. # Options here can still be overridden by cmd line args. # Place this file at config/sidekiq.yml and Sidekiq will # pick it up automatically. --- :verbose: false :concurrency: 5 :timeout: 25 :max_retries: 14 # Sidekiq will run this file through ERB when reading it so you can # even put in dynamic logic, like a host-specific queue. # http://www.mikeperham.com/2013/11/13/advanced-sidekiq-host-specific-queues/ :queues: - critical - default - mailers - low # you can override concurrency based on environment production: :concurrency: <%= ENV.fetch("SIDEKIQ_CONCURRENCY") { 5 } %> development: :max_retries: 0 — Reply to this email directly, view it on GitHub <#5402>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAWX2SR7MEGRWGE62YDYTVQXWRJANCNFSM5ZYQLD7Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jeremiemv]

@mperham we haven't touched the redis gem in forever since the February 6.4.0 update.
Reverting sidekiq fixes the issue.

[jeremiemv]

And no recent openssl updates.
https://devcenter.heroku.com/changelog-items/2434

[mperham]

I don’t see any issues either nor am I aware of any issues. https://github.com/redis/redis-rb/blob/master/CHANGELOG.md#470

6.5.0 added support for the new redis-client gem but I don’t see how that could have broken low-level SSL socket operations in the existing redis gem. @casperisfine ?

[byroot]

Yeah I don't see it either.

@jeremiemv a couple questions to hopefully have a few theories to explore:

• Are all the backtraces exactly like this? Or does it fail at random points (e.g. here it fails in Redis#evalsha is it always the case).
• unexpected eof while reading suggest the connection was closed by the server. Anything weird in the Redis server logs?

[byroot]

We changed:

conn.evalsha(@lua_zpopbyscore_sha, keys: keys, argv: argv)

for

conn.evalsha(@lua_zpopbyscore_sha, keys, argv)

But that should make any difference: https://github.com/redis/redis-rb/blob/70dab9c149e739b450fa0c6ad660646f58e23170/lib/redis/commands/scripting.rb#L75-L96

[byroot]

Also since it happens on staging, I assume you can test more various things.

• If you migrate to redis-client does the issue still happen?
• What about hiredis-client ?
• Any way to connect without SSL for a while? It's possible that the SSL layer is hiding the underlying issue.

NB: I'm just giving ideas to hopefully get a bit more info on what the problem is.

[jeremiemv]

Are all the backtraces exactly like this? Or does it fail at random points (e.g. here it fails in Redis#evalsha is it always the case).

Yes. And same in staging and prod.

unexpected eof while reading suggest the connection was closed by the server. Anything weird in the Redis server logs?

Only logs I see are the regular metrics samples. No warning or error.

I found another application log. Not sure if this is helpful additional info.

Jun 24 09:47:02 samosa-app-production app[web] ERROR E, [2022-06-24T16:47:02.839752 #25] ERROR -- : [admin] [c9c8b2c0-66dc-9b92-9170-1a926f7abb34] [ActiveJob] Failed enqueuing SyncUserJob to Sidekiq(default): OpenSSL::SSL::SSLError (SSL_read: unexpected eof while reading)

Also since it happens on staging, I assume you can test more various things.

If you migrate to redis-client does the issue still happen?

Will give it a try.

What about hiredis-client ?

Will give it a try.

disabling SSL.

Not sure it is possible with Redis 6 on Heroku.
https://devcenter.heroku.com/articles/heroku-redis#security-and-compliance
Will investigate.

Thank you for looking into this BTW!

[byroot]

Yes. And same in staging and prod.

Oh wow, color me intrigued. Something in the EVALSHA specifically must cause the server to close the connection somehow.

[jeremiemv]

No SSL and hiredis-client are both a no.

SSL:
As of Redis version 6, Hobby plans support both TLS and unencrypted connections. Production plans require TLS connections. You must enable TLS in your Redis client’s configuration in order to connect to a Redis 6 database.
We are using production plans.

hiredis-client:
Because of the above, it is a no go.
https://github.com/mperham/sidekiq/wiki/Heroku#redis
The redis gem has supported the verify_mode key since version 4.0.2. The hiredis gem does not work in this environment, so do not include it in your Gemfile if you are using redis with rediss:// and OpenSSL::SSL::VERIFY_NONE.

Going to see if I can make redis-client work.

[jeremiemv]

Ah. It just happened in staging with sidekiq 6.4.2

OpenSSL::SSL::SSLError: SSL_read: unexpected eof while reading
  /app/vendor/ruby-3.1.2/lib/ruby/3.1.0/openssl/buffering.rb, line 214, in sysread_nonblock
  /app/vendor/ruby-3.1.2/lib/ruby/3.1.0/openssl/buffering.rb, line 214, in read_nonblock
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb, line 55, in block in _read_from_socket
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb, line 54, in loop
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb, line 54, in _read_from_socket
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb, line 47, in gets
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb, line 382, in read
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb, line 311, in block in read
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb, line 299, in io
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb, line 310, in read
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb, line 161, in block in call
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb, line 279, in block (2 levels) in process
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb, line 420, in ensure_connected
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb, line 269, in block in process
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb, line 356, in logging
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb, line 268, in process
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/client.rb, line 161, in call
  /app/vendor/bundle/ruby/3.1.0/gems/scout_apm-5.1.1/lib/scout_apm/instruments/redis.rb, line 32, in block in call_with_scout_instruments
  /app/vendor/bundle/ruby/3.1.0/gems/scout_apm-5.1.1/lib/scout_apm/tracer.rb, line 34, in instrument
  /app/vendor/bundle/ruby/3.1.0/gems/scout_apm-5.1.1/lib/scout_apm/tracer.rb, line 44, in instrument
  /app/vendor/bundle/ruby/3.1.0/gems/scout_apm-5.1.1/lib/scout_apm/instruments/redis.rb, line 31, in call_with_scout_instruments
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis.rb, line 263, in block in send_command
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis.rb, line 262, in synchronize
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis.rb, line 262, in send_command
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/commands/scripting.rb, line 110, in _eval
  /app/vendor/bundle/ruby/3.1.0/gems/redis-4.6.0/lib/redis/commands/scripting.rb, line 97, in evalsha
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.4.2/lib/sidekiq/scheduled.rb, line 55, in zpopbyscore
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.4.2/lib/sidekiq/scheduled.rb, line 35, in block (2 levels) in enqueue_jobs
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.4.2/lib/sidekiq/scheduled.rb, line 30, in each
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.4.2/lib/sidekiq/scheduled.rb, line 30, in block in enqueue_jobs
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.4.2/lib/sidekiq.rb, line 95, in block in redis
  /app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb, line 63, in block (2 levels) in with
  /app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb, line 62, in handle_interrupt
  /app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb, line 62, in block in with
  /app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb, line 59, in handle_interrupt
  /app/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.2.5/lib/connection_pool.rb, line 59, in with
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.4.2/lib/sidekiq.rb, line 92, in redis
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.4.2/lib/sidekiq/scheduled.rb, line 29, in enqueue_jobs
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.4.2/lib/sidekiq/scheduled.rb, line 108, in enqueue
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.4.2/lib/sidekiq/scheduled.rb, line 100, in block in start
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.4.2/lib/sidekiq/util.rb, line 56, in watchdog
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-6.4.2/lib/sidekiq/util.rb, line 65, in block in safe_thread

My apologies, the timing really coincided with the sidekiq upgrade in staging and the following prod release.

No idea what the root cause is yet :/

[mperham]

Is there something bad about the Lua that causes Redis to reset the connection? Perhaps using a command or option that is not supported in the version you're running?

[byroot]

hiredis-client:
Because of the above, it is a no go.

Note that hiredis-client is the hiredis binding for redis-client and it does support SSL. Hence why I suggested it. It's interesting to try it because it's a radically different implementation so it's likely to report errors in a different way (or maybe not because in the end it's OpenSSL too).

Ah. It just happened in staging with sidekiq 6.4.2

Ah that makes more sense now :)

I'm still willing to help debug this any way I can. Once you manage to get more information feel free to open an issue on redis/redis-rb or redis-rb/redis-client.