Net::HTTP::Persistent::Error: too many connection resets

[treeder]

This was reported a while back, but it was closed so making a new one: #19

We get this error a LOT and from searching around the web, it seems other people get it too and they all seem to trace back to net-http-persistent (http://goo.gl/c1Qes). We recently had a user who was getting this pretty consistently, and easily, from a heroku app so we had him try changing the backing gem to rest-client and typhoeus and the problem disappeared using either of those. All our gems are using the 'rest' gem (https://github.com/iron-io/rest) so we just had the user change the backing gem like so:

    Rest::Client.new(:gem=>:typhoeus)

Here are some examples of what we're seeing on a regular basis, all of these from today:

Net::HTTP::Persistent::Error: too many connection resets (due to end of file reached - EOFError) after 0 requests on 73380140, last used 1363039742.9996092 seconds ago

Net::HTTP::Persistent::Error: too many connection resets (due to end of file reached - EOFError) after 0 requests on 82025000, last used 1363039458.9997427 seconds ago

Net::HTTP::Persistent::Error: too many connection resets (due to end of file reached - EOFError) after 1 requests on 28986080, last used 59.305080555 seconds ago -- ["/usr/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:959:in 

too many connection resets (due to end of file reached - EOFError) after 16 requests on 34995740, last used 0.059532367 seconds ago

And so on. All on the same line it appears:

/usr/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:959:in `rescue in request'
/usr/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:968:in `request'

It would be great if we could find a fix because I really like your library, it performs almost as good as typhoeus with no binary dependency which is great.

[featalion]

Hey guys, we found the error. It is depend on version of Ruby. So, please close the issue.

Sorry to disturb you.

[mikerudolph]

@featalion which version of Ruby did you update to for fixing this issue? I am seeing a similar failure due to

too many connection resets (due to Timeout::Error - Timeout::Error) after 0 requests on 23848520, last used 1365149977.7872036 seconds ago

[featalion]

@mikerudolph , the oldest version I tested is 1.9.3-p327. But possible earlier, like p192, works too.

[rehevkor5]

@featalion I just tested my code which exhibits this problem on ruby 1.9.3p392 (2013-02-22 revision 39386) [x86_64-linux] on an Amazon EC2 instance, and I still get the problem.

Do you know what specific change in Ruby you thought resolved this issue?

[featalion]

As I remember, changes in Ruby SSL (non-block reading or so, C code). But I'm not pretty sure it works for any use cases. I tested linux/darwin x64 MRI Rubies and in my cases issues disappear at p327. Also tested p392 and it was OK.
For us the issue was that Ruby does not work with SSL socket correctly while reading/writing.
It is not improbable that still appears for some of your use cases.

[dekz]

Evident for me on linux-x86_64 1.9.3-p327, typing this error (unable to copy). On net-http-persistent 2.7, and 2.8.

lib/net/http/persistent.rb:959:in 'rescue in request': too many connection resets (due to SSL_read:: wrong version number - OpenSSL::SSL::SSLError) after 4 requests on 10261120, last used 0.008279738 seconds ago (Net::HTTP::Persistent::Error)

It worked for some time in a multithreaded test case, then the error appeared after a minute. Second run happened after quite quicker.

[featalion]

I'm not sure it is thread-safe. For example, program opens one socket and N threads reading/writing at the same time.

[drbrain]

net-http-persistent opens a new HTTP socket per thread.

Is there a good way for me to reproduce this? I think I will need both the ruby version and the OpenSSL version along with a test script to reproduce it.

[featalion]

In my case net-http-persistent did not read/write socket (1.9.3-p0). @dekz is it possible remote end respond with bad data? But I'm still interesting in this because it possibly will happen again on our service. Will help test if you'll provide all stuff @drbrain asked for.

[mislav]

I was bitten by the same issue as @mikerudolph above:

too many connection resets (due to Timeout::Error - Timeout::Error) after 0 requests

This is an issue separate from EOFError and SSLError mentioned above, and is easily reproducible. It will occur if the connection was configured with a read timeout and the server didn't reply in a timely fashion. Net::HTTP is going to raise a Timeout::Error and net-http-persistent handles this, among other errors, by resetting the connection, retrying, and ultimately aborting with the misleading message quoted above.

I fixed this for myself by monkeypatching net-http-persistent from master to not handle Timeout::Error. In the case of a server taking too long to generate a response, I don't think resetting the connection does us any good since it wasn't a connection problem to begin with. It would be much better to leave the original Timeout::Error unhandled and let the user retry the request if they will.

[ifduyue]

I also encountered this today:

#<Net::HTTP::Persistent::Error: too many connection resets (due to Net::ReadTimeout - Net::ReadTimeout) after 1784 requests on 2561033320, last used 244.59479 seconds ago>

ruby 2.0.0p451 (2014-02-24 revision 45167) [x86_64-darwin12.5.0]
net-http-persistent (2.9.4)

[mislav]

@ifduyue Yours isn't the same issue, as it seems you're getting valid "too many connection resets" errors due to read timeouts. It's probably that you're not giving enough time for the remote server to respond. Try increasing the timeout setting.

[ifduyue]

@mislav thx, I'll try.

[hut8]

I'm getting this issue with 2.0.0.-p353, and have been trying to debug it for quite a while. I increased my timeout to 600 seconds. That should be way more than enough for the request I'm taking (when doing this in the browser it takes < 10 seconds -- long, but not ten minutes) I don't think I have done 10582 requests; I believe I've only done about 8688 at this point in the program (unless the difference is made by resetting the connection before this via idle timeouts).

I'm using the same connection object because it's wrapped in a Mechanize object, but each request should be almost instantaneous (the only processing I do to the result is checking the status) -- so I'm not hitting an idle timeout, right?

What does connection resets even mean? Is that when a Net::HTTP::Persistent connection makes a new underlying connection? Why is EOFError appearing here? It seems like this should be throwing a timeout error, since 600 seconds is my read_timeout

home/lbowen/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/openssl/buffering.rb:174:in `sysread_nonblock': too many connection resets (due to end of file reached - EOFError) after 10582 requests on 21232720, last used 600.798127325 seconds ago (Net::HTTP::Persistent::Error)
        from /home/lbowen/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/openssl/buffering.rb:174:in `read_nonblock'
        from /home/lbowen/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/net/protocol.rb:153:in `rbuf_fill'
        from /home/lbowen/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil'
        from /home/lbowen/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/net/protocol.rb:144:in `readline'
        from /home/lbowen/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line'
        from /home/lbowen/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new'
        from /home/lbowen/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/net/http.rb:1406:in `block in transport_request'
        from /home/lbowen/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/net/http.rb:1403:in `catch'
        from /home/lbowen/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/net/http.rb:1403:in `transport_request'
        from /home/lbowen/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/net/http.rb:1376:in `request'
        from /home/lbowen/.rvm/gems/ruby-2.0.0-p353/gems/net-http-persistent-2.9/lib/net/http/persistent.rb:986:in `request'
        from /home/lbowen/.rvm/gems/ruby-2.0.0-p353/gems/mechanize-2.7.3/lib/mechanize/http/agent.rb:259:in `fetch'
        from /home/lbowen/.rvm/gems/ruby-2.0.0-p353/gems/mechanize-2.7.3/lib/mechanize.rb:440:in `get'
        from ./log-replay.rb:157:in `block in replay_log' # <-- that's the end of my code