Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The client connection should be aborted on write errors #22

Open
SpComb opened this issue Aug 16, 2017 · 1 comment
Open

The client connection should be aborted on write errors #22

SpComb opened this issue Aug 16, 2017 · 1 comment
Labels
bug

Comments

@SpComb
Copy link
Contributor

SpComb commented Aug 16, 2017

The Kontena::Websocket::Client::Connection#write method will do multiple partial write calls to write out the complete websocket frame. If any of those partial write calls fail, then we might end up with a partial websocket frame written out on the socket, which would corrupt any further writes. For safety, even a non-partial write failure for a websocket frame should be considered a corrupted connection.

Currently any exceptions raised from #write do not explicitly mark the connection as dead. If they happen to be handshake/ping/pong frames written by the #read thread, then they will propagate out and break the read loop. However, if the writes come from other threads doing separate #send calls, then they will not abort the #read thread, or prevent any further #send calls, which can result in corrupted connection.
@SpComb SpComb added the bug label Aug 16, 2017
@SpComb
Copy link
Contributor Author

SpComb commented Oct 10, 2017

Here's a case of something like this happening. Looks like for the case of wss:// connections, the OpenSSL::SSL::SSLSocket itself will fail on write retries:

W, [2017-10-10T09:58:30.916696 #1]  WARN -- Kontena::WebsocketClient: Broken pipe (Errno::EPIPE)
/usr/lib/ruby/2.4.0/openssl/buffering.rb:386:in `syswrite_nonblock'
/usr/lib/ruby/2.4.0/openssl/buffering.rb:386:in `write_nonblock'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client/connection.rb:113:in `block in write'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client/connection.rb:85:in `nonblocking_timeout'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client/connection.rb:112:in `write'
/usr/lib/ruby/gems/2.4.0/gems/websocket-driver-0.6.5/lib/websocket/driver/hybi.rb:228:in `send_frame'
/usr/lib/ruby/gems/2.4.0/gems/websocket-driver-0.6.5/lib/websocket/driver/hybi.rb:191:in `frame'
/usr/lib/ruby/gems/2.4.0/gems/websocket-driver-0.6.5/lib/websocket/driver/hybi.rb:133:in `binary'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:305:in `block in send'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:420:in `block in with_driver'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:417:in `synchronize'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:417:in `with_driver'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:304:in `send'
/app/lib/kontena/websocket_client.rb:246:in `send_notification'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/async.rb:7:in `dispatch'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'

W, [2017-10-10T09:59:14.559699 #1]  WARN -- Kontena::WebsocketClient: SSL_write: bad write retry (OpenSSL::SSL::SSLError)
/usr/lib/ruby/2.4.0/openssl/buffering.rb:386:in `syswrite_nonblock'
/usr/lib/ruby/2.4.0/openssl/buffering.rb:386:in `write_nonblock'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client/connection.rb:113:in `block in write'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client/connection.rb:85:in `nonblocking_timeout'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client/connection.rb:112:in `write'
/usr/lib/ruby/gems/2.4.0/gems/websocket-driver-0.6.5/lib/websocket/driver/hybi.rb:228:in `send_frame'
/usr/lib/ruby/gems/2.4.0/gems/websocket-driver-0.6.5/lib/websocket/driver/hybi.rb:191:in `frame'
/usr/lib/ruby/gems/2.4.0/gems/websocket-driver-0.6.5/lib/websocket/driver/hybi.rb:133:in `binary'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:305:in `block in send'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:420:in `block in with_driver'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:417:in `synchronize'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:417:in `with_driver'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:304:in `send'
/app/lib/kontena/websocket_client.rb:260:in `send_request'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'

W, [2017-10-10T09:59:22.175902 #1]  WARN -- Kontena::RpcClient: SSL_write: bad write retry (OpenSSL::SSL::SSLError)
/usr/lib/ruby/2.4.0/openssl/buffering.rb:386:in `syswrite_nonblock'
/usr/lib/ruby/2.4.0/openssl/buffering.rb:386:in `write_nonblock'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client/connection.rb:113:in `block in write'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client/connection.rb:85:in `nonblocking_timeout'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client/connection.rb:112:in `write'
/usr/lib/ruby/gems/2.4.0/gems/websocket-driver-0.6.5/lib/websocket/driver/hybi.rb:228:in `send_frame'
/usr/lib/ruby/gems/2.4.0/gems/websocket-driver-0.6.5/lib/websocket/driver/hybi.rb:191:in `frame'
/usr/lib/ruby/gems/2.4.0/gems/websocket-driver-0.6.5/lib/websocket/driver/hybi.rb:133:in `binary'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:305:in `block in send'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:420:in `block in with_driver'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:417:in `synchronize'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:417:in `with_driver'
/usr/lib/ruby/gems/2.4.0/gems/kontena-websocket-client-0.1.0/lib/kontena/websocket/client.rb:304:in `send'
/app/lib/kontena/websocket_client.rb:260:in `send_request'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
(celluloid):0:in `remote procedure call'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:45:in `value'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/proxy/sync.rb:22:in `method_missing'
/app/lib/kontena/rpc_client.rb:72:in `request'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/async.rb:7:in `dispatch'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
/usr/lib/ruby/gems/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'

E, [2017-10-10T09:59:22.789547 #1] ERROR -- Kontena::WebsocketClient: websocket error: read deadline expired while waiting 5.0s for pong

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@SpComb