405 errors for all requests with Ruby versions > 2.3 #8
Description
I tried setting this up with Ruby 2.5.1 and all of my requests were being rejected with 405 errors. For example:
# using Ruby 2.5.1
$ https_proxy=http://localhost:8000 curl -vI https://google.com --insecure
* Rebuilt URL to: https://google.com/
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8000 (#0)
* Establish HTTP proxy tunnel to google.com:443
> CONNECT google.com:443 HTTP/1.1
> Host: google.com:443
> User-Agent: curl/7.54.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: WEBrick/1.4.2 (Ruby/2.5.1/2018-03-29) OpenSSL/1.1.1b
Server: WEBrick/1.4.2 (Ruby/2.5.1/2018-03-29) OpenSSL/1.1.1b
< Date: Fri, 10 May 2019 10:47:37 GMT
Date: Fri, 10 May 2019 10:47:37 GMT
< Content-Length: 0
Content-Length: 0
* Ignoring Content-Length in CONNECT 200 response
< Connection: close
Connection: close
<
* Proxy replied OK to CONNECT request
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; O=mitm.proxy; OU=ca; OU=CA; CN=google.com
* start date: May 9 20:46:12 2019 GMT
* expire date: May 8 20:46:12 2020 GMT
* issuer: C=US; O=mitm.proxy; OU=ca; CN=CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> HEAD / HTTP/1.1
> Host: google.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 405 Method Not Allowed
HTTP/1.1 405 Method Not Allowed
< Content-Type: text/html; charset=ISO-8859-1
Content-Type: text/html; charset=ISO-8859-1
< Server: WEBrick/1.4.2 (Ruby/2.5.1/2018-03-29) OpenSSL/1.1.1b
Server: WEBrick/1.4.2 (Ruby/2.5.1/2018-03-29) OpenSSL/1.1.1b
< Date: Fri, 10 May 2019 11:22:36 GMT
Date: Fri, 10 May 2019 11:22:36 GMT
< Content-Length: 315
Content-Length: 315
< Connection: close
Connection: close
<
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
When I downgraded to 2.4 the issue was still present, but downgrading to Ruby 2.3 seems to have fixed it:
$ ruby -v
ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin14]
$ https_proxy=http://localhost:8000 curl -vI https://google.com --insecure
* Rebuilt URL to: https://google.com/
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8000 (#0)
* Establish HTTP proxy tunnel to google.com:443
> CONNECT google.com:443 HTTP/1.1
> Host: google.com:443
> User-Agent: curl/7.54.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: WEBrick/1.3.1 (Ruby/2.3.0/2015-12-25) OpenSSL/1.0.2f
Server: WEBrick/1.3.1 (Ruby/2.3.0/2015-12-25) OpenSSL/1.0.2f
< Date: Fri, 10 May 2019 11:29:20 GMT
Date: Fri, 10 May 2019 11:29:20 GMT
< Content-Length: 0
Content-Length: 0
* Ignoring Content-Length in CONNECT 200 response
< Connection: close
Connection: close
<
* Proxy replied OK to CONNECT request
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / DHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; O=mitm.proxy; OU=ca; OU=CA; CN=google.com
* start date: May 9 20:46:12 2019 GMT
* expire date: May 8 20:46:12 2020 GMT
* issuer: C=US; O=mitm.proxy; OU=ca; CN=CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> HEAD / HTTP/1.1
> Host: google.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
* HTTP/1.1 proxy connection set close!
< Proxy-Connection: close
Proxy-Connection: close
< Connection: close
Connection: close
< Location: https://www.google.com/
Location: https://www.google.com/
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Date: Fri, 10 May 2019 11:29:18 GMT
Date: Fri, 10 May 2019 11:29:18 GMT
< Expires: Sun, 09 Jun 2019 11:29:18 GMT
Expires: Sun, 09 Jun 2019 11:29:18 GMT
< Cache-Control: public, max-age=2592000
Cache-Control: public, max-age=2592000
< Server: gws
Server: gws
< Content-Length: 220
Content-Length: 220
< X-Xss-Protection: 0
X-Xss-Protection: 0
< X-Frame-Options: SAMEORIGIN
X-Frame-Options: SAMEORIGIN
< Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
<
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
My server proxy code is as per the example:
require 'evil-proxy'
proxy = EvilProxy::MITMProxyServer.new Port: 8000
trap "INT" do proxy.shutdown end
trap "TERM" do proxy.shutdown end
proxy.start
I haven't had much time to dig into this but I managed to narrow it down to the following error:
Exception: NoMethodError: undefined method `<<' for {:read_timeout=>60, :continue_timeout=>nil, :debug_output=>nil}:Hash
--
0: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/net/protocol.rb:223:in `writing'
1: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/net/protocol.rb:206:in `write'
2: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/net/http/generic_request.rb:334:in `write_header'
3: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/net/http/generic_request.rb:127:in `exec'
4: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:1492:in `block in transport_request'
5: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:1491:in `catch'
6: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:1491:in `transport_request'
7: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:1464:in `request'
8: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:1234:in `head'
9: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/webrick/httpproxy.rb:222:in `block in do_HEAD'
10: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/evil-proxy-0.2.0/lib/evil-proxy/agentproxy.rb:35:in `block in perform_proxy_request'
11: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start'
12: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/evil-proxy-0.2.0/lib/evil-proxy/agentproxy.rb:27:in `perform_proxy_request'
13: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/webrick/httpproxy.rb:221:in `do_HEAD'
14: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/evil-proxy-0.2.0/lib/evil-proxy/httpproxy.rb:111:in `block (2 levels) in <class:HTTPProxyServer>'
15: (pry):1:in `rescue in proxy_service'
16: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/pry_instance.rb:387:in `eval'
17: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/pry_instance.rb:387:in `evaluate_ruby'
18: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/pry_instance.rb:355:in `handle_line'
19: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/pry_instance.rb:274:in `block (2 levels) in eval'
20: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/pry_instance.rb:273:in `catch'
21: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/pry_instance.rb:273:in `block in eval'
22: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/pry_instance.rb:272:in `catch'
23: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/pry_instance.rb:272:in `eval'
24: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/repl.rb:77:in `block in repl'
25: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/repl.rb:67:in `loop'
26: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/repl.rb:67:in `repl'
27: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/repl.rb:38:in `block in start'
28: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/input_lock.rb:59:in `__with_ownership'
29: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/input_lock.rb:77:in `with_ownership'
30: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/repl.rb:38:in `start'
31: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/repl.rb:13:in `start'
32: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-0.12.2/lib/pry/pry_class.rb:200:in `start'
33: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-byebug-3.7.0/lib/byebug/processors/pry_processor.rb:117:in `block in resume_pry'
34: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-byebug-3.7.0/lib/byebug/processors/pry_processor.rb:30:in `block (2 levels) in run'
35: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/byebug-11.0.1/lib/byebug/helpers/eval.rb:94:in `allowing_other_threads'
36: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-byebug-3.7.0/lib/byebug/processors/pry_processor.rb:30:in `block in run'
37: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-byebug-3.7.0/lib/byebug/processors/pry_processor.rb:29:in `catch'
38: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-byebug-3.7.0/lib/byebug/processors/pry_processor.rb:29:in `run'
39: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-byebug-3.7.0/lib/byebug/processors/pry_processor.rb:113:in `resume_pry'
40: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/pry-byebug-3.7.0/lib/byebug/processors/pry_processor.rb:65:in `at_line'
41: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/byebug-11.0.1/lib/byebug/context.rb:98:in `at_line'
42: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/webrick/httpproxy.rb:121:in `rescue in proxy_service'
43: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/webrick/httpproxy.rb:117:in `proxy_service'
44: /Users/xriley/.rvm/gems/ruby-2.5.1/gems/evil-proxy-0.2.0/lib/evil-proxy/agentproxy.rb:52:in `service'
45: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/webrick/httpserver.rb:96:in `run'
46: /Users/xriley/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/webrick/server.rb:307:in `block in start_thread'
Caused by: NoMethodError: undefined method `<<' for {:read_timeout=>60, :continue_timeout=>nil, :debug_output=>nil}:Hash
Did you mean? <
To me, it looks like a change to Net::HTTP or `` after Ruby 2.3.0 where the config hash is getting passed to something that expects a String for logging. If anyone has any tips on how to narrow this down I'd appreciate it!