Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use server helper in in_http #2395

Merged
merged 8 commits into from
May 13, 2019

Conversation

ganmacs
Copy link
Member

@ganmacs ganmacs commented Apr 23, 2019

Which issue(s) this PR fixes:

I couldn't find any issue about HTTPS support of in_http.
#2124

What this PR does / why we need it:

I replaced an implementation of in_http with server helper to prepare for HTTPS support.

Docs Changes:

no

Release Note:
no

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
I think adding HTTP Keep-Alive to server helper(L4 helper) is a bit
confusing. So I don't implement HTTP Keep-Alive feature on server
helper now.

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
@ganmacs ganmacs requested a review from repeatedly April 23, 2019 08:58
@ganmacs ganmacs added the v1 label Apr 23, 2019
@cosmo0920
Copy link
Contributor

#2124 is related issue.

@repeatedly
Copy link
Member

I tested this patch and I got lock releated error.

2019-05-10 11:39:30.000000000 +0900 test.http: {"k":"hello","k1":1234}
2019-05-10 11:39:30 +0900 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::HttpInput title=:event_loop error_class=ThreadError error="deadlock; recursive locking"
2019-05-10 11:39:30 +0900 [error]: #0 unexpected error error_class=ThreadError error="deadlock; recursive locking"
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin_helper/server.rb:789:in `synchronize'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin_helper/server.rb:789:in `close'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin_helper/server.rb:416:in `close'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin/in_http.rb:483:in `on_write_complete'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin/in_http.rb:236:in `block in on_server_connect'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin_helper/server.rb:429:in `block in on'
  2019-05-10 11:39:30 +0900 [error]: #0 (eval):9:in `on_write_complete'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:147:in `on_writable'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin_helper/server.rb:754:in `block in on_writable'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin_helper/server.rb:750:in `synchronize'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin_helper/server.rb:750:in `on_writable'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:190:in `on_writable'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2019-05-10 11:39:30 +0900 [error]: #0 /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-05-10 11:39:30 +0900 [error]: #0 unexpected error error_class=ThreadError error="deadlock; recursive locking"
  2019-05-10 11:39:30 +0900 [error]: #0 suppressed same stacktrace
  • conf
<source>
  @type http
  bind 0.0.0.0
  body_size_limit 32m
  <transport tls>
    insecure true
  </transport>
</source>

<match test.**>
  @type stdout
</match>
  • test client
require 'net/http'
require 'net/https'
require 'msgpack'

record = {'msgpack' => {'k' => 'hello', 'k1' => 1234}.to_msgpack}

def post(path, params)
  http = Net::HTTP.new("127.0.0.1", 9880)
  http.use_ssl = true
  http.verify_mode = OpenSSL::SSL::VERIFY_NONE
  req = Net::HTTP::Post.new(path, {})
  req.set_form_data(params)
  http.request(req)
end

res = post("/test.http?time=#{Time.now.to_i}", record)
puts res.body

Could you check?

@ganmacs
Copy link
Member Author

ganmacs commented May 13, 2019

Could you check?

Thank you for reviewing it.
In my environment, it would work with this patch(#2413).

@repeatedly
Copy link
Member

I confirmed it works with 2413.

@repeatedly repeatedly merged commit f3a6066 into fluent:master May 13, 2019
@repeatedly
Copy link
Member

Thanks!

@ganmacs ganmacs deleted the use-server-helper-in-in_http branch May 13, 2019 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants