Which HTTP library is most recommended?

[jhubert]

We're experiencing the same timeout and failed connection issue with Net::HTTP as outlined in #298. I'm looking at using a different lib such as Patron or Typhoeus but there seems to be a lot of chatter about problems with both of them if I look back through the issues.

I understand from the README and various Issues that there is hesitation to recommend a single library. I also appreciate the flexibility of being able to use whichever one we want. But, since we don't use any other HTTP library right now, it would be really helpful to understand which one is the most consistent and most supported specifically for ES.

Is there a general preference?

[Hamada92]

Hey, I am in the same boat as you, and just wondering if a library is necessary. The documentation says:

Keep in mind, that for optimal performance, you should use a HTTP library which supports persistent ("keep-alive") connections, e.g. Patron or Typhoeus.

But keep-alive was optional in HTTP 1.0, and now it's the default in any HTTP 1.1 request without having to use a library, so I still don't understand the reason behind having to use an HTTP library.

[karmi]

Hi @jhubert, sorry for the delay!

Unless I'm mistaken, Net::HTTP still doesn't support persistent connections, and that's definitely something which is important when working with Elasticsearch (to prevent the overhead of opening and closing connections, related file descriptors, etc).

In my own applications, I use either Typhoeus or Patron. The issues reported in the Rails integrations repository were/are mostly coming from threaded environments (eg. Puma), but I use Puma and Patron in a Rails application myself, without any extra connection pooling or wrapping, so I was never able to pinpoint the issue to a specific code problem.

All that said, I'd argue that Net::HTTP is good only for the most basic HTTP handling in Ruby (comparable to something like WEBrick vs Thin/Puma/etc). I don't have the link to the benchmarks, but the performance was always incomparable between Net::HTTP and anything Libcurl based (such as Curb, Patron or Typhoeus).

I suggest to just try one of these two — Typhoeus and Patron —, both are nicely supported through the Faraday abstract wrapper, so there are no big code changes, and running some example requests and benchmarks.

[karmi]

Hi, any news here? :)

[majormoses]

I would hardly call it a real benchmark but switching from Faraday to Typhoeus was a significant slow down (went from about a minute to about 2 minutes) when doing reindexing if the indices are small enough to be handled by Faraday. No other code change was made.

[karmi]

Hello, closing this issue, we're nearing a one year anniversary :) Please re-open or open a new one if you'd like to continue the discussion!