response.body is ASCII-8BIT when Content-Type is text/xml; charset=utf-8

[auxbuss]

First time using faraday, so I might be doing things incorrectly, but the response.body encoding in the following is ASCII-8BIT:

  def self.search(term)
    connection = Faraday.new(url: 'https://en.wikipedia.org')
    response = connection.get do |req|
      req.options = { :timeout => 5, :open_timeout => 3 }
      req.url '/w/api.php' , action: 'opensearch', format: 'xml', search: term
    end
    puts response.body.encoding
  end

In 1.9.2 this causes REXML to throw an Encoding::CompatibilityError.

I couldn't find a way to force faraday to provide response.body in UTF-8.

What is the preferred solution to this?

Just encountered the same issue. Any ideas?

[auxbuss]

Workaround I used is:

response.body.force_encoding('utf-8')

Yahuda has a dissertation about the problem here.

[technoweenie]

I'm pretty sure Faraday just passes on the response body from the underlying adapter. I'm not sure I want to raise errors or perform lossy conversions of the data in Faraday. That can be done in a custom middleware if you really need it.

[auxbuss]

Fair enough. If the problem is elsewhere, as it appears, I guess it will be cleaned up in time. It's not a show stopper for me.

[mislav]

Closing because it's not a bug with Faraday.

[chrismo]

I'm not sure the underlying adapter - at least net/http - does any encoding transformation. You can set Ruby's Encoding.default_external to something like 'US-ASCII', then hit an endpoint with Content-Type = '...; charset=utf-8' ... net/http will parse the charset string and make it available, but does nothing to the encoding of the body string. Maybe net/http should be responsible for that, but if it isn't, the ParseJson middleware (for example) can blow up.

[chrismo]

Did some more research on this - some of the underlying adapters handle the Content-Type charset, some don't:

EM-HTTP-Request does. [commit].
Patron does. [commit].
HTTPClient does [commit] [issue].
Typhoeus and Excon (and net/http) don't appear to.

I guess the nicest thing to do would be to perhaps offer an optional middleware for adapters that don't try, but, yeah, I'd agree, this probably shouldn't be Faraday's responsibility.

[cyzgbw]

try https://github.com/qhwa/string_utf8

[etipton]

@chrismo you're my hero. Thanks for doing that research!