Possible deep-rooted issue of Conversion of String Encodings in http response body.

[luke-hill]

Preface

This is a bit of a long convoluted chain (So apologies in advance). I'm not even sure if this is 100% fixable. But I'll post it here and try include as much info.

When using Selenium::Webdriver.logger in standard selenium practice, you can pipe debug messages. The following issue occurs (Page under test is: https://www.citizensadvice.org.uk/immigration/ - A national charity website).

Simple reproduction case

1. Grab text of footer using selenium. With the footer on this website, we have a © character.
2. If selenium logging is set, pipe some debug info to the selenium logger (This is a standard ruby logger)
3. Whilst debugging the info here. We get a Ruby Logger Kernel.warn message

log writing failed. "\xC2" from ASCII-8BIT to UTF-8

Root cause

The root cause issue (I believe), is with this process flow. The capitalised letters refer to which library is doing the code processing at that time.

1. SELENIUM - A request payload is built up using a standard API Payload structure
2. NET::HTTP - The request is sent using the Selenium HTTP adapter (Which is just an instance of the net http one).
3. NET::HTTP - The request object is returned with all the standard properties - This is where the non-conformancy is generated.
4. SELENIUM - Selenium then processes this through a series of check to ensure safety
5. LOGGER - If configured it will do a debug-trace log (We have this set to on) - This is where the error is bubbled up.

Next steps?

I am not 100% sure. I think it would be nice to know why this encoding conversion is happening. I can patch it fine my end. But it has been mentioned to try trace exactly where / why it's occurring.

[luke-hill]

Extra infos:

This only occurs when you get a (c) character. Some people at work have told me a quick-fix to this, which works well. But it would be good to know why when finding a (c) character using this library the encoding changes from UTF-8 (The standard one in our filesystem), to ASCII.

A colleague of mine mentioned that maybe it's not ASCII but an "Unknown Binary" encoding, and that ASCII is being defaulted erroneously?

Would it be possibly to have the http response body set to be the correct encoding at all times?

[jeremyevans]

ASCII-8BIT encoding means binary. It is currently expected that net-http returns bodies with this encoding, though there is a long standing feature request to add encoding support: https://bugs.ruby-lang.org/issues/2567

[jeremyevans]

This is possibly addressed by #17, can you check if that will work for you? You'll have to modify the code using Net::HTTP to set the new force_response_body_encoding support.

[luke-hill]

Sure thing. I'll have a go at this in the coming days.

[luke-hill]

Managed to get a free 10 mins.

A quick copy and paste of the 2 raw files in your SHA did not fix the issue.

We currently fix the issue by running Selenium::WebDriver.logger.io.binmode as a single line method invocation.

The io method does the following

      def io
        @logger.instance_variable_get(:@logdev).dev
      end

The logger in question is a Selenium::WebDriver.logger instance.

[jeremyevans]

That's expected. As I wrote previously: "You'll have to modify the code using Net::HTTP to set the new force_response_body_encoding support."

[luke-hill]

Just curious with the fixes that went in in 2022 by yourself and merged by nurse whether this is in a future Ruby version.

I have patched this in a gem I maintain here: https://github.com/site-prism/automation_helpers specifically with this patch class here: https://github.com/site-prism/automation_helpers/blob/main/lib/automation_helpers/patches/selenium_logger.rb

Any thoughts/updates on what I could/should be doing?

[jeremyevans]

This pull request I discussed above is in Ruby 3.2. It was renamed from force_response_body_encoding to just response_body_encoding: 6233e6b

Unfortunately, I'm at a conference and do not currently have time to look at the links you sent.

[luke-hill]

Cheers. That's great to hear. In the next week or so I'll re-test this on ruby 3.2 and update my findings. Obviously I'm assuming it isn't a backport candidate as it's just a cosmetic feature.

The links I sent were to a patch based on date / gem version. So I'll need to factor in this and add a date much further in the future.

[luke-hill]

As an update. Real life got in the way. I'm still nowhere near ready to verify this is fixed or not. But I will get to it. At some point :D

[luke-hill]

Hi @jeremyevans - Would it be possible to include the string representation of the site I scanned in the minitest suite. I have just re-tested on ruby 3.2.2 and the bug is still there.

Console warns (In red)

log writing failed. "\xE2" from ASCII-8BIT to UTF-8
log writing failed. "\xE2" from ASCII-8BIT to UTF-8
log writing failed. "\xE2" from ASCII-8BIT to UTF-8
log writing failed. "\xE2" from ASCII-8BIT to UTF-8
log writing failed. "\xE2" from ASCII-8BIT to UTF-8
log writing failed. "\xC2" from ASCII-8BIT to UTF-8

A small sample script (Using capybara) is as follows (Some bits ommitted)

$ visit('https://www.citizensadvice.org.uk/immigration/') # nil
$ copyright = page.find('[data-testid="copyright"]') # #<Capybara::Node::Element:0x00007f2cedab2360>
$ copyright.text # Copyright ©2024 Citizens Advice. All rights reserved.

[jeremyevans]

@luke-hill I'm not sure. However, the question doesn't seem related to net-http. If you are not already, you may want to try HTTP#response_body_encoding= to set the expected response body encoding. Alternatively, make sure your logger is logging to a binary file and not a text file.

[luke-hill]

Currently I have a 1-line patch which is. @logger.instance_variable_get(:@logdev).dev#binmode

The logger does currently log to a file, this is deliberate.

What I'm looking for is something to be fixed here so that I don't need to run the patch. If I need to retain the patch because the issue isn't fixed that's fine. Just thought I'd mention this hasn't fixed the underlying bug

[jeremyevans]

Apparently, your patch is to a logger. If you provide your logger an already open File instance in binary mode, shouldn't that resolve the problem without the need to patch?

This doesn't sound like a bug at all, it sounds like an improper Logger configuration issue. Can you explain why you think this is a net-http issue?

[luke-hill]

Because the error message is generated there and I'll be honest it's been over 3.5 years since I first found this. So I'm not 100% sure.

My original work was done here: site-prism/automation_helpers@e4c8597 from February 2021 and I'd been suffering with it for a while before that.

The Logger is just configured out of the box, no customisations e.t.c. - imo if the ruby Logger out of the box cannot handle the (c) character then that needs fixing. Maybe it needs to go to ruby/logger. Again I'm not 100%. I was just chasing up something from a while back here.