`puppet module install` breaks on HTTP redirects.

[defnull]

Describe the Bug

Calling puppet module install or any other code that internally uses Puppet::HTTP::Client may break with obscure invalid encoding:"783: unexpected token at '\x1F\x8B\b'" or similar errors if HTTP redirects are involved and the final server supports HTTP compression.

This is caused by Puppet::HTTP::Redirector internally used by Puppet::HTTP::Client to follow HTTP redirects. The redirect logic copies all request headers from the original Net::HTTPRequest object to a new one, including an Accept-Encoding header that was implicitly added by Net::HTTP during the first request.

Net::HTTP supports HTTP compression. It will automatically add an Accept-Encoding header and later decompress the response body, but only if that header was not already set by the caller. Puppet::HTTP::Redirector adds this header explicitly, disabling the transparent compression handling in Net::HTTP and resulting in a response body that may contain gzip compressed binary data instead of the expected result.

Feeding gzip compressed binary data into a JSON decoder triggers the unexpected token at '\x1F\x8B\b' error from above.

Expected Behavior

HTTP redirects should be handled gracefully and not cause obscure errors.

Steps to Reproduce

Steps to reproduce the behavior:

1. Run puppet module install --module_repository http://forge.puppetlabs.com puppetlabs-stdlib
2. The redirect from http to https will trigger the error.

Environment

• Version: 7 and 8
• Platform: any

[github-actions]

Migrated issue to PUP-11983

[joshcooper]

Thanks @defnull, we can reproduce as well. One option is to exclude the Accept-Encoding header when creating the new request:

puppet/lib/puppet/http/redirector.rb

Line 52 in 03b3df3

request.each do |header, value|

Or possibly passing down a set of headers to be set on the new request:

puppet/lib/puppet/http/client.rb

Line 378 in 03b3df3

request = redirector.redirect_to(request, response, redirects)

See also a23ee91

[defnull]

I think excluding Accept-Encoding when copying the old headers to the new request is the way to go. That's probably the easiest solution and does not require changes in any of the internal APIs.