403 response from page while cURL is successful

[rachelwigell]

I'm getting a 403 response from this page using this code:

def agent
    @agent ||= begin
      agent = Mechanize.new
      agent.user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36'
      agent.request_headers = {
            'cache-control' => 'max-age=0',
            'referer' => 'https://www.google.com/',
            'user-agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36'
        }
      agent
    end
  end


agent.get('https://www.kobo.com/us/en/ebook/the-city-in-glass')

I am able to successfully get the body of the page with a 200 response using equivalent cURL (I have cURL 8.4.0 installed):

curl 'https://www.kobo.com/us/en/ebook/the-city-in-glass' \
  -H 'cache-control: max-age=0' \
  -H 'referer: https://www.google.com/' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36'

Does anyone have any ideas for why this would be successful through cURL but not Mechanize? I have version 2.12.2 of Mechanize installed.

[flavorjones]

@rachelwigell When I run curl with the -v option it looks like I'm receiving a 403! Can you double-check the response code in your environment?

{ [5 bytes data]
< HTTP/2 403
< date: Fri, 04 Oct 2024 20:34:05 GMT
< content-type: text/html; charset=UTF-8
...

[rachelwigell]

@flavorjones thanks for your reply! Confirming I get a 200, but interestingly there has been a lot of variance across machines. It seems possibly some older versions of cURL are getting the 403 response. It has also been only on Macbooks that we've been able to get a 200 response, and we've started to wonder if Apple Automatic Verification is playing a role.

HTTP/2 200
date: Thu, 03 Oct 2024 20:01:26 GMT
content-type: text/html; charset=utf-8