403 error on scrapper but website is accessible on browser

[nomanilyas16]

Hi, I'm trying to access this website but getting 403 error but the same website is accessible with the browser.

Thanks in advance

[flavorjones]

It's likely that in your browser you have cookies set by this site (corresponding to the "I agree" button on the public access disclaimer page). Mechanize supports cookies, but when you create a new instance, its cookie jar will be empty (similar to what an incognito browser session has).

Your Mechanize agent should probably click through that initial public access disclaimer page, thereby creating the necessary cookie.

[nomanilyas16]

Even the disclaimer page is also throwing 403 error with Mechanize. I have tried it.

[nomanilyas16]

You can check in screenshot that I am trying to get the disclaimer page and it's throwing 403 error

[flavorjones]

Well, then that site may be blocking unknown user agents. You may want to try having Mechanize use one of the ones built in, see https://www.rubydoc.info/gems/mechanize/Mechanize#AGENT_ALIASES-constant