[BUG] google.com blocks non-JavaScript requests

[typicalninja]

What package is this bug report for?
all packages

Describe the bug
Currently any project using google-sr will break as google have disabled the ability to search with javascript off, which we depend on to get the results. i am currently testing to see if there is any way to fix this. we are most likely looking at a full rewrite here

Screenshots

Links

Here are some links we can use for ref:
https://news.ycombinator.com/item?id=42719865
https://news.ycombinator.com/item?id=42743344
https://techcrunch.com/2025/01/17/google-begins-requiring-javascript-for-google-search/

[enzon19]

Do you have any idea how long it will take to fix this? Thanks!

[typicalninja]

within this week most likely. i am hoping it should be just a simple update. but for some results the layout google serves is different from the js to non-js page, which might take longer to implement.

[typicalninja]

okay it looking bit harder than i expected. google is actually gating everything behind javascript it seems. still looking to see what i can do

[CatsMiaow]

I'm not sure, but when I extract the curl of the second redirected page and call it, it responds with HTML data.
Presumably it generates some value on the first call and sends it as a cookie on the second call.

[typicalninja]

I've been getting positive results using an older user agent string, such as: Links (2.29; Linux 6.11.0-13-generic x86_64; GNU C 13.2; text), (ref: https://news.ycombinator.com/item?id=42743344)

google serves an old version of the search page with no js requirement for old user agents, still testing as this version differs from the one we used to have