How can I implement this with websites that perform cookie consent?

[NeverBeLazyG]

like this page ?

https://www.golem.de/news/yunxing-chinas-ueberschalljet-prototyp-soll-mach-4-erreicht-haben-2410-190360.html

[unclecode]

Hi @NeverBeLazyG In such cases, you can use the JS code field and pass the JavaScript code. Detect the ID or class name of the button, then run the code to mimic a click action. After that, the crawling process will occur. When you pass the JS code, the crawler library will execute it, wait briefly, and then proceed with the crawl. Please let me know if you encounter any issues. Check the doc https://crawl4ai.com/mkdocs/basic/page-interaction/

[NeverBeLazyG]

thank you

[NeverBeLazyG]

as i understand, there is no way to automate this universally. its always some manuall action needed to find the click button

[NeverBeLazyG]

When I think about it now, I use a Chrome extension called: idontcareaboutcookies. It can automatically handle 99% of all cookie consent requests. If you could integrate something like that into your crawler, it would be a game changer. It would then be like a universal crawler that works almost everywhere. Maybe you can implement that someday in the future.

[unclecode]

@NeverBeLazyG It can be somewhat automated using heuristic techniques and a vision model, and it's been in our plan. I've seen the Chrome extension animation before, but it apparently doesn't work recently due to issues, including allegations of data being sold without user consent. You're right, it would be very helpful. Thanks for the suggestion; I'll add it to the backlog.

[NeverBeLazyG]

Are we talking about the same extension? For me, this extension still works flawlessly since years at 99% of websites. for me It is the best extension available, and I always install it first. It saves me from so many annoying cookie notifications and still works perfectly, even on websites in different languages. In my opinion, it is one of the best extensions that have ever been released.

[unclecode]

@NeverBeLazyG Interesting, I think I can find the source code for this extension and check their approach. By the way, I just updated something interesting, where in the new version, you can use your own Chrome - the one already installed on your computers - which has access to your preference settings. If you've already installed this extension, you can browse pages without an issue, then use that browser to crawl; it'll work. I'll release that soon, and I guess it'll work perfectly. Meanwhile, I'll also check the code for this one. Thanks again for reminding me.

[NeverBeLazyG]

@NeverBeLazyG Interesting, I think I can find the source code for this extension and check their approach. By the way, I just updated something interesting, where in the new version, you can use your own Chrome - the one already installed on your computers - which has access to your preference settings. If you've already installed this extension, you can browse pages without an issue, then use that browser to crawl; it'll work. I'll release that soon, and I guess it'll work perfectly. Meanwhile, I'll also check the code for this one. Thanks again for reminding me.

thats sound amazing