Selenium base UC mode: find_element work but find_elements doesn't

[alicesin615]

Hi, trying out the UC mode on linux machine (Chrome ver 117.0.5938.92) to scrape some public links.
Tested using sb.find_element which works as expected. But find_elements return me empty array. Tried time.sleep(10) and checked if the elements that I'm looking out for have been loaded via screenshot.
If I were to run in collab using standard google chrome driver, I'm able to scrape the expected links for current page.

with SB(uc=True, agent=agent, uc_subprocess=False, xvfb=True) as sb:
    sb.open(url)
    sb.wait_for_ready_state_complete()
    links = sb.find_elements(
        locator, limit=3
    )
    time.sleep(10)

May I know what else could be a potential cause in such scenario?

[mdmintz]

SeleniumBase uses a good user-agent by default. It's possible that you gave it a bad one that will detect your bot. It's best to leave it as the default value. Also, uc_subprocess doesn't change anything anymore, so you shouldn't need to use it.

For opening URLs in stealth mode, use uc_open_with_reconnect(url, reconnect_time), where reconnect_time is a time (in seconds) that is longer than the time it takes to finish loading the URL you want to navigate to. eg: uc_open_with_reconnect(url, 4). Then, you don't need to use sb.wait_for_ready_state_complete().

Assuming the selector is correct, sb.find_elements(selector) returns a list of WebElements. Then extract the link from each one.

[alicesin615]

understand your point about leaving the user agent as the default value set within sb is already a good one, and im wondering if I could do a rotation of user agents as one of the measures to lower the chance of getting undetected?

[mdmintz]

You can change the user agent. There’s an agent arg for that. I’m just saying that it might not help, and it might make things worse.

[alicesin615]

Thank you! Works as expected now