selenium

Short introduction to selenium

You can:

• execute .js commands, ...
• interact with websites,
• ...

Selenium is availible in lots of different programming languages - not only python.

Installation

Install selenium

pip3 install selenium

Set up drivers

Then you normally need download a driver for your desired browser: URL: http://www.seleniumhq.org/download/ refer to Third Party Drivers, Bindings, and Plugins

URL: https://www.selenium.dev/downloads/ go to Platforms Supported by Selenium

Direct links:

• Chromium: https://sites.google.com/chromium.org/driver/
• Firefox/Gecko: https://github.com/mozilla/geckodriver/releases

Then copy the files to the right place & adapt permissions.

Example for the geckodriver (Firefox):

Ubuntu

sudo mv geckodriver /usr/bin/
chmod a+x /usr/bin/geckodriver

Windows

Just add your executable of the Gecko driver to your Path variables. For example:

# local (per `cmd.exe` session)
set PATH=%PATH%;C:\Users\usr\AppData\Local\Programs\geckoDriver

# global
setx PATH C:\Users\usr\AppData\Local\Programs\geckoDriver /m

# Or just add path via GUI

If path is not set correctly you probably get an error message like Exception AttributeError: "'Service' object has no attribute 'process'" in <bound method Service.__del__ of <selenium.webdriver.firefox.service.Service object at 0x1006890d0>> ignored

Now we are ready to go! :D

Usage

Basic example (python)

import time

from selenium import webdriver
from pyvirtualdisplay import Display               # Optional: use virtual display for use in headless env (servers without X)

display = Display(visible=0, size=(800, 600))      # Not necessarily needed
display.start()                                    # Not necessarily needed

browser= webdriver.Chrome()                        # For Firefox: browser = webdriver.Firefox()
browser.get("https://github.com/TeamFlowerPower")   # Access page
wait = 5
print(f"Waiting for {wait} seconds that page is completely loaded...")
time.sleep(wait)
print(browser.current_url)                          # Print URL

print(browser.page_source.encode('utf-8'))

contentXPattern = r'//*[@id="MainPart_lbUsersInLineAheadOfYou"]'

result = None
try:
    someVarTmp = browser.find_element_by_xpath(contentXPattern)      # Raises an NoSuchElementException when pattern not found
    someVar = someVarTmp.text                                        # Convert to text
    print(f"Users ahead of me: {someVar}")
except Exception as ex:
    exName = type(ex).__name__                                       # Get exception name for printing
    print(f"XPath not found ({exName})")
    #browser.quit()
    result = None
else:
    result = {someVar: browser}

browser.quit()
display.stop()                                     # Not necessarily needed

Note: quit() may raise a warning/exception in Firefox; there is also close() where this message does not show up using Firefox but it does not close the window either.

Do cool .js stuff

browser.execute_script("document.getElementById('email').setAttribute('value', 'recipient@gmail.com')");

Basic example (JavaScript)

TODO

Promises

Most in modern Selenium with JavaScript is based on Promises: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_promises

Here also arrow-functions become handy.

// Traditional anonymous function
function (name){
  return `hello ${name}`;
}
# becomes:
(name) => {
    return `hello ${name}`;
}

namedArrowFunction = (name) => `hello ${name}`;     // Only omit the `return` for one-line functions (no curly braces)

_{About template literals -> `${}`}

Chaining

Allows sequentiality of promises:

A common need is to execute two or more asynchronous operations back to back, where each subsequent operation starts when the previous operation succeeds, with the result from the previous step. We accomplish this by creating a promise chain.

Here's the magic: the then() function returns a new promise, different from the original

_{(https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_promises#chaining)}

Some notes about Selectors

XPath, CSS, ...

Starts-with / containment / ends-with / ...

# XPath
//div[starts-with(@class,'class-name-fragment-start')]
//*[contains(@class,'class-name-fragment')]
//div[ends-with(@class,'class-name-fragment-start')]
# `*`: any element


# CSS
div[class^='class-name-fragment-startswith']
div[class*='class-name-fragment-contains']
div[class$='class-name-fragment-endswith']

Find element by attribute

The Attribute is role (with the name button) in this example:

# XPath
//div[@role = 'button']
# CSS
div[role = 'button']

Additional Resources

• Selenium doc as pdf