AutoGUI interaction

[szymsza]

After trying out the devtools protocols in BugHog, the two simple implementations I had prepared worked only for Chromium 70+ and Firefox 127+. Even though there were some other protocols implemented in previous browser versions, it would require a lot of effort to implement all of these and correctly match them to the supported browser versions. Further, there would likely be no way to target versions approx. 20-50.

As a result, I decided to use PyAutoGUI as you suggested. It works in all browsers and versions and for practical usage it seems almost as good as the devtools protocols.

Instead of url_queue.txt, users can choose to provide the experiment configuration in interaction_script.cmd. These are the commands implemented so far:

• NAVIGATE url Terminates the previous browser window and opens a new browser window on the specified URL. Further, it waits some time (1 sec for Chromium, 2 secs for Firefox) for the page to load.
• CLICK_POSITION x y Clicks on specific coordinates on the screen (not necessarily the browser window), The argument values can be absolute in pixels, percentage of the screen, or a combination of both - e.g., CLICK_POSITION 100 50% clicks 100px from the left screen border and 50% of the screen height.
• CLICK element_id Clicks on an element with the specified ID. Currently, the ID can be one of one, two, three, four, five, six. This is because PyAutoGUI can search for the location of a visual match on the screen. Therefore, I prepared styles in res/bughog.css that style elements with these IDs to boxes of distinct colors. This allows us to bypass the limitation of having to know the exact screen coordinates of an element.
• WRITE text Types the text into the focused element.
• PRESS key
• HOLD key
• RELEASE key
• HOTKEY key1 key2 ... A combination of HOLD and RELEASE for multiple keys. E.g., HOTKEY ctrl c.
• SLEEP seconds Usually should not be necessary to use because navigation implicitly includes sleeping.
• SCREENSHOT file_name Captures the screen and stores the result in logs/screenshots/{PROJECT}-{EXPERIMENT}-{file_name}-{BROWSER}-{VERSION}.jpg. Very useful for debugging

A simple experiment can be found in Support/AutoGUI. It got successfully reproduced in all versions of both browsers.

We can possibly implement some browser-specific behaviour as well, e.g., bookmarking a string where the script would include only BOOKMARK text and based on the browser version, the correct shortcut would be pressed and screen positions clicked.

Besides this, I made the following changes:

• Extracted the default file templates to separate files and added templates for all file types
• Implemented adding and modifying interaction_script.cmd and url_queue.txt from the web UI
• Implemented a custom highlighting mode for interaction_script.cmd in the experiment editor
• Fixed loading resources from /res/

[GJFR]

I took a first jab at merging. The button for creating a url queue and interaction script isn't displayed currently. This is partly due to the hacky way in which experiment folders are distinguished from files like url_queue.txt (for which I take full responsibility :)).

I'm gonna have a more in-depth look at how to properly solve this later.

BugHog/bci/web/vue/src/components/poc-editor.vue

Lines 295 to 297 in e815589

	<button v-if="files && Object.keys(files).length === 0" class="no-style" onclick="create_config_dialog.showModal()">
	<v-icon name="fa-plus" class="text-indigo-500"/>
	</button>