Open
Description
Certain use cases (e.g. SalesForce, LinkedIn) would greatly benefit from reading and writing website markup and the DOM (i.e. by simplifying tokenization of screenshots).
-
Implement a Chrome Plugin that sends and receives messages from PuterBot via native messaging: https://developer.chrome.com/docs/extensions/mv3/nativeMessaging/ . On every change to the Document Object Model (DOM) of the currently opened page, the extension should send a message to PuterBot.
-
Implement an adapter in PuterBot to receive and store the messages with https://github.com/Rayquaza01/nativemessaging. To start with, this can look something like this:
browser.py:
import nativemessaging
def main():
# based on https://github.com/Rayquaza01/nativemessaging#sample
reply_num = 0
while True:
message = nativemessaging.get_message()
print(message)
nativemessaging.send_message(nativemessaging.encode_message(str(reply_num))
reply_num += 1
if __name__ == "__main__":
main()
In the extension:
// from https://github.com/Rayquaza01/nativemessaging#sample
function onReceived(response) {
console.log(response);
}
// runtime.connectNative
var port = browser.runtime.connectNative("application_name");
port.onMessage.addListener(onReceived);
port.postMessage("hello");
// runtime.sendNativeMessage
browser.runtime.sendNativeMessage("puterbot", document).then(onReceived);
// based on https://stackoverflow.com/questions/8882502/how-to-track-dom-change-in-chrome-extension
let observer = new MutationObserver(mutations => {
for(let mutation of mutations) {
browser.runtime.sendNativeMessage("puterbot", mutation).then(onReceived);
}
});
observer.observe(document, { childList: true, subtree: true });
Once this is working, we will want to add the necessary functionality to record.py.