Add Chrome browser event in database during recording#744
Conversation
abrichr
left a comment
There was a problem hiding this comment.
Thank you @KrishPatel13 ! I left a few comments, happy to chat about any of it if you like.
If you could show some console output and/or screenshots in the PR description that would be very much appreciated!
|
An update for Chrome Extension Issue: After spending about 4-5 hours debugging nativeMessaging, I was not able to fix the issue. I dropped the idea of using nativeMessaging. Reasons:
Hence, After an hour of research, I recalled that I had used WebSockets in my assignment in CSC309 to communicate between an web server and my java running process. The task there was that whenever the user click submit I had to sent that api request to my java running client and then, connect to db and save some info about the user (registering). I thought this might relevant for the task that we are working on too. Hence, I quickly made a web_socket_extension example to demo this idea for constant communication between extension and a desktop running process (in our case a .py file). I made a dumpy Commit for the Web Socket "Ping" Example: |
|
My next step would be to log the dom changes instead of "ping" message. |
|
My goal is to at least, make a prototype by tonight: record.py (dummy): that runs and save all the dom changes in a sqlite db and once this is done, I will then manipulate the actual record.py next week so that it records the dom changes in our actual db. (our main goal). I hope this sounds like a good plan to tackle this issue :-) |
|
Update: Now I am able to log DOM changes Commit: d9f9792 |
|
Now, I can save all the DOM messages sent by the extension into a db synchronously. Commit: 1b606fb Some Screenshots for the work: and This commit proves a working prototype for the chrome extension issue. Now my next steps shall be to integrate this into our actual application. |
04d4658 to
1b606fb
Compare
8a20800 to
c3d1275
Compare
c3d1275 to
925db61
Compare
…#744) * add old chrome extension files * add old changes back, related to chrome extension, in openadapt dir * try web socket solution, add asyncio and websockets libraries to poetry * now we can log the dom changes ina a desctop running python process * remove the old nativeMessaging extension code * rename dir * save the dom chages into dummy db: chrome.db * fix actions tests * ran `poetry run black --preview . --exclude 'alembic'` * remove sockets.py * fix falke8 formatting issues * ran `poetry run black --preview . --exclude 'alembic'` * add browser event code * chrome extension work wip * fix reset_db python file. * now, extension messages can be seen in logs when the recording is started. * till trying to fix the issue: record functione execution gets stuck at read_browser_events due to asyncio.runForever * browser_events can be seen in db * ran black and flake8 * add documentation in extension side js files and remove unused files * OpenAdaptAI/OpenAdapt#744 (review) Update readme.md * OpenAdaptAI/OpenAdapt#744 (comment) remove mlds logo * OpenAdaptAI/OpenAdapt#744 (comment) add todo * OpenAdaptAI/OpenAdapt#744 (comment) add NAMED_CONSTANT * OpenAdaptAI/OpenAdapt#744 (comment) remove unused lines in content.js * OpenAdaptAI/OpenAdapt#744 (comment) uncomment for `alembic upgrade` * OpenAdaptAI/OpenAdapt#744 (comment) resolve this * OpenAdaptAI/OpenAdapt#744 (comment) remove unwanted obfuscate keys * OpenAdaptAI/OpenAdapt#744 (comment) used joinedload * OpenAdaptAI/OpenAdapt#744 (comment) remove unwanted return * OpenAdaptAI/OpenAdapt#744 (comment) Ran the following: - `alembic revision --autogenerate -m "regenrate with browser_event table"` - `alembic upgrade head` * OpenAdaptAI/OpenAdapt#744 (comment) remove unwanted commetns lines * OpenAdaptAI/OpenAdapt#744 (comment) remove commented lines * https://github.com/OpenAdaptAI/OpenAdapt/pull/744/files#r1668802201 added this todo * OpenAdaptAI/OpenAdapt#744 (comment) optimize browser events record by adding an option to include to record or not like audio * OpenAdaptAI/OpenAdapt#744 (comment) move constants to config * OpenAdaptAI/OpenAdapt#744 (comment) update readme * update readme again * OpenAdaptAI/OpenAdapt#744 (comment) update config default for RECORD_BROWSER_EVENTS * OpenAdaptAI/OpenAdapt#744 (comment) removed unessary comment in events.py * Update openadapt/models.py OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * OpenAdaptAI/OpenAdapt#744 (comment) resolve d * https://github.com/OpenAdaptAI/OpenAdapt/pull/744/files/3836c3d165bb11c5615c83a0df65f6cdc155f93f#r1676068390 resolved * ran black on openadapt dir `poetry run black --preview openadapt/` * OpenAdaptAI/OpenAdapt#744 (comment) fixed faiing tests * update the form type script * add the record browser flag in classification group * OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Update openadapt/config.py: OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Update openadapt/record.py OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Update openadapt/config.py: https://github.com/OpenAdaptAI/OpenAdapt/pull/744/files/aa7b1ae7df8fb7045abb5d71d1d41a09f2d8bea2#r1691430453 https://github.com/OpenAdaptAI/OpenAdapt/pull/744/files/aa7b1ae7df8fb7045abb5d71d1d41a09f2d8bea2#r1691430453 Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Remove 3 unused functions owing to: OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) * Update openadapt/record.py: OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Rename owing to: OpenAdaptAI/OpenAdapt#744 (comment) * Rename const: OpenAdaptAI/OpenAdapt#744 (comment) * Remove unused: OpenAdaptAI/OpenAdapt#744 (comment) * Use ClassVar again: OpenAdaptAI/OpenAdapt#744 (comment) * Update openadapt/models.py: OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Update openadapt/record.py: OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Update openadapt/record.py: OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Move string to const: OpenAdaptAI/OpenAdapt#744 (comment) * Correct Grammatical errors: OpenAdaptAI/OpenAdapt#744 (comment) * fix visualize and reformat * fix alembic migrations * Intersection/MutationObserver; _repr_ignore_attrs; fix process_events * synchronize timestamps * attachWindowEventListeners * recreate alembic migration * sync; buffer * add dtw.py (wip) * remove sorting and syncing; working experiments/dtw.py * fix content.js * report screenX/screenY; fix getScreenCoordinates; compute coordinate differences * wip * remove observers; composite distance; task_by_name * replace eventBuffer with coordMappings; compute tlbr-screen in browser.py * assign_browser_events in events.py * black; docstrings * flake8 * update test * black * add test_browser.py --------- Co-authored-by: pate1595 <kmp.patel@utoronto.ca> Co-authored-by: Krish Patel <65433817+KrishPatel13@users.noreply.github.com>
…#744) * add old chrome extension files * add old changes back, related to chrome extension, in openadapt dir * try web socket solution, add asyncio and websockets libraries to poetry * now we can log the dom changes ina a desctop running python process * remove the old nativeMessaging extension code * rename dir * save the dom chages into dummy db: chrome.db * fix actions tests * ran `poetry run black --preview . --exclude 'alembic'` * remove sockets.py * fix falke8 formatting issues * ran `poetry run black --preview . --exclude 'alembic'` * add browser event code * chrome extension work wip * fix reset_db python file. * now, extension messages can be seen in logs when the recording is started. * till trying to fix the issue: record functione execution gets stuck at read_browser_events due to asyncio.runForever * browser_events can be seen in db * ran black and flake8 * add documentation in extension side js files and remove unused files * OpenAdaptAI/OpenAdapt#744 (review) Update readme.md * OpenAdaptAI/OpenAdapt#744 (comment) remove mlds logo * OpenAdaptAI/OpenAdapt#744 (comment) add todo * OpenAdaptAI/OpenAdapt#744 (comment) add NAMED_CONSTANT * OpenAdaptAI/OpenAdapt#744 (comment) remove unused lines in content.js * OpenAdaptAI/OpenAdapt#744 (comment) uncomment for `alembic upgrade` * OpenAdaptAI/OpenAdapt#744 (comment) resolve this * OpenAdaptAI/OpenAdapt#744 (comment) remove unwanted obfuscate keys * OpenAdaptAI/OpenAdapt#744 (comment) used joinedload * OpenAdaptAI/OpenAdapt#744 (comment) remove unwanted return * OpenAdaptAI/OpenAdapt#744 (comment) Ran the following: - `alembic revision --autogenerate -m "regenrate with browser_event table"` - `alembic upgrade head` * OpenAdaptAI/OpenAdapt#744 (comment) remove unwanted commetns lines * OpenAdaptAI/OpenAdapt#744 (comment) remove commented lines * https://github.com/OpenAdaptAI/OpenAdapt/pull/744/files#r1668802201 added this todo * OpenAdaptAI/OpenAdapt#744 (comment) optimize browser events record by adding an option to include to record or not like audio * OpenAdaptAI/OpenAdapt#744 (comment) move constants to config * OpenAdaptAI/OpenAdapt#744 (comment) update readme * update readme again * OpenAdaptAI/OpenAdapt#744 (comment) update config default for RECORD_BROWSER_EVENTS * OpenAdaptAI/OpenAdapt#744 (comment) removed unessary comment in events.py * Update openadapt/models.py OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * OpenAdaptAI/OpenAdapt#744 (comment) resolve d * https://github.com/OpenAdaptAI/OpenAdapt/pull/744/files/3836c3d165bb11c5615c83a0df65f6cdc155f93f#r1676068390 resolved * ran black on openadapt dir `poetry run black --preview openadapt/` * OpenAdaptAI/OpenAdapt#744 (comment) fixed faiing tests * update the form type script * add the record browser flag in classification group * OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Update openadapt/config.py: OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Update openadapt/record.py OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Update openadapt/config.py: https://github.com/OpenAdaptAI/OpenAdapt/pull/744/files/aa7b1ae7df8fb7045abb5d71d1d41a09f2d8bea2#r1691430453 https://github.com/OpenAdaptAI/OpenAdapt/pull/744/files/aa7b1ae7df8fb7045abb5d71d1d41a09f2d8bea2#r1691430453 Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Remove 3 unused functions owing to: OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) * Update openadapt/record.py: OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Rename owing to: OpenAdaptAI/OpenAdapt#744 (comment) * Rename const: OpenAdaptAI/OpenAdapt#744 (comment) * Remove unused: OpenAdaptAI/OpenAdapt#744 (comment) * Use ClassVar again: OpenAdaptAI/OpenAdapt#744 (comment) * Update openadapt/models.py: OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Update openadapt/record.py: OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Update openadapt/record.py: OpenAdaptAI/OpenAdapt#744 (comment) OpenAdaptAI/OpenAdapt#744 (comment) Co-authored-by: Richard Abrich <richard.abrich@mldsai.com> * Move string to const: OpenAdaptAI/OpenAdapt#744 (comment) * Correct Grammatical errors: OpenAdaptAI/OpenAdapt#744 (comment) * fix visualize and reformat * fix alembic migrations * Intersection/MutationObserver; _repr_ignore_attrs; fix process_events * synchronize timestamps * attachWindowEventListeners * recreate alembic migration * sync; buffer * add dtw.py (wip) * remove sorting and syncing; working experiments/dtw.py * fix content.js * report screenX/screenY; fix getScreenCoordinates; compute coordinate differences * wip * remove observers; composite distance; task_by_name * replace eventBuffer with coordMappings; compute tlbr-screen in browser.py * assign_browser_events in events.py * black; docstrings * flake8 * update test * black * add test_browser.py --------- Co-authored-by: pate1595 <kmp.patel@utoronto.ca> Co-authored-by: Krish Patel <65433817+KrishPatel13@users.noreply.github.com>




What kind of change does this PR introduce?
This PR implements the chrome extension to write DOM and DOM events to the database.
Summary
Motivation: Related Issue: #51
Checklist
Other information
Supercedes #364