Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/browser replay extended #875

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Feat/browser replay extended #875

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a3ab704
add BrowserReplayStrategy; support browser modes record/replay
abrichr Sep 11, 2024
2592b08
minor refactor
abrichr Sep 11, 2024
6dd5e38
Merge branch 'main' into feat/browser_replay
abrichr Sep 11, 2024
4d1f96a
black/flake8
abrichr Sep 11, 2024
a5e28b9
update README
abrichr Sep 11, 2024
a8ca9d3
improve README
abrichr Sep 11, 2024
0f42308
add BrowserReplayStrategy to README
abrichr Sep 11, 2024
1d0e7c9
wip
abrichr Sep 16, 2024
0c42483
wip
abrichr Oct 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add BrowserReplayStrategy to README
  • Loading branch information
@abrichr
abrichr committed Sep 11, 2024
commit 0f42308aff34166e0420460adb04a8d4681c99fc
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,7 @@ Other replay strategies include:
- [`StatefulReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/stateful.py): Early proof-of-concept which uses the OpenAI GPT-4 API with prompts constructed via OS-level window data.
- (*)[`VisualReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/visual.py): Uses [Fast Segment Anything Model (FastSAM)](https://github.com/CASIA-IVA-Lab/FastSAM) to segment active window.
- (*)[`VanillaReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/vanilla.py): Assumes the model is capable of directly reasoning on states and actions accurately. With future frontier models, we hope that this script will suddenly work a lot better.
- (*)[`BrowserReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/browser.py): Uses the browser extension to read the visible DOM, and refers to recorded browser events to identify target elements.


The (*) prefix indicates strategies which accept an "instructions" parameter that is used to modify the recording, e.g.:
Expand Down
Loading