Closed
Description
Feature request
From https://ateam-members.slack.com/archives/C050HTZ8HE0/p1708072362007929:
Google released Gemini 1.5 and tested it with 10 million tokens: https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/#sundar-note
Gemini v1 API documentation: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini#sample_requests
PROBLEM STATEMENT:
We wish to implement an openadapt.adapter
for Gemini 1.5, and parameterize/refactor StatefulReplayStrategy
.
COMPLETION CRITERIA:
This Issue will be complete once there is:
- An
openadapt.adapter
that implements the Gemini API - A refactored
StatefulReplayStrategy
that:
a. supports the larger context window size
b. implements a more general prompt (e.g. implement screenshots in https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/stateful.py#L126 and remove / simplify https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/events.py#L719)
Motivation
Support the largest context window model available in order to include full resolution Set-of-Mark prompts (i.e. via #543 + microsoft/SoM#19) across full recording.