Updating generic agent to use the new goal_object #21
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ThibaultLSDC
commented
Sep 17, 2024
| }, | ||
| { | ||
| "type": "image_url", | ||
| "image_url": {"url": image["base64"]}, |
There was a problem hiding this comment.
add image detail
|
|
gasse
changed the title
gasse
force-pushed
the
goal_images_update
branch
from
84646f8
to
3a68405
Compare
ThibaultLSDC
commented
Oct 11, 2024
Comment on lines
+128
to
+136
|
|
||
| res = [] | ||
| for m in messages: | ||
| if isinstance(m, list): | ||
| assert all(el["type"] == "text" for el in m) | ||
| res.append("\n".join(el["content"] for el in m)) | ||
| else: | ||
| res.append(m["content"]) | ||
| return "\n".join(res) |
There was a problem hiding this comment.
@recursix could we get your opinion there ?
recursix
reviewed
Oct 15, 2024
| @@ -74,10 +74,15 @@ def obs_preprocessor(self, obs: dict) -> dict: | |||
| @cost_tracker_decorator | |||
| def get_action(self, obs): | |||
|
|
|||
| goal_object = obs.pop("goal_object", []) | |||
| if len(goal_object) == 0: | |||
There was a problem hiding this comment.
just remove [] and let the error raise itself
recursix
reviewed
Oct 15, 2024
Comment on lines
+131
to
+135
| if isinstance(m, list): | ||
| assert all(el["type"] == "text" for el in m) | ||
| res.append("\n".join(el["content"] for el in m)) | ||
| else: | ||
| res.append(m["content"]) |
There was a problem hiding this comment.
make a generic function prompt_obj_to_str in llm_util.py that can be reused here and in e.g. fit-token
recursix
reviewed
Oct 15, 2024
| @@ -150,3 +159,11 @@ def parser(response: str) -> tuple[dict, bool, str]: | |||
|
|
|||
| def experiment_config(): | |||
| return exp_args | |||
recursix
reviewed
Oct 15, 2024
| ] | ||
| self._prompt = self._update_detail(self._prompt) | ||
|
|
||
| def _update_detail(self, prompt): |
There was a problem hiding this comment.
no need to pass self._prompt
recursix
reviewed
Oct 15, 2024
| msg = msg.get("content", "No Content") | ||
| if isinstance(msg, str): | ||
| msg = [{"type": "text", "text": msg}] | ||
| messages.append(f"# Message {i}\n") |
There was a problem hiding this comment.
this codes makes me feel like a lot of this code should be encapsulated in a prompt object.
recursix
reviewed
Oct 15, 2024
| @@ -7,6 +7,13 @@ | |||
| from agentlab.agents.generic_agent.generic_agent_prompt import GenericPromptFlags, MainPrompt | |||
| from agentlab.llm.llm_utils import count_tokens | |||
|
|
|||
|
|
|||
| def merge_texts(texts): | |||
There was a problem hiding this comment.
have a single one of this in llm utils
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This will be necessary if we want to run our agent on VWA.
It slightly alters the behavior of the original generic_agent, we might want to look into it.