Repository navigation
Implement model cursor for visual feedback #760
Description
Feature request
Update: see #760 (comment) for the latest requirements.
We want to be able to give the model the ability to:
- paint a red dot on its suggested target location
- look at the screenshot with the dot on it,
- optionally self correct.
Thank you @LunjunZhang for the suggestion 🙏
This involves creating a CursorReplayStrategy (based on the VanillaReplayStrategy) that implements the required prompting.
Motivation
Correct errors, e.g. missed segmentations.
Possibly related: https://arxiv.org/abs/2406.09403:
Humans draw to facilitate reasoning: we draw auxiliary lines when solving geometry problems; we mark and circle when reasoning on maps; we use sketches to amplify our ideas and relieve our limited-capacity working memory. However, such actions are missing in current multimodal language models (LMs). Current chain-of-thought and tool-use paradigms only use text as intermediate reasoning steps. In this work, we introduce Sketchpad, a framework that gives multimodal LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts planning and reasoning according to the visual artifacts it has drawn.
...
Sketchpad substantially improves performance on all tasks over strong base models with no sketching, yielding an average gain of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial reasoning (83.9%), and visual correspondence (80.8%). All codes and data are in this https URL.
Activity
abrichr commentedon Jun 17, 2024
/bounty $1000
algora-pbc commentedon Jun 17, 2024
💎 $1,000 bounty • OpenAdaptAI
Steps to solve:
/attempt #760with your implementation plan/claim #760in the PR body to claim the bounty❗ Important guidelines:
Thank you for contributing to OpenAdaptAI/OpenAdapt!
Subh231004 commentedon Jun 17, 2024
/attempt #760
Options
Anshgrover23 commentedon Jun 17, 2024
/attempt #760
Implementation Plan for Model Cursor Feedback (Issue #760)
Create CursorReplayStrategy: I'll develop a new CursorReplayStrategy class extending VanillaReplayStrategy.
Paint Red Dot: I'll implement a method to paint a red dot on the target location within a given image.
Screenshot Capture: I'll implement a method to capture a screenshot and overlay the red dot on it.
Self-Correction: I'll add an optional self-correction mechanism based on the screenshot with the dot.
Testing: I'll write and execute unit tests to ensure the functionality works as intended.
Documentation: I'll update the project documentation to include usage instructions for the new strategy.
Pull Request: I'll submit a PR for review, incorporating any feedback provided.
This plan will systematically address the issue by creating a targeted strategy, ensuring it functions correctly, and updating the documentation for users.
Options
[-]Implement model cursor feedback[/-][+]Implement model cursor for visual feedback[/+]abrichr commentedon Jun 20, 2024
@Subh231004 please keep the discussion related to your pull request on your pull request and not here. I have replied to your comment there.
onyedikachi-david commentedon Jun 25, 2024
/attempt #760
Ahmadkhan02 commentedon Jul 2, 2024
/attempt #760
4 remaining items
stdthoth commentedon Sep 12, 2024
/attempt #760
Options
stdthoth commentedon Sep 12, 2024
@abrichr i am working on it now... could you possibly assign this to me for a week ?
abrichr commentedon Sep 12, 2024
Hi @stdthoth , thank you! Can you please clarify your request?
I just updated the description to include more details about the current approaches, recreated here:
I believe the next step here is to systematically evaluate the performance of these in a repeatable way (e.g. programmatically). Only then will we be able to implement the requirement to:
Please let me know if you have any questions!
Edit: if you prefer, you can also implement a novel approach, without evaluating these ones. But we will be unable to award the bounty until we can confirm that your approach outperforms all of these.
Edit: https://visualsketchpad.github.io/ may perform very well.
hoklims commentedon Nov 19, 2024
/attempt #760
algora-pbc commentedon Nov 19, 2024
💡 @hoklims submitted a pull request that claims the bounty. You can visit your bounty board to reward.
Girma35 commentedon Jan 13, 2025
Hi , is this issues still open ?
Implement model cursor for visual feedback (OpenAdaptAI#760)
neoandmatrix commentedon May 17, 2025
Hey @abrichr is this issue still available? If it is then would like to give it a try according to updated requirements in #760 (comment). Should i directly go with a PR with my approach for this or first should discuss the approach?
abrichr commentedon May 23, 2025
@neoandmatrix thank you for your interest!
Please propose an approach first. Note that it should be distinct from those already implemented.
TanCodeX commentedon May 29, 2025
/attempt #760
blocator23 commentedon Aug 1, 2025
/attempt #760
blocator23 commentedon Aug 3, 2025
I'll develop a source code based on a classification model and Python library in order to:
MAVRICK-1 commentedon Aug 19, 2025
/attempt #760
abrichr commentedon Jan 17, 2026
Closing as part of the architecture transition to modular packages. The legacy codebase has been frozen at v0.46.0.
Model cursor visual feedback would be implemented in the new modular architecture.
For the new modular architecture, please open issues in the relevant package repository:
See PR #960 for details on the new architecture.