Description
This feature allows to add annotations / comments to the UI using Copilot, in order to provide feedback – much like adding a comment in Google Docs. The developer can then use these comments to improve the UI – or give the comments to an AI to implement.
Tier
Gated
License
Proprietary
Motivation
Background
Iterating on UI often requires implementing->trying->adjusting = "sanding". Sometimes the "feedback" on the UI is your own, sometimes it might come from a stakeholder – and increasingly, the recipient is an AI.
Problem
If you are working on your own, "sanding" either means going back and forth between testing and fixing, OR taking notes, then implementing changes in one go before testing again.
If you use an AI coding agent, you either tell the AI to implement a fix as you find it, wait, test some more, OR you take notes and ask the AI to implement everything before testing again.
Solution
Ability to add any number of comments to any view in an application, optionally indicating 0-N attachments (component selections) that are related to the comment.
Comments are stored in a file in the project (for now).
Attachments (selections) are marked in a way that aims to maximize the chance that an AI can understand what code to change, and to minimize the chance that they go "stale" when minor changes are made to the UI. This is one of the main challenges, and might include create- and attach-locations, hierarchy path, component id, debug id – needs some testing to find a suitable solution for now.
Notes
A working prototype exists.
Big picture / future
This is the first step in something that can grow in the future; we expect AI assisted workflows to go asynchronous, with developers/stakeholders providing feedback and an agent reacting to the feedback.
Even without AI, easily soliciting feedback from the team and stakeholders for work in progress is quite valuable.
This is a first small step towards that future.
Requirements
Nice-to-haves
Risks, limitations and breaking changes
Risks
- Comment attachments getting "stale", i.e. UI changes making the component references invalid, is perhaps the main challenge and risks limiting the usefulness. AIs should be able to take the comments from the file and implement, but there is a risk this does not work as well as we hope.
- File format not able to communicate well enough what needs to be changed.
Limitations
Breaking changes
Out of scope
A screenshot would greatly help both AI and humans to understand the intention in some cases, especially if the UI has drifted from how it looked when the comment was added. This is out-of-scope for now, as it's unclear how to best acquire a screenshot.
It would be nice to be able to draw on the UI or otherwise add annotations that are not tied to component selections, e.g point to an empty area. This is out-of-scope for now, as it requires screenshot or some other way to communicate non-structured annotations.
We will revisit the mechanism for storing / sharing comments in the future, for now a file-based approach is fine.
Materials
No response
Metrics
No response
Pre-implementation checklist
Pre-release checklist
Security review
None
Description
This feature allows to add annotations / comments to the UI using Copilot, in order to provide feedback – much like adding a comment in Google Docs. The developer can then use these comments to improve the UI – or give the comments to an AI to implement.
Tier
Gated
License
Proprietary
Motivation
Background
Iterating on UI often requires implementing->trying->adjusting = "sanding". Sometimes the "feedback" on the UI is your own, sometimes it might come from a stakeholder – and increasingly, the recipient is an AI.
Problem
If you are working on your own, "sanding" either means going back and forth between testing and fixing, OR taking notes, then implementing changes in one go before testing again.
If you use an AI coding agent, you either tell the AI to implement a fix as you find it, wait, test some more, OR you take notes and ask the AI to implement everything before testing again.
Solution
Ability to add any number of comments to any view in an application, optionally indicating 0-N attachments (component selections) that are related to the comment.
Comments are stored in a file in the project (for now).
Attachments (selections) are marked in a way that aims to maximize the chance that an AI can understand what code to change, and to minimize the chance that they go "stale" when minor changes are made to the UI. This is one of the main challenges, and might include create- and attach-locations, hierarchy path, component id, debug id – needs some testing to find a suitable solution for now.
Notes
A working prototype exists.
Big picture / future
This is the first step in something that can grow in the future; we expect AI assisted workflows to go asynchronous, with developers/stakeholders providing feedback and an agent reacting to the feedback.
Even without AI, easily soliciting feedback from the team and stakeholders for work in progress is quite valuable.
This is a first small step towards that future.
Requirements
Ability to add/remove/update comments on a view
Comments are persisted in a human- and AI-readable file in the project
Ability attach selections (component/element references) to comments
Comment attachments are referenced in a durable way, with the intent to maximize the chance humans and AIs find the relevant code.
Documentation
License check (login required)
Feature flag
Nice-to-haves
Risks, limitations and breaking changes
Risks
Limitations
Breaking changes
Out of scope
A screenshot would greatly help both AI and humans to understand the intention in some cases, especially if the UI has drifted from how it looked when the comment was added. This is out-of-scope for now, as it's unclear how to best acquire a screenshot.
It would be nice to be able to draw on the UI or otherwise add annotations that are not tied to component selections, e.g point to an empty area. This is out-of-scope for now, as it requires screenshot or some other way to communicate non-structured annotations.
We will revisit the mechanism for storing / sharing comments in the future, for now a file-based approach is fine.
Materials
No response
Metrics
No response
Pre-implementation checklist
Pre-release checklist
Security review
None