Skip to content

Create chat message attachments via drag-and-drop from files, notebook cells #248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Jul 7, 2025

Conversation

andrii-i
Copy link
Collaborator

@andrii-i andrii-i commented Jun 30, 2025

Problem

There is no way to quickly drag content from the file browser or notebook cells directly into the chat input area, users need to manually attach files and notebook cells to chat messages using the attach button.

Proposed solution

Use updated attachments API supporting cells and selection ranges introduced in #247 to allow users to drag content from the file browser or notebook cells directly into the chat input area to create attachments:

  • Drag files from file browser to create file attachments
  • Drag notebook cells (single or multiple) to create cell attachments

Testing instructions:

  • Attach a file: drag and drop file from file browser into chat input
  • Attach a cell: drag and drop a cell (or multiple cells) from a notebook into chat input
  • Attach multiple cells: drag and drop multiple cells from a notebook into chat input (all cells from the same notebook would be added as a single attachment)

dnd_jul3

Not implemented

Ability to drag text selection to create an attachment is not implemented in this PR as JupyterLab's drag events only carry plain text, no context about which cell/notebook they came from. Since Jupyter Chat attachments are references to specific locations in files we can't create meaningful attachments from dragged text selection as they are essentially plain text with unknown origin.

This can be solved by enhancing JupyterLab's drag system upstream and implementing custom MIME type.

@andrii-i andrii-i added the enhancement New feature or request label Jun 30, 2025
Copy link
Contributor

Binder 👈 Launch a Binder on branch andrii-i/jupyter-chat/dragndrop

@andrii-i andrii-i changed the title Dragndrop Create chat message attachments via drag-and-drop for files and cells Jun 30, 2025
@andrii-i andrii-i changed the title Create chat message attachments via drag-and-drop for files and cells Create chat message attachments via drag-and-drop from files, notebook cells Jun 30, 2025
@andrii-i
Copy link
Collaborator Author

andrii-i commented Jul 3, 2025

Updated demo video is in the PR description, now it shows current / latest state of this PR.

@andrii-i andrii-i requested review from JGuinegagne and ellisonbg July 3, 2025 09:43
@andrii-i andrii-i force-pushed the dragndrop branch 2 times, most recently from 41e5bfe to 093a40b Compare July 3, 2025 10:35
Copy link
Member

@dlqqq dlqqq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrii-i Thank you for building this! The new UX is awesome. Left some feedback below, mostly concerning type safety.

BTW:

Ability to drag text selection to create an attachment is not implemented in this PR as JupyterLab's drag events only carry plain text, no context about which cell/notebook they came from. Since Jupyter Chat attachments are references to specific locations in files we can't create meaningful attachments from dragged text selection as they are essentially plain text with unknown origin.

I checked on this and I couldn't find a reference for where text selections are handled by JupyterLab (searching for mimedata.setData()). This may actually be native behavior from the browser, since text drags seems to be handled as "cut and paste" by default in browsers.

I think this is OK. Since modern LLM token limits are higher, users should attach the entire file/cell to maximize the context the LLM receives anyways. The persona may be expected to intelligently handle large files by running a grep call first to see which lines in a large file are relevant.

If we're able to reach consensus that this behavior is OK, we should open a follow-up issue to remove the selection fields from the IAttachment models, since they will not be used.

@ellisonbg
Copy link
Collaborator

I have been testing this locally, a few comments and questions:

  • I love the ability to drag a cell. I also see you can attach multiple cells from the same notebook, but they appear as separate attachments. In the underlying data model, is is a single notebook file attachment with a list of cell attachments, or multiple notebook file attachments each with a single cell?
  • Eventually (can be in a separate PR) it would be nice to also drag entire files that are already open, rather than just from the file browser.
  • I see the "x" button to remove an attachment has a small click area. That can also be a follow up issue/PR.
  • What exact format are the paths in? Relative to server root? URL style with forward slashes (even on Windows)? URL encoding?

@andrii-i
Copy link
Collaborator Author

andrii-i commented Jul 3, 2025

Thank you for looking into this @ellisonbg.

I love the ability to drag a cell. I also see you can attach multiple cells from the same notebook, but they appear as separate attachments. In the underlying data model, is is a single notebook file attachment with a list of cell attachments, or multiple notebook file attachments each with a single cell?

Currently if multiple cells are dragged into the chat, single INotebookAttachment is created with cells added as INotebookAttachment.cells: INotebookAttachmentCell[]. Here's a video and a screenshot of this interaction:

image
dragndrop_mult_cells

Eventually (can be in a separate PR) it would be nice to also drag entire files that are already open, rather than just from the file browser.

Currently one could select all cells of a notebook and it would be attached as a single instance of INotebookAttachment with multiple cells. There is no way to drag entire non-notebook files that are already open, would make sense if there would be. I'd say UI needs discussion so it would make sense to follow up in a separate PR.

I see the "x" button to remove an attachment has a small click area. That can also be a follow up issue/PR.

I agree with this, would prefer to follow up on this in a separate issue/PR as attachment previews are currently displayed via custom AttachmentPreviewList component

/**
* The Attachments component.
*/
export function AttachmentPreviewList(props: AttachmentsProps): JSX.Element {
return (
<Box className={ATTACHMENTS_CLASS}>
{props.attachments.map(attachment => (
<AttachmentPreview {...props} attachment={attachment} />
))}
</Box>
);
}
similar to MUI Chips. I'd say that using MUI Chips and other pre-made components as much as possible would improve usability, functionality, and UIUX overall.

What exact format are the paths in? Relative to server root? URL style with forward slashes (even on Windows)? URL encoding?

Attachment paths are:

  1. Paths are relative to ContentsManager.root_dir as documented in the interfaces
  2. Re forward slashes, JupyterLab uses POSIX-style paths internally regardless of OS. Server Contents API takes care of that
  3. No URL encoding, JupyterLab style POSIX paths are stored raw

This follows JupyterLab's path conventions.

@ellisonbg
Copy link
Collaborator

ellisonbg commented Jul 3, 2025 via email

@andrii-i
Copy link
Collaborator Author

andrii-i commented Jul 3, 2025

@ellisonbg that's correct, I haven't understood the question. Multiple separate drags would create multiple INotebookAttachment objects, each with one cell. Agree that it would make sense to immediately group them based on the notebook, this would require additional logic so maybe in a separate PR.

@ellisonbg
Copy link
Collaborator

Separate PR is fine for that.

Copy link

@JGuinegagne JGuinegagne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor non-blocking comments

@andrii-i
Copy link
Collaborator Author

andrii-i commented Jul 4, 2025

Updated demo / video preview is in the PR description.

@ellisonbg
Copy link
Collaborator

I have tested this locally again. As long as the above code comments have been addressed, I think this is ready to go. @andrii-i can you resolve the comments above that have been addressed? Once those are clean I can merge.

The UX of this is soo nice - can't wait to use it :-)

@ellisonbg
Copy link
Collaborator

Looks good, merging

@ellisonbg ellisonbg merged commit 2b54aa4 into jupyterlab:main Jul 7, 2025
13 of 14 checks passed
@andrii-i andrii-i deleted the dragndrop branch July 7, 2025 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants