Media tracking implementation not compatible with customized org-attach-id-dir

[bgutter]

Please update gptel first -- errors are often fixed by the time they're reported.

• I have updated gptel to the latest commit and tested that the issue still exists

Bug Description

I believe that gptel-send does not correctly locate media files referenced by org-mode "attachment" links when using org-attach's "ID based" storage support. Instead, gptel will gather an incorrect path to the image file, determine that the file is not readable, and then fail to encode and send it to the LLM.

I spent some time debugging this and have a fix that works for me, but I am not sure if it is appropriate.

First, in gptel--parse-media-links, make sure to use org-expand if the org entity is an attachment. Without this, gptel will try to open a relative file path that does not exist.

        (when-let* ((link (org-element-context))
                    ((gptel-org--link-standalone-p link))
                    (raw-link (org-element-property :raw-link link))
                    (raw-path (org-element-property :path link))
                    (type (org-element-property :type link))
                    (path (if (string= type "attachment")
                              (org-attach-expand raw-path)
                            raw-path))

Second, in gptel--realize-query, set the current buffer appropriately before calling gptel--parse-buffer. Without this, org-attach-expand will not function correctly.

          (setq full-prompt (with-current-buffer (plist-get info :buffer)
                              (gptel--parse-buffer ;prompt from buffer or explicitly supplied
                               gptel-backend (and gptel--num-messages-to-send
                                                  (* 2 gptel--num-messages-to-send)))))

If this issue report and proposed resolution passes your sniff test, I can put together a pull-request.

Backend

None

Steps to Reproduce

1. Customize org-attach-id-dir to some random location on your machine, not in the same working directory as your chat buffer
2. Attach an image file and link to it
3. Enable media tracking
4. Try to send it, asking "What is in this image?"

Without workaround, llm will say that it cannot see images. With workaround, it will describe image content.

Alternatively, you can set gptel-expert-commands to t and use the JSON inspect query support;

1. Perform steps 1-3 above
2. Inspect query JSON in the gptel transient

Without workaround, JSON payload will reflect that the attachment link was not detected as media. You'll just see the link text in the preceding :text node. With workaround, the media will be represented as an encoded entry.

Additional Context

Reproduced on 2 machines

• NixOS, Emacs 30.1
• Windows 11, Emacs 29.something IIRC

Backtrace

Log Information

[karthink]

when using org-attach's "ID based" storage support.

Can you give me an example of such a link?

[bgutter]

Yes, here is an example conversation showing a non-working org-attach link, as well as a plain file link which does work.

:PROPERTIES:
:GPTEL_MODEL: gpt-4o-mini
:GPTEL_BACKEND: ChatGPT
:GPTEL_SYSTEM: You are a large language model living in Emacs and a helpful assistant. Respond concisely.
:END:

*** What is this?                                                    :ATTACH:
:PROPERTIES:
:ID:       c5b532d5-5788-440f-b6ad-929a9373c265
:END:

[[attachment:test.png]]


It appears to be a reference to an attachment in a document or note, specifically an image file named "test.png." The syntax suggests it's likely from a system that supports media attachments, such as Org mode in Emacs.

*** Try again

[[/home/brandon/tmp/test.png]]

This is an image of a cat, specifically a breed known for its distinct features and striking appearance. If you have any questions about cats or specific breeds, feel free to ask!

***

In the conversation above, [[attachment:test.png]] is saved on disk at /home/brandon/tmp/org/attachments/c5/b532d5-5788-440f-b6ad-929a9373c265/test.png. It was stored there by calling org-attach, selecting 'org-attach-method (a)', choosing the file test.png from my filesystem. I then called org-insert-link to insert an attachment link to the attached version of the file.

Here is a minimal init.el to reproduce;

(customize-set-variable 'org-attach-id-dir "/home/brandon/tmp/org/attachments/")
(customize-set-variable 'gptel-track-media t)
(customize-set-variable 'gptel-default-mode 'org-mode)

(defun new-conversation ()
  (interactive)
  (let* ((file-path (concat "/home/brandon/tmp/llm-chats/" (format-time-string "%Y/%m/%d/%I-%M-%S-%p") "/chat.org"))
         (directory (file-name-directory file-path)))
    (unless (file-exists-p directory)
      (make-directory directory t))
    (switch-to-buffer (gptel "*ChatGPT*"))
    (write-file file-path)))

The function new-conversation creates a gptel buffer and saves it to a file, which is required for org-attach.

[karthink]

Thanks for the explanation.

This is tricky to do correctly. The parsing sequence works like this:

1. Copy prompt to a temp buffer
2. Strip property drawers from the buffer, and/or other elements in gptel-org-ignore-elements.
3. Call gptel--parse-buffer, which calls gptel--parse-media-links.

So attachment link resolution will fail for two reasons:

1. The parsing happens in a temp buffer, not the original file. There are many reasons this is required.
2. All property blocks are gone (by default), so there is no way to find the attachment directory (via the ID or DIR properties).