Please help me test tool-use (function calling)

[karthink]

Important

Tool-use related guidelines for this repo:

• Looking for tools? Check the wiki or llm-tool-collection.
• If you want to report a bug with tool-use in gptel, create a new issue (bug-report).
• If you have a question about writing, using or configuring tools, create an issue or a discussion.
• If you wish to share a tool you've written, feel free to add it to the wiki.
• If you write a nifty tool (or suite of tools) that you think other users might find useful, please raise a PR against llm-tool-collection. You can also add it to the wiki as in the above step.

This thread is closed. You can reply to existing messages in this thread but I won't be actively monitoring the conversation.

---

Note

Current status of tool-use:

This feature is now merged, and available in the master branch.

Backend	with streaming	without streaming	Structured arguments/output	Limitations
Anthropic	✓	✓	✓
OpenAI-compatible	✓	✓	✓
Gemini	✓	✓	Untested	Tools must take arguments
Ollama	❌	✓	Untested	streaming will be turned off

---

I've added tool-use/function calling support to all major backends in gptel -- OpenAI-compatible, Claude, Ollama and Gemini.

Demos

screencast_20241222T075329.mp4

gptel-tool-use-filesystem-demo.mp4

Same demo as the previous one, but with permissions and tool result inclusion turned on:

gptel-tool-use-filesystem-confirm-demo.mp4

Call to action

Please help me test it! It's on the feature-tool-use branch. There are multiple ways in which you can help. Ranked from least to most intensive:

1. Switch to the feature-tool-use branch and just use gptel as normal -- no messing around with tool use. Adding tool use required a significant amount of reworking in gptel's core, so it will help to catch any regressions first. (Remember to reinstall/re-byte-compile the package after switching branches!)

2. Switch to the branch, define a tool or two, and try using gptel (instructions below). Let me know if something breaks.

3. Same as 2, but suggest ways that the feature can be improved, especially in the UI department.

---

What is "tool use"?

"Tool use" or "function calling" is LLM usage where

• you include a function specification along with your task/question to the LLM.
• The LLM optionally decides to call the function, and supplies the function call arguments.
• You run the function call, and (optionally) feed the results back to the LLM. gptel handles this automatically.
• The LLM completes the task based on the information received.

You can use this to give the LLM awareness of the world, by providing access to APIs, your filesystem, web search, Emacs etc. You can get it to control your Emacs frame, for instance.

How do I enable it in gptel?

There are three steps:

0. Use a model that supports tool use. Most of the big OpenAI/Anthropic/Google models do, as do llama3.1 and the newer mistral models if you're using Ollama.

1. (setq gptel-use-tools t)

2. Write tool definitions. See the documentation of gptel-make-tool. Here is an example of a tool definition:

Tool definition example

(setq gptel-tools         ;; <-- Holds a list of tools
      (list        
       (gptel-make-tool   ;; <-- This is a tool definition
        :function (lambda (location unit)
                    (url-retrieve-synchronously (format "api.weather.com/..."
                                                        location unit)))
        :name "get_weather" ;; <-- Javascript style, snake_case name
        :description "Get the current weather in a given location"
        :args (list '(:name "location"
                      :type "string"
                      :description "The city and state, e.g. San Francisco, CA")
                    '(:name "unit"
                      :type "string"
                      :enum ("celsius" "farenheit")  ;; <-- enum types help reduce hallucinations, optional
                      :description
                      "The unit of temperature, either 'celsius' or 'fahrenheit'"
                      :optional t)))))

And here are a few simple tools for Filesystem/Emacs/Web access. You can copy and evaluate them in your Emacs session:

Code:

Some tool definitions, copy to your Emacs

(gptel-make-tool
 :function (lambda (url)
             (with-current-buffer (url-retrieve-synchronously url)
               (goto-char (point-min)) (forward-paragraph)
               (let ((dom (libxml-parse-html-region (point) (point-max))))
                 (run-at-time 0 nil #'kill-buffer (current-buffer))
                 (with-temp-buffer
                   (shr-insert-document dom)
                   (buffer-substring-no-properties (point-min) (point-max))))))
 :name "read_url"
 :description "Fetch and read the contents of a URL"
 :args (list '(:name "url"
               :type "string"
               :description "The URL to read"))
 :category "web")

(gptel-make-tool
 :function (lambda (buffer text)
             (with-current-buffer (get-buffer-create buffer)
               (save-excursion
                 (goto-char (point-max))
                 (insert text)))
             (format "Appended text to buffer %s" buffer))
 :name "append_to_buffer"
 :description "Append text to the an Emacs buffer.  If the buffer does not exist, it will be created."
 :args (list '(:name "buffer"
               :type "string"
               :description "The name of the buffer to append text to.")
             '(:name "text"
               :type "string"
               :description "The text to append to the buffer."))
 :category "emacs")

;; Message buffer logging tool
(gptel-make-tool
 :function (lambda (text)
             (message "%s" text)
             (format "Message sent: %s" text))
 :name "echo_message"
 :description "Send a message to the *Messages* buffer"
 :args (list '(:name "text"
               :type "string"
               :description "The text to send to the messages buffer"))
 :category "emacs")

;; buffer retrieval tool
(gptel-make-tool
 :function (lambda (buffer)
             (unless (buffer-live-p (get-buffer buffer))
               (error "Error: buffer %s is not live." buffer))
             (with-current-buffer  buffer
               (buffer-substring-no-properties (point-min) (point-max))))
 :name "read_buffer"
 :description "Return the contents of an Emacs buffer"
 :args (list '(:name "buffer"
               :type "string"
               :description "The name of the buffer whose contents are to be retrieved"))
 :category "emacs")


(gptel-make-tool
 :function (lambda (directory)
	     (mapconcat #'identity
                        (directory-files directory)
                        "\n"))
 :name "list_directory"
 :description "List the contents of a given directory"
 :args (list '(:name "directory"
	       :type "string"
	       :description "The path to the directory to list"))
 :category "filesystem")

(gptel-make-tool
 :function (lambda (parent name)
             (condition-case nil
                 (progn
                   (make-directory (expand-file-name name parent) t)
                   (format "Directory %s created/verified in %s" name parent))
               (error (format "Error creating directory %s in %s" name parent))))
 :name "make_directory"
 :description "Create a new directory with the given name in the specified parent directory"
 :args (list '(:name "parent"
	       :type "string"
	       :description "The parent directory where the new directory should be created, e.g. /tmp")
             '(:name "name"
	       :type "string"
	       :description "The name of the new directory to create, e.g. testdir"))
 :category "filesystem")

(gptel-make-tool
 :function (lambda (path filename content)
             (let ((full-path (expand-file-name filename path)))
               (with-temp-buffer
                 (insert content)
                 (write-file full-path))
               (format "Created file %s in %s" filename path)))
 :name "create_file"
 :description "Create a new file with the specified content"
 :args (list '(:name "path"
	       :type "string"
	       :description "The directory where to create the file")
             '(:name "filename"
	       :type "string"
	       :description "The name of the file to create")
             '(:name "content"
	       :type "string"
	       :description "The content to write to the file"))
 :category "filesystem")

(gptel-make-tool
 :function (lambda (filepath)
	     (with-temp-buffer
	       (insert-file-contents (expand-file-name filepath))
	       (buffer-string)))
 :name "read_file"
 :description "Read and display the contents of a file"
 :args (list '(:name "filepath"
	       :type "string"
	       :description "Path to the file to read.  Supports relative paths and ~."))
 :category "filesystem")

An async tool to fetch youtube metadata using yt-dlp

(defun my/gptel-youtube-metadata (callback url)
  (let* ((video-id
          (and (string-match
                (concat
                 "^\\(?:http\\(?:s?://\\)\\)?\\(?:www\\.\\)?\\(?:youtu\\(?:\\(?:\\.be\\|be\\.com\\)/\\)\\)"
                 "\\(?:watch\\?v=\\)?" "\\([^?&]+\\)")
                url)
               (match-string 1 url)))
         (dir (file-name-concat temporary-file-directory "yt-dlp" video-id)))
    (if (file-directory-p dir) (delete-directory dir t))
    (make-directory dir t)
    (let ((default-directory dir) (idx 0)
          (data (list :description nil :transcript nil)))
      (make-process :name "yt-dlp"
                    :command `("yt-dlp" "--write-description" "--skip-download" "--output" "video" ,url)
                    :sentinel (lambda (proc status)
                                (cl-incf idx)
                                (let ((default-directory dir))
                                  (when (file-readable-p "video.description")
                                    (plist-put data :description
                                               (with-temp-buffer
                                                 (insert-file-contents "video.description")
                                                 (buffer-string)))))
                                (when (= idx 2)
                                    (funcall callback (gptel--json-encode data))
                                    (delete-directory dir t))))
      (make-process :name "yt-dlp"
                    :command `("yt-dlp" "--skip-download" "--write-auto-subs" "--sub-langs"
                               "en,-live_chat" "--convert-subs" "srt" "--output" "video" ,url)
                    :sentinel (lambda (proc status)
                                (cl-incf idx)
                                (let ((default-directory dir))
                                  (when (file-readable-p "video.en.srt")
                                    (plist-put data :transcript
                                               (with-temp-buffer
                                                 (insert-file-contents "video.en.srt")
                                                 (buffer-string)))))
                                (when (= idx 2)
                                    (funcall callback (gptel--json-encode data))
                                    (delete-directory dir t)))))))

(gptel-make-tool
 :name "youtube_video_metadata"
 :function #'my/gptel-youtube-metadata
 :description "Find the description and video transcript for a youtube video.  Return a JSON object containing two fields:

\"description\": The video description added by the uploader
\"transcript\": The video transcript in SRT format"
 :args '((:name "url"
          :description "The youtube video URL, for example \"https://www.youtube.com/watch?v=H2qJRnV8ZGA\""
          :type "string"))
 :category "web"
 :async t
 :include t)

As seen in gptel's menu:

See the documentation for gptel-make-tool for details on the keyword arguments.

Tip

@jester7 points out that you can get the LLM to write these tool definitions for you, and eval the Org Babel blocks to use them right away.

Important

Please share tools you write below so I can use them to test for issues.

In this case, the LLM may choose to ask for a call to get_weather if your question is related to the weather, as in the above demo video. You can help it along by saying something like:

Use the provided tools to accomplish this task: ...

Notes

• Tools can be asynchronous, see the documentation of gptel-make-tool for an example.
• You can use the tool definition schema to get the LLM to generate JSON for you. This is one of the ways to get LLM APIs to generate JSON output.
• Right now tool call results are automatically sent back to the LLM. We'll add a way to make this optional, for when the tool needs to run for side-effects only. For now you can make your tool return nil if you want to run it for side-effects only.
• It is possible to force the LLM to use provided tools -- this switch has not yet been implemented.
• LLMs may use tools in parallel if multiple tools are specified -- this is fully supported in gptel.

[karthink]

@agzam -- in case you're interested.

[agzam]

Oh wow, this is very cool. So many interesting ideas to try. I'm excited, will give it a go. Thank you!

[metachip]

Excellent. Will give it a go.

[jester7]

Wow, this works great. I just created tools for being able to cat files, ls, and creating new files. I had trouble with OpenAI calling the tools but Claude Sonnet works fine. I even had Claude write a couple of tools, then eval the emacs lisp blocks inside the org mode buffer, and have Claude immediately start using them.

I'll help you test these tools step by step:

1. First, let's create the subdirectories "1" and "2" inside claude-scripts:

Now let's create hello world files in each directory:

Let's verify the structure using the tree_view tool:

Let's verify the content of both files:

Great! I've tested all the tools:

1. Created directories "1" and "2"
2. Created hello.txt files in each directory with different content
3. Used tree_view to show the structure
4. Read both files to verify their contents

All tools are working as expected! The tree view shows the correct directory structure, and the file contents are exactly as we wrote them. Would you like to try something else with these tools?

[karthink]

I even had Claude write a couple of tools, then eval the emacs lisp blocks inside the org mode buffer, and have Claude immediately start using them.

Holy bootstrap, Batman!

I had trouble with OpenAI calling the tools

Did it throw an error or just ignore the tools? If it was silent failure you can check the *gptel-log* buffer after running (setq gptel-log-level 'info).

[jester7]

It seems to be calling the tools but fails at the end. It also failed with gemini, llama, and qwen for me but I have to double check my tools because for simpler use cases I think it was working a while ago. This same prompt works fine with the Claude models sonnet and haiku.

Here is the Messages buffer and attached is the log
gptel-tool-use-log-openai.txt
:
Querying OpenAI... gptel: moving from INIT to WAIT gptel: moving from WAIT to TYPE gptel: moving from TYPE to TOOL error in process sentinel: let: Wrong type argument: stringp, nil error in process sentinel: Wrong type argument: stringp, nil

[karthink]

It seems to be calling the tools but fails at the end. It also failed with gemini, llama, and qwen for me but I have to double check my tools because for simpler use cases I think it was working a while ago. This same prompt works fine with the Claude models sonnet and haiku. Here is the *Messages* buffer and attached is the log [gptel-tool-use-log-openai.txt](https://github.com/user-attachments/files/18225771/gptel-tool-use-log-openai.txt) : ` Querying OpenAI... gptel: moving from INIT to WAIT gptel: moving from WAIT to TYPE gptel: moving from TYPE to TOOL error in process sentinel: let: Wrong type argument: stringp, nil error in process sentinel: Wrong type argument: stringp, nil

Strange, that data stream does not look like what I'd expect at all. As a result, the tool call from OpenAI is not being parsed correctly. I'll look into this. Independent of that, it looks like there's an error in gptel's tool call handler. Could you repeat this experiment but after M-x toggle-debug-on-error, and paste the backtrace here?

[jester7]

Update: it seems Gemini, Llama, and Qwen models work only if I make a request that requires a single tool call. For example I did a request to each to summarize a URL and do a directory listing on my local machine and these types of interactions work.

[karthink]

it seems Gemini, Llama, and Qwen models work only if I make a request that requires a single tool call

Could you try it with this OpenAI backend?

(gptel-make-openai "openai-with-parallel-tool-calls"
  :key YOUR_OPENAI_API_KEY
  :stream t
  :models gptel--openai-models
  :request-params '(:parallel_tool_calls t))

Parallel tool calls are supposed to be enabled by default, so I'm not expecting that this will work, but it would be wise to verify.

[karthink]

Here is the Messages buffer and attached is the log
gptel-tool-use-log-openai.txt

Could you also share the tool definitions you used in this failed request? I'd like to try reproducing the error here.

[ProjectMoon]

For ollama, I see the tool being sent to ollama in the gptel log buffer, but none of the models ever actually seem to use the tools. Have tried with Mistral Nemo, Qwen 2.5, Mistral Small, Llama 3.2 vision.

[karthink]

@ProjectMoon Could you share the tools you wrote so I can try to reproduce these issues?