Request for comment: Use Org mode properties drawer to configure system prompt/temp for the whole subtree

[doctorguile]

I ran into the same bug as #135

But ultimately even if I didn't run into this bug, I think the transient menu is not as convenient if you have a big list of predefined combos.

For an example of very convenient way to have access to a set of user defined system prompt/model/temperature, etc see

https://github.com/Bin-Huang/chatbox

where you can set a system prompt , temperature and model for each chat thread, and continue on that thread without having to repeat the context (sys prompt, etc)

the ideal way to do this in emacs is in org mode, having a header and set system prompt , temperature and model as property drawer.

then all subtree of that header will inherit these properties and you can have a lot of useful prompt / temperature always ready to go

example

* Emacs
:PROPERTIES:
:SYSPROMPT: You are a emacs expert. You can help me by answering my questions. You can also ask me questions to clarify my intention.
:temperature: 0.1
:model: GPT4
:END:

** How to call REST api?
conversation re emacs continue in this subtree...

* Python
:PROPERTIES:
:SYSPROMPT: you are a python expert. respond to the task with detailed step by step instructions and code. list out all the files to be created and how to put them together
:temperature: 0.2
:model: GPT3.5
:END:

** How to setup a webserver with http3 support?
conversation re python continue in this subtree...

Does anyone think this is a good addition? This is how org is used (inheriting parent props) for a lot of other use cases

[karthink]

This works on a per-file basis in Org-mode right now when you save the buffer. (Except for Ollama models, where it's more annoying to capture state.)

I'm not convinced it's worth adding per-heading support, but I'll think about it.

[NightMachinery]

@karthink I think it's worth adding. I like to have a mega file where I store all my chats, and each chat is a new heading. Opening a new file for small chats is not worth it.

I also put chats under headings where they make sense. For example, I have a heading How to do X and I have a subheading LLM Chatlog and under that GPT4 and under this heading I have the actual chat. Linking to a file instead makes things messy.

[Ypot]

Take a look at "inline-tasks". That would allow to change the sysprompt by adding an "isolated" subheading, which means that you can end the body of the headline using a: "*************** END".

I agree with the inheritance.

An example, in an old suggestion:
#103 (comment)

[karthink]

I also put chats under headings where they make sense. For example, I have a heading How to do X and I have a subheading LLM Chatlog and under that GPT4 and under this heading I have the actual chat. Linking to a file instead makes things messy.

This sounds good to me, but I'm not sure I understand what you mean fully. You can already have an LLM Chatlog > GPT4 subheading, and store/continue the conversation there, if you have added a GPTEL_TOPIC property to the heading. (With org-set-property or more easily with gptel-set-topic)

What's missing here is the specification of the model and backend parameters for this specific chat. So if these are not the default values, you would have to change them manually from elisp or from the transient menu. If other chat logs in the same file use different models/backends, you'd have to switch manually each time before continuing those chats. Is this the problem you want to solve?

[NightMachinery]

I also put chats under headings where they make sense. For example, I have a heading How to do X and I have a subheading LLM Chatlog and under that GPT4 and under this heading I have the actual chat. Linking to a file instead makes things messy.

This sounds good to me, but I'm not sure I understand what you mean fully. You can already have an LLM Chatlog > GPT4 subheading, and store/continue the conversation there, if you have added a GPTEL_TOPIC property to the heading. (With org-set-property or more easily with gptel-set-topic)

What's missing here is the specification of the model and backend parameters for this specific chat. So if these are not the default values, you would have to change them manually from elisp or from the transient menu. If other chat logs in the same file use different models/backends, you'd have to switch manually each time before continuing those chats. Is this the problem you want to solve?

Yes, I like to be able to set all the parameters using property drawers. I want the chats to record all their inputs in the text file, so that years later, I can revisit them and know exactly what parameters etc. I used.

---

Another problem is that the prefix (gptel-prompt-prefix-alist) needs to be hardcoded to a single string, e.g., *** . Setting this to a regex, e.g., ^\*+\s+ seems more sensible. This would allow us to put the chats in any heading depth.

[karthink]

Yes, I like to be able to set all the parameters using property drawers.

There are some ambiguities in the behavior of gptel that I need to sort out to do this, I'll explain soon.

I want the chats to record all their inputs in the text file, so that years later, I can revisit them and know exactly what parameters etc. I used.

This (archival) is a different -- and simpler -- issue from being able to set the parameters. You can do this right now with a little elisp:

(org-entry-put nil "GPTEL_TEMPERATURE" gptel-temperature)
(org-entry-put nil "GPTEL_MODEL" gptel-model)
(org-entry-put nil "GPTEL_BACKEND" gptel-backend)

and so on, placed in a function in before-save-hook perhaps. (gptel-backend is a semi-opaque structure so you might prefer to write its fields as properties instead.)

Another problem is that the prefix (gptel-prompt-prefix-alist) needs to be hardcoded to a single string, e.g., *** . Setting this to a regex, e.g., ^*+\s+ seems more sensible. This would allow us to put the chats in any heading depth.

I don't see how a regexp can work. What is the corresponding string that will be inserted?

Note that the prompt prefix doesn't have to be a heading at all, it can be a string like "Prompt:" (with the corresponding response prefix as "Response:\n") and you can have chats at any heading depth.

[karthink]

where you can set a system prompt , temperature and model for each chat thread, and continue on that thread without having to repeat the context (sys prompt, etc)

the ideal way to do this in emacs is in org mode, having a header and set system prompt , temperature and model as property drawer.

Okay, I gave this some thought, and there are many subtle ambiguities in the exact behavior of gptel in the presence of the suggested org properties that I need to resolve before I can add it.

---

To explain these ambiguities, here is some background:

gptel has slightly different behavior depending on whether gptel-mode is turned on. The main differences are the visual indicators (header bar or mode-line) and saving the state of the chat to disk when writing the buffer to disk.

---

Presently ambiguous or undesirable behaviors:

1. With gptel-mode turned on, the user writes an org buffer to disk. The prompt/response boundaries don't depend on the backend and will always be written at (point-min). Under which heading should the other model parameters be written?

2. Should the behavior be the same irrespective of the state of gptel-mode? If yes, this will mean always searching for Org properties up the document tree before sending a request. Even with an org-element cache this is a small burden on the users who use gptel quite simply in Org mode, which is the majority. Without org-element caching parsing the buffer can be a lot of extra work.

3. Suppose the user redirects the prompt/response using the transient menu, such as reading from and echoing the response to the minibuffer. This interaction is basically independent of the current buffer. Should the model parameters be read from the local Org properties? How about when using partial buffer text as a prompt but redirecting the response to a new gptel session? How about when selecting a region that spans two Org headings with two different sets of specified model parameters?

4. Model parameters are specified as Org properties at some heading level in the document. The user uses the transient menu to change the active backend/model. What takes priority? If the values set from the menu are ignored in favor of the org properties, how do you avoid confusion when the response isn't what the user expects? Consider that the properties might be set several levels above where the user is currently typing. This essentially introduces a second, independent and overriding way of setting model parameters to gptel.

There are a couple more ambiguities to resolve, but I'd like to get some input on the above to find the least-surprising behavior.

[NightMachinery]

@karthink

What is the corresponding string that will be inserted?

Nothing. I can easily create the new heading using org-mode hotkeys itself, there is no need to insert it automatically.

Note that the prompt prefix doesn't have to be a heading at all, it can be a string like "Prompt:" (with the corresponding response prefix as "Response:\n") and you can have chats at any heading depth.

This is a good workaround, but I prefer headings, as they integrate into org naturally.

Plus, in the future, headings allow for some fancy features. E.g., we can use tags to exclude a subtree from the conversation history. We can use the natural branching of the org headings to create branching conversations. (By branching, I mean like when you edit a message in ChatGPT’s website and the UI creates a new branch of the conversation.)

[karthink]

Take a look at "inline-tasks". That would allow to change the sysprompt by adding an "isolated" subheading, which means that you can end the body of the headline using a: "*************** END".

My understanding from talking to Org maintainers is that inline tasks should be avoided whenever possible -- they're having some trouble with parsing that syntax reliably. This is similar to how the preferred LaTeX math delimiters in Org are \( and \) and how $...$ should be avoided, even though it's allowed. So I'm inclined to find other solutions.

[karthink]

Nothing. I can easily create the new heading using org-mode hotkeys itself, there is no need to insert it automatically.

Then you can (setf (alist-get 'org-mode gptel-prompt-prefix-alist nil t) nil), i.e. don't use any prefix.

Plus, in the future, headings allow for some fancy features. E.g., we can use tags to exclude a subtree from the conversation history.

This is a good idea, but I'd prefer not to add more special behavior, like tags with special meanings for gptel. Perhaps we can reuse a state that already has special meaning in Org, like excluding a subtree if it's commented (as in * COMMENT Some title).

We can use the natural branching of the org headings to create branching conversations. (By branching, I mean like when you edit a message in ChatGPT’s website and the UI creates a new branch of the conversation.)

I already do this often with gptel-set-topic.

EDIT: I might have misunderstood what you meant by "branching conversations". When in a branch, I'm guessing the context from before the branch point is included in the conversation. Setting GPTEL_TOPIC simply creates a new conversation starting at that heading, so it's not what you suggested.

[Ypot]

Take a look at "inline-tasks". That would allow to change the sysprompt by adding an "isolated" subheading, which means that you can end the body of the headline using a: "*************** END".

My understanding from talking to Org maintainers is that inline tasks should be avoided whenever possible -- they're having some trouble with parsing that syntax reliably. This is similar to how the preferred LaTeX math delimiters in Org are \( and \) and how $...$ should be avoided, even though it's allowed. So I'm inclined to find other solutions.

Hi!
Inline tasks are quite popular between advanced users and they will stay. What is in discussion is their syntax, a new one is being in discussion:
https://lists.gnu.org/archive/html/emacs-orgmode/2023-08/msg00708.html

But if you agree inline tasks could be the perfect tool, it would be necessary in the future just adjusting to the new syntax.