Question about how core models are setup

[krschacht]

@Rmpanga, you were asking:

One question I did have was the model structure.

I see the attendant was for Conversations to mirror OpenAI threads. However I noticed that Assistants do not mirror OpenAI assistants

Was the original intent of Assistants to mirror OpenAI assistants?

[krschacht]

When I created all these models initially, my intention was to mirror the OpenAI schema closely, but I did intentionally diverge from it in a few small ways. This was the original PR and I expanded on that a bit:

#10

Off the top of my head, there were two tricky things: one was minor in that “Thread” is a core class in ruby so I renamed that to “Conversation”. The other tricky thing is a bit more squishy…

OpenAI spec’d out a whole bunch of models for use with their new Assistants API but as I started to implement it I found that they hadn’t finished it. A bunch of basic functionality wasn’t supported through the new API so I fell back to their old “chat” API but I used the new models I had created anyway, just to future proof.

But just last month (April) they finally released Assistants API v2.0: https://platform.openai.com/docs/assistants/whats-new

I have not dug into this yet, but ruby-openai just released support for it so that should make our lives a bit easier. I am not sure yet in what ways they may have shifted their schema a bit.

[krschacht]

I guess one thing I should say too is: let me know what differences that you’re seeing and we can either plan to address those or I can share if there was an intent behind it.

[Rmpanga]

Hi @krschacht , hope you had a good weekend.

I have noticed a significant differences between the "chat" api and vs OpenAI Assistant api. The biggest one was Invocation differences. I summarized it below

• Chat: The chat api is pretty simple, you simple call `@client. chat(...) and receive a response.

• Assistant: A bit more complex, I summarized below

  • 1. You need to create a OpenAI Thread object: thread = @client.threads.create
  • 2. Then create an OpenAI Message object(s) associated with a thread message = @client.messages.create(...., thread["id"])
  • 3. Create an OpenAI Run object that is associated with the Thread & OpenAI assistant Id run = @client.runs.create(thread_id: thread_id, assistant_id: assistant_id )
  • 4. Lastly either poll or stream the response

  I skipped a few details like adding configuration params such as completion tokens & tools etc but that's the general summary.

  Edit: Just in case its not clear @client = OpenAI::Client.new(...)

[Rmpanga]

Now back to the Assistants model. If we were to support the new v2 API for Assistant would you recommend reusing the Assistant model?

Currently assistants seems to store and display chat api conversations between users and AI models (ChatGPT, Anthropic, etc). But I may be missing something

[krschacht]

I'm still wondering what you meant by this:

I noticed that Assistants do not mirror OpenAI assistants

I just checked and our schema is:

    t.bigint "user_id", null: false
    t.string "model"
    t.string "name"
    t.string "description"
    t.string "instructions"
    t.jsonb "tools", default: [], null: false

While OpenAI's schema for Assistant is:

model
name
description
instructions
tools

When I created these models — including Assistant — I was preparing for a world where we'd use them all for the Assistants API.

The thing that is tricky is the statefulness of the new API. They're doing it to be efficient, I assume. But we don't want to store all Threads and Messages on their server and have that be the source of truth. Instead, we want all that data stored in the our DB and that should be the source of truth. The tricky part is: when we do push the data from our system up to theirs?

My immediate reaction is, we double-create at the time of creation. So a user sends submits a message, we save it in our DB and we submit it as a new Message on the Thread for OpenAI. But one nice feature of our app is the ability to switch assistants in the middle of a conversation. To support this we probably need some kind of "catch up" mechanism so if you're far into a conversation and then you switch to GPT-4, it creates this Thread for the first time and submits all messages back-and-forth for the first time (even though these messages were generated by Claude 3).

Maybe we need a synced boolean (not sure what the best name is, pushed could work too) where we check a Conversation and Message to see if it's already been sent or not and "catch up" before sending a new one.

[Rmpanga]

My thought was that an Assistant created in the db was associated with an OpenAI assistant. For example on the left pane we have a ChatGPT 3.5 model which I assumed behind the scenes created an OpenAI assistant, stored its OpenAI assistant id, and used the assistant api to receive a response.

But when I was for the Run objects did not see any created, although they are required to receive a response from OpenAI assistants. That was my first clue that the Assistant model was behaving differently I was original expecting.

[Rmpanga]

@krschacht Maybe looking at an example may help.

# app/model/concerns/user/registerable.rb



  module User::Registerable
  extend ActiveSupport::Concern

  included do
    after_create :create_initial_assistants
  end

  private

  def create_initial_assistants
    assistants.create! name: "GPT-4",   model: "gpt-4-turbo-2024-04-09",  images: true
    assistants.create! name: "GPT-3.5", model: "gpt-3.5-turbo-0125",    images: false
    assistants.create! name: "Claude 3 Opus", model: "claude-3-opus-20240229", images: true
    assistants.create! name: "Claude 3 Sonnet", model: "claude-3-sonnet-20240229", images: true
  end
end

The create_initial_assistants is called when the user record is first created. However there is not linked to an external OpenAI assistant. The above assistants use ChatCompletionAPI

My confusion is how would we decide to use the Chat API and when to use the Assistant API methods.

[krschacht]

Well, once we migrate to the new Assistants v2 API endpoints I think we will no longer be using the Chat API endpoints at all. Anthropic still has chat endpoints which are stateless, like the old OpenAI chat endpoints. But Assistants API is now stateful.

The key question is still, when do we push data from our system over to the Assistants API? My preliminary thinking is that we do it at the time of new message creation, rather than at the time of assistant creation.

Here is what a rough implementation plan might look like:

1. We need to review the Assistant v2 docs closely and ensure our models are set up for it. I found this migration guide which may summarize all the differences. It's probably worth a PR just to enact these changes in the models & the corresponding associations between those models: https://platform.openai.com/docs/assistants/migration
2. I think we should add a boolean flag on Assistant, Messages, and Conversations which could be called synched. It defaults to false.
3. I think we add helper methods inside each of these models to perform the sync so then on an instance of any of these we can say: assistant.sync and this sets does all the appropriate API calls to ensure that the OpenAI assistant is updated. Likewise for conversation.sync and message.sync. I think we should treat the conversation one as the primary, meaning when we call conversation.sync it checks all the associated messages and ensures they are synced and it checks the associated assistant to ensure it's synced. Note: we're using ruby-openai gem so we get a nice ruby SDK for all these calls.
4. Inside GetNextAIMessageJob (I should rename that job, it's too wordy) we can add a helper method called something like sync_openai_assistant and somewhere near the top of that perform method we call it. Probably right after that puts line that outputs ### Wait for reply we call conversation.sync and it does it's thing to ensure anything that is not yet synced gets pushed over.
5. The one other place we'd need to wire things up is in settings/assistant. I think at the time you edit an assistant (e.g. change it's name or custom instructions) we add an after_save hook. I think if the assistant is already synced, then we do an update to that sync. But if the assistant is not yet synced then we wait until someone starts chatting with it and that spins up a GetNextAIMessageJob which will trigger everything. Note: we'd need a background job for updating the assistant in the case it's already synced and an update is being performed.

[Rmpanga]

Got it. This makes sense now. I agree with your initial assessment to push data during Message creation vs Assistant creation.
Treating the Conversation as the primary entity that syncs all the associated entities makes a lot of sense.

We need to review the Assistant v2 docs closely and ensure our models are set up for it. I found this migration guide which may summarize all the differences. It's probably worth a PR just to enact these changes in the models & the corresponding associations between those models: https://platform.openai.com/docs/assistants/migration

Im in the middle of doing this and plan on creating a PR with some of these changes.

[krschacht]

That sounds great, thanks @Rmpanga ! The item (1) on that list I just jotted down could easily be a stand-alone PR so it may be a good starting point.

I'm curious to ask you: what's your motivation for wanting to upgrade the project to the new Assistants API? Don't get me wrong, I'm happy to have you contributing in any way that you'd like so I'm not discouraging you. But personally, it hasn't been high on my list since OpenAI has been slow to add support for Assistants API and I don't think it will actually give the app any new capabilities beyond the Chat completions API. I mostly regard it as "upgrading plumbing to future-proof the app." If you haven't browsed this yet, here's the list of things that are top of mind for me: https://github.com/allyourbot/hostedgpt/milestone/6

Again, I'm happy to take any contributions! so don't take this as discouragement. I'm just curious if there's something in particular that's been motivating you on the Assistants API side.

[Rmpanga]

My motivation is to have the AI answer questions about a set a set of files. For example I have files for Company A's trainings docs & each document is training information about a specific employee type. I want to have the AI have the information needed to answer them.

The way I found it is by using this Assistant's File Search. Maybe there is another way to achieve this functionality without Assistants API? However the research I have found is that Assistants API is the native way to achieve this.

[krschacht]

@Rmpanga Ah, yes, you're absolutely right. I hadn't considered this yet but we really do need to migrate over to the Assistants API in order to use that built-in tool.

I believe we can use tools on the chat API but only ones that we define. Whereas the couple built in tools have to be done through Assistants.

Full speed ahead! :)

[krschacht]

@Rmpanga I wanted to raise one question about this potential migration over to the new Assistants API:

I was talking with someone who recently told me that they thought the Assistant’s built-in file search (built-in RAG) wasn’t very good. Have you tried it? Is it working well for you? I think we should evaluate this before doing the big migration, but maybe you already have.

Also, I have another way to potentially do file search. Right now I’m in the process of implementing Tools and I’ve decided just to re-implement all the OpenAI tools. One of the things the Assistant API would give us is access to their tools (web browsing, dalle, code interpreter). I think it may not be that difficult to re-implement the RAG implementation they have and there’s a decent chance we could make it better than theirs.

I’ve been talking with the creator of langchainrb and he has already implemented multiple RAG so collaborating with this project could be a way to get it quickly.

Also, while I’m implementing tools, I am having to reconcile a bunch of subtle differences between OpenAI API and Anthropic API. It’s not too bad, but as soon as we move from Chat API to the Assistants API then it creates a lot more differences between the two backends. Meanwhile, a bunch of people are requesting Groq and other backends so it would be nice to minimize the differences between all those.

All of this is making me think it’s premature to move this project over to the Assistants API. This is all pretty new thinking on my part. This space is moving quickly :)

[Rmpanga]

Yes I used it and I have think its performs well however I admit it is inconsistent. The more tasks you have the assistant perform in the instruction set the less consistent the results are. My assumption is that it's performance will improve over time.

I am very curious what you learned from the creator of langchainrb. If you have found any docs you like read feel free and send them.

To this point:

Also, while I’m implementing tools, I am having to reconcile a bunch of subtle differences between OpenAI API and Anthropic API. It’s not too bad, but as soon as we move from Chat API to the Assistants API then it creates a lot more differences between the two back-ends.

Yes I agree that OpenAI assistants implementation introduce major differences. It seems like the other AI service providers do not have a similar application yet. Maybe they will eventually and if they do will the data models mirror the Open AI ones such as Runs, Threads and Steps? Lots of open questions

Below gives you a rough idea of how I was thinking of implementing this

1. Create an external Id column for Assistant model
2. Create a type column in the Assistant model. Type would contain whether the assistant is of type Chat, Rag
3. Based on the Assistant's type, AI Service Provider call the appropriate API

This will likely change once I learn more about langchainrb

[Rmpanga]

I think I found the project: https://github.com/patterns-ai-core/langchainrb

[krschacht]

Yep, that's the project. And the creator hangs out regularly in the ruby-ai discord:
https://discord.gg/k4Uc224xVD

You can start a discussion in the hosted_gpt channel and tag him andreibondarev if you want him to weigh in on anything.