LLM integration

[josevalim]

We want to integrate LLMs as part of Livebook itself. There are at least four distinct levels this can happen:

1. Code completion (may or may not need a LLM) (options: Codeium, Copilot, fine-tuned Repl.it model)

2. Chat-based (which also includes selecting code and asking to document it as well as explaining exceptions) (options: Codeium, Copilot, OpenAI, Claude)

3. Semantic-search over all installed packages in the runtime (may or may not need a LLM) (options: Codeium, Copilot, OpenAI, Claude)

4. Function completion (we can allow kino/smart-cells to register functions which we hook into prompts, similar to HF Agents) (options: OpenAI)

This is a meta-issue and we are currently doing proof of concepts on different areas. There is no clear decision yet. We will most likely allow users to "bring their own LLM" for most of these categories, especially 2-3-4 (either commercial or self-hosted).

[vitalis]

@josevalim I tested all the suggested solutions: Codeium, Copilot, OpenAI, Claude.
None of them will be useful with Elixir(btw Codeium is the worst)...
In my opinion it will be best to take small open source model and train it explicitly for Elixir..
We can host it on huggingface for free.
I think (and this assumption must be validated) that the huggingface rate limit for open source project will be sufficient for existing Livebook users...
Anyway.. I'm here to help with any direction you will choose...

[josevalim]

@vitalis for completeness, you tested for which particular categories above? How were your tests done and how did you assess quality?

And, of course, even if we decide to train our own model, there is no guarantee it will perform better. For code completion there are smaller models available but how would we proceed to train llama2 or similar and which data + resource do we need for such?

Thanks!

[vitalis]

Code completion: None of the solutions is helpful with Elixir, I tested Copilot before you spoke about this idea on twitch stream and tested all other solutions in the past days...
In my (subjective) opinion none of them will provide what YOU expect from code completion, they will do more harm than good..
Chat-based: ChatGPT(4) is the best solution (from the suggested solutions), but it's $20 a month.. I'm not sure many people will pay this in order to work with Livebook / learn Elixir
Semantic-search: again... none of the solutions is helpful with Elixir

[vitalis]

From my understanding it will be really difficult to optimize llama2 model or similar, llama2 is a huge model and it will require a lot of GPU resources and time and it will be very expensive...

[vitalis]

At this moment I don't have a solution.. but I have an idea I want to research next week and I'll update you

[michaelterryio]

If you did chat-based with GPT you wouldn't be using the subscription, you'd have users plug in a pay-as-you-go API key, right? Lots and lots of people are already using GPT for way more than Livebook, so I don't think a GPT token requirement would be the limiter in traction.

For the record, the chat-based workflow has been a tremendous accelerant to my Elixir app development, even without a built-in integration with Livebook.

[vitalis]

@michaelterryio
It's not just "plug in a pay-as-you-go API key"... it requires development.. I agree that chat-based workflow can be major improvement in app development
My (subjective) opinion is: you don't need Livebook/GPT integration in order to use GPT as chat-based workflow..
Again, in my (subjective) opinion if time will be invested in integration with any LLM it should be something game changing... and not something mainstream.. the trend today is adding GPT generator to every app.. and from what I see it's useless... (using GPT directly and not via integration is much better)

[vitalis]

@josevalim
Tested all other available models: CodeWhisperer, StarCoder, WizardCoder...and many more
I'm testing the models with a simple task of writing "bubble sort"
The worst model at this moment is CodeWhisperer, Elixir is not supported at all..
But here is an example of bubble sort by WizardCoder:

Didn't test replit.. to complex and no free plan..

I'll continue the research next week

[josevalim]

StarCoder, WizardCoder, and the replit one would have to be fine-tuned first. I am not expecting them to give good results out of the box.

[vitalis]

@josevalim
Is there existing dataset for Elixir?

[seivan]

I think 1 and 2 are related. You could create embeds of all the cells in a notebook and store them in some vector table (sqlite?).
When querying the chat, create embeds of the query and previous context and use that.
This would let you at least ask questions about your notebook. Which is useful, not sure how impactful it would be. Though not sure how useful if the models don't understand Elixir well enough :/