p1 : LLM-based code completion engine at the edge

[ggerganov]

Intro

This project is an attempt to implement a local code completion engine utilizing large language models (LLM).
Think of it as an open-source alternative to Github Copliot that runs on your device.

We will explore how feasible it is to apply existing open-source LLMs for this task. We will utilize the existing ggml technology to achieve efficient on-device inference. The implementation will be community driven, similar to the llama.cpp project. Everyone is welcome to join and collaborate in the process.

The name of the project p1 stands for the first official project developed and supported by ggml.ai

Primary objectives

• The code generation inference works completely offline, on the local device
• Multiple programming languages are supported, including C, C++, Python, Javascript
• Integration with text editors: NeoVim, VSCode, others
• Code-completion based on local context (i.e. contents in the current file, near the current cursor position)
• Support for CPU-only, GPU-only and mixed CPU/GPU inference
• Support for Apple Silicon devices
• Support for CUDA devices (i.e. NVIDIA video cards)

Secondary objectives

• Support for on-premise server deployment
• Code-completion based on non-local context (i.e. extra files, user-selected context)
• LLM-assisted coding ("rewrite code section in certain way", "explain selected code", etc)
• Language server guided sampling
• Voice control support via whisper.cpp

Implementation strategy

The llama.cpp project has already demonstrated almost all necessary ingredients for creating an efficient code completion engine. The major effort in p1 will be to find a way to combine all these ingredients into a single application that is easy to extend and integrate with other applications.

The main ingredients are:

• Fast CPU-only, GPU-only and mixed CPU/GPU inference of LLMs with Metal (Mac OS) and CUDA (Linux and Windows)
• Efficient quantization of the models in terms of storage, quality and performance
• Speculative sampling for extra performance (not yet explicitly demonstrated, but should be clear how to achieve)
• Context management and reuse
• REST API

Based on the inference performance that we have achieved so far, we should aim at utilizing models in the range of 3B - 20B parameters in order to achieve generation speeds of about ~50 tok/s or more on modern hardware. Candidate models are:

• StarCoder
• WizardCoder
• replit-code
• ggml-code (model trained by ggml.ai for source code, TBD)
• others

For speculative sampling, we will try to utilize small fine-tuned models for specific programming languages. These "draft" models can be in the order of a few tens of million of parameters and their main purpose will be to just improve the inference speed of the main model. The inference of the draft models will be very fast and can easily be done in parallel on the CPU.

The main programming language of p1 will be C++ as usual.

Collaboration

I'm hoping to utilize the llama.cpp momentum and receive help from existing contributors. Ideally, I wish that we manage to form a small dev team that will drive the main development of p1 and will be able to make decisions autonomously. I think the above description sets the main vision of the project, but there are still many design decisions left to be made and I don't want to be the sole person making those decisions. I wish this to be a collaborative effort. Hopefully, p1 will become a place where capable people find ways to demonstrate their skills by creating a useful tool for everyone to use. I'm sure that in the process, we will find a lot of interesting problems to solve and potentially set the stage for new LLM applications in the future.

For that purpose, I will be inviting collaborators to the ggml-org organization based on their expressed interest and past contributions. The main goal will be to eventually delegate the entire p1 project to the newly formed dev team. In case we don't succeed in forming the dev team the way I imagine it, then I'll still try to keep p1 alive. It will just probably take a bit longer to complete. And of course, there is also the possibility that this effort turns out to be not as easy as I imagine it to be, in which case p1 will simply die :)

If you are new to ggml and llama.cpp, but what you read here sounds interesting, make sure to first get familiar with llama.cpp as it is the closest thing to what I imagine p1 will become.

For now, lets brainstorm for a bit in the discussion below and see how things go :)

Related work

• https://github.com/ravenscroftj/turbopilot
• https://github.com/fauxpilot/fauxpilot
• https://github.com/TabbyML/tabby
• https://github.com/abetlen/ggml-python/tree/main/examples/replit
• https://github.com/ggerganov/whisper.cpp/tree/master/examples/whisper.nvim#todo

[abetlen]

@ggerganov awesome idea, looking forward to seeing this!

Here are just some random thoughts from a similar project I was working on using the replit-code model:

• Initially targetting a drop-in replacement for github-copilot is going to be the highest leverage approach since you don't need to worry about editor plugins. The cost to this approach is some incompatibility with the tokenizers and you won't be able to do e.g. speculative sampling. To accomplish this you need an OpenAI compatible webserver and then you just need to change a few variables in your editor copilot plugin settings. Here's the README from my WIP replit server it has documentation for the vscode and vim plugins.
• Prompt processing time is going to dominate performance, the vscode copilot plugin sends about 1.5k prompt tokens per request so CUDA / OpenCL / Metal acceleration is a must even for smaller models.
• HTTP request interruption is tricky and you'll need to handle it in a good way or else the server will get backed up. The copilot plugin will cancel http requests to the server if the user moves the cursor and will send a subsequent request at the new cursor position. The python server I'm using can detect when the request has been cancelled but it's not possible to safely interrupt a larger ggml_graph_compute call so you'll probably need to reduce batch size in some cases to periodically check if processing should be cancelled.

[ggerganov]

Great feedback! I saw your efforts recently in this area and was really intrigued.

• I think speculative sampling is a must in order to achieve reasonable performance. But I don't see the reason why utilizing the existing copilot API prevents us from using it. I might be misunderstanding the way speculative sampling works, but I think it is completely part of the inference backend and it is irrelevant of the API.

• The next major advances in ggml will be specifically in prompt processing performance via implementing a custom quantized GEMM that replaces the existing BLAS calls. The BLAS APIs currently support either F16 and F32 precision which prevents us from taking full advantage of quantized data. The idea is to implement quantized GEMM and even if it does not utilize all the tricks of an optimal BLAS implementation it should still be faster because it will work with less data.

  Also another "trick" I was thinking about is that we should definitely utilize prompt reusing. For example, lets say we have processed a prompt of 2k tokens for the current cursor position, and the user moves a row up. Instead of recomputing an entirely new prompt, we can simply cut the last row from the already processed prompt and reuse it. This is of course a simplified strategy, but just to illustrate the idea of reducing the prompt computation

• Good point. I think we can easily extend ggml backend to support interruption. But this is something I haven't considered until now

[clxyder]

I would like to suggest the following the project: https://github.com/morph-labs/rift

[ghost]

I think the question we want to ask ourselves before doing all this is whether or not these models produce good code that is actually helpful. If the programmer has to spend more time fixing the generated code than writing it themselves that sort of defeats the purpose of the completion engine.

As a start I think it would be easiest to simply copy-and-paste code snippets from the editor into those models (basically the local context so to speak) just to see how well they can infer our intent and generate the next lines. This is simple and can be done with existing interfaces (including llama.cpp) to these LLMs. If everyone is satisfied with the performance then we can go forwards with an editor plugin and work on usability, performance, and so forth.

[ggerganov]

Yup, we should totally do that. But I'm not really worried about the models not being good enough. Even if the existing models are not great, they will be at some point soon. The assumption is that since Github Copilot works and it is already more than an year old technology in production, the open-source alternatives should be a within reach. Also, assuming there really isn't some super secret unknown sauce - it's just an efficient inference of quantized models.

[cwegener]

The assumption is that since Github Copilot works

That is the elephant in the room though. What does Github Copilot currently produce? I actually don't know, as I haven't used Github Copilot myself yet. I have only watched other people use it. And have only used Codeium.com myself.

As always with assumptions, I think it would be good to spell them out. I.e. "What is it in Github Copilot that works and that people like?"

I think there could be two distinct use cases for a code completion engine:

• Case A: Generate a piece of code that is sufficiently self contained (e.g. a function)
• Case B: Generate the next line of code, such that I don't have to manually type it all myself (which does sound very much what I would personally understand as 'code completion')

I think that the expectations on the model performance is different for each of the above two cases.

For case A, the generated code could be in a form that I as the human programmer cannot easily read and comprehend. And if that generated code passes a unit test, the human programmer probably doesn't have a reason to understand that piece of code.

For case B, the generated code should definitely be in a form that I as a human programmer can read and grok. Because I will be the one who will maintain that code going forward.

So, I think that some clarification on the distinction between these two general cases could be helpful as a goal setting statement in the mission statement?

[ggerganov]

Personally, I find myself using Github Copilot mostly as a "current line completion" tool (i.e. Case B) and this is already quite useful. I can't comment on how it is being used in general by other people, but I don't think it really matters. p1 will provide the framework for utilizing local LLMs anyway we like. The generation quality we have little control of, unless we start training our own models, which is not a priority atm.

In terms of mission statement, I think we can state that p1 will aim to support all common use cases of LLMs - either Case A, Case B or some other use case that makes sense.

[ghost]

Also, assuming there really isn't some super secret unknown sauce - it's just an efficient inference of quantized models.

I think the "secret sauce" here is the fact that Copilot was trained on the entire corpus of Github code, as well as the fact that the model is specifically designed for programming rather than conversation. For instance there has been talk about how the original Llama tokenizer doesn't work well for the syntax for programming languages.

On the inference speed side, those improvements will be for ggml/llama.cpp as a whole rather than just for p1 and that's definitely something that I endorse 😃

[cwegener]

Personally, I find myself using Github Copilot mostly as a "current line completion" tool

I agree. This is how I have seen most people use Copilot and it is also how I've used other AI completion tools myself. I also think that this mode is the most useful.
But others might disagree and might say that letting a LLM generate a complete function + tests is more useful to them.

The generation quality we have little control of, unless we start training our own models, which is not a priority atm.

I do agree. Your earlier comment on future models improving continuously is important to have spelled out.

The generation quality we have little control of, unless we start training our own models, which is not a priority atm.

Awesome. That sounds great.

In summary, I think that the original mission statement is really good. And spelling out these clarifying points - support of use cases. independence from model training and independence from associated future model improvements - will help painting an even clearer picture in the minds of interested parties (I'm a visual person 😉 ).

[GeorvityLabs]

@ggerganov you should have a look at WizardCoder 15B

[hfassold]

Exactly ! It is much better. See table 1 of WizardCoder paper at https://arxiv.org/pdf/2306.08568.pdf

[howard0su]

We can use LORA models for the different languages. If LORA approach can even work well, shall we even consider training the whole code base to feed model much more context? (this is not even possible for any existing copilot product.)

[ggerganov]

Yes! Good points. The "train a local model on your codebase" idea is very interesting.

[SlyEcho]

Not just interesting, I'm sure there are some institutions that can't use any assistant that can cause code to leak in or out.

[kviolka]

Do you know of any resources how to do such a thing? For instance, would it be possible to train a system on the Drupal codebase to help improve the project?

[ar-jan]

Would you consider adding a more general "Coding assistance" to the objectives? Personally I find this more useful than Copilot-like completions because you can be more explicit about your intent.

I've played with Sourcegraph Cody, which has various standard recipes as well as a chat prompt interface. Some of these recipes could fall under "LLM-assisted code transformation", but you can also do e.g. "Explain selected code" which is helpful for exploring new codebases. I find the chat interface useful to ask things like "The function my_function() results in this error , how to fix this?"

Cody Recipes

[SlyEcho]

I was investigating this when we were doing the server refactoring.

There are existing projects like fauxpilot that do this by imitating the OpenAI API. The Copilot plugin can be directed to it by some config changes.

There are some things that are missing or not ideal in llama.cpp:

1. Can't generate multiple outputs in parallel. Not sure how hard it would be to achieve. As a workaround, we can generate multiple outputs serially (OpenAI only allows 5, anyway).
2. Need to get logits for all tokens, I think support is already there, but the data needs to be exposed.

[ggerganov]

1. I hope we can solve that soon. It has been mostly demonstrated in the baby-llama example in llama.cpp
2. I think we already support this via logits_all. If not - it is trivial to add

[SlyEcho]

Alright, I will try to hack some demo up.

[bullno1]

One thing I think is under explored is language server guided sampling.
A lot of languages already have language servers.

Language servers can be queried for a list of legal symbols at different scopes/context.
We can restrict sampling to target only those symbols so the code is at least compilable.

Of course, in contexts where new symbols can be introduced, it's a bit problematic.
I have not read the LSP properly yet to know how to query it for "completion trigger" like . or ->.

[Alignment-Lab-AI]

hello i have actually been working on this front for a while, i have a few implementations of an autocoding system on my github, (just simple scripts in various states of working)

https://lablab.ai/event/anthropic-ai-hackathon/entropic-alignment/py-interpreter

this is an implementation of a similar solution using anthropics claude from my submission for the hackathon which i completed to get access to 100k context

additionally ongoing and continuous development of a third system has been ongoing (i didnt do too much of the leg work on this one, but he is one of the two others involved with the hackathon project above)

ive also got a community on my server which is composed almost entirely of AI researchers and developers, mostly who focus on open sourced projects. i very likely will be able to bring a few people interested who are highly skilled developers on board as an initial seed for a dev team

https://discord.gg/n9hXaBPWxx heres the invite if anyones interested in joining or contributing to super cool projects :)

i throw gpus at people with cool projects when i can afford it :D

[domluna]

@Alignment-Lab-AI hey this looks really cool! I've been working on something similar at https://www.datagptd.com/

[mirek190]

The best model for coding is a WizardCoder-15B right now.
For coding has level of CharGPT 3.5. Is better than anything right now offline.

[czkoko]

microsoft's phi-1 maybe will be open source in the near future.
If phi-1 can really create such a good result with only 1.3B parameters, it should be the best choice. But it seems to only support python.
https://arxiv.org/pdf/2306.11644.pdf

[hfassold]

Sounds good, but WizardCoder is better.

[eryk-mazus]

I think it's worth checking out the internals of the Copilot: link
They utilize the simple logistic regression to predict whether it is worth to make a suggestion and jaccard index to extract similar snippets from the repository.

[SlyEcho]

That's very neat!

[walking-octopus]

I think it's worth investigating beam search for LLaMA and StarCoder. For instruction-based one-shot code generation, or most kinds of problem solving with smaller models in general, disabling it results in a cascade of snowballed hallucinations.

[Alignment-Lab-AI]

even with fine tuning?
i have a personal project im actively working on that involves a few elements of this, can i pick your brain a bit?

[eryk-mazus]

Beam search is for generation and it slows down the inference

[N3mes1s]

Seems there is a new player in town, 3B

A new fine tuned code model called instruct-glaive from
@csahil28
achieves an open record of 63.5%+ on HumanEval benchmark surpassing WizardCoder. The model is just 3B parameters, 5x smaller than WizardCoder that held 57%+

https://huggingface.co/sahil2801/replit-code-instruct-glaive

https://github.com/abacaj/code-eval for eval

[wsxiaoys]

Hi @ggerganov,

I'm the maintainer of https://github.com/TabbyML/tabby, a project aiming to create an open-source alternative to GitHub Copilot. We initially built our inference engine using https://github.com/OpenNMT/CTranslate2, as it provided the best support for CUDA + INT8 inference at that time. However, we are actively exploring options to make it suitable for on-device usage through INT8 inference and Core ML. You can find the relevant links here:

1. https://tabbyml.github.io/tabby/docs/self-hosting/apple
2. support coreml [TAB-13] TabbyML/tabby#185

Tabby itself focuses more on the engineering aspect of Copilot, such as implementing a caching strategy in the IDE extension and parsing code into Tree Sitter tags to improve suggestions. We remain neutral regarding the choice of inference engine and model. Therefore, I can envision a clear path for collaboration. Integrating ggml into Tabby's Rust inference core should be straightforward, similar to what we did with CTranslate2.

[ggerganov]

Hi @wsxiaoys,

TabbyML looks awesome - it would be great if we find ways to integrate ggml in the near future. CTranslate2 is a really great framework and from my observations it is the better choice in many aspects. I hope at some point ggml will offer support for low-bit quantization that would be easy to integrate in 3rd party projects such as TabbyML. The API is not there yet, but we are working on it

[heaven00]

and I would love to help, because I am also trying to achieve the same goal

Though mostly just reading papers and building an understanding at the moment. Would love to know if this is going forward.
Thank you both for your contributions!

[Freed-Wu]

Related work

https://github.com/ravenscroftj/turbopilot
https://github.com/fauxpilot/fauxpilot
https://github.com/TabbyML/tabby

Hey, what is the difference between p1 with the above projects? Will p1 replace them to provide faster completion speed by a new technology? Or just use same technology but have a better engineering code to provide better user experience? What is the largest novelty of p1?

[ggerganov]

It's mostly something I want to implement from scratch. It's hard to say what would be the advantages. My guess is it will be mostly a learning experience with some potential for higher performance than the alternatives + opportunities to do some fun open-source collaboration.

[tbressers]

Hi @ggerganov et al,

What could be a huge 'selling point' for this project, is to enable building / fine-tuning LLM models for use with own, private code. This is something which is not (and will never be!) provided by closed-source LLM's like OpenAI's GPT4.

Maybe it would be good to focus this program on such use?

regards - Thijs

[oaguy1]

I’ve been looking into doing a similar project and would love to assist in any way I can. My starting point was going to be a new model fine-tuned off Llama 2 using the dataset from Code Clippy. I am new to fine-tuning LLMs, but have been doing some reading and think a PEFT QLoRA training combined with quantizing the model using ggml would get us there. I can train some PoCs locally before committing to serious cloud GPU time (I’ve been playing w/ Lambda Labs cloud, which seems reasonably priced at an A10 for $0.60 an hour).

As for local inferencing, I’ve been thinking WASM might be a good fit here. Using WASM one could embed llama.cpp into VS Code. A sufficiently quantized model should fit within the 4GB RAM limit imposed by WASM (referenced in #97). Such an implementation is sufficiently performant, then it would mean the plug-in could work with minimal user config, at least to start. I believe offering an option to running llama.cpp via an external binary should also he supported, but something lower touch to start could be advantageous for users just trying it out.

[ggerganov]

Interesting idea about the WASM - something to consider definitely.

I guess the actual work on p1 will start when we add full support of non-LLaMA models to llama.cpp. Likely going to happen in the second half of August.

I have no expertise when it comes to training or fine tuning, so any help with this would be appreciated

[monatis]

I can give a hand in training/finetuning here.

Although adding support for other architectures in llama.cpp may help us use high-quality of-the-shelf models out there, I think after LLaMA-2 it's quite plausible to finetune it for our exact use case, e.g., completion-based, instruction-based, or chat-based. We can also collect data from users who agree to share their usage information with us and iteratively re-finetune the model for better quality.

[ichDaheim]

@ggerganov
FIrst of all:
This is so great work all of you are doing with llama.cpp. I'am really excited what possibilities your work offers and haevliy impressed from your efforts and knowledge all of you @lama.cpp are showhing.

Concernig the P1 project:
I'am not very much into the programming part here (i'am more the java guy), but my impression is that you have all the basic parts together as far as i understand from the examples.
You have:

• server.cpp with a OpenAI compliant API
• LoRa finetunig
• embeddings
  The "only" thing that seems to be missing is a compatbile IDE Client - and "putting it all together". Which is quite a lot of work i'am sure. And i reamphisize here that i do NOT understand the code here but that seems to be a "makeable" effort to me.

Having a local copilot clone at hand would make my live so much more comfortable and exciting.
Making a point at the end to all this talking: One vote up for this project and please make it happen.

[ggerganov]

Most of the parts are indeed available, but there are still many pieces that need to be improved and supported in the core ggml and llama.cpp software and these require further time and resources. At the moment, they are more important to complete, because that would enable a variety of projects that work on top of them to grow. Putting time in p1 would slow things considerably, so for now this is still on hold. Plus, there are already some very good alternatives for local pilot available and some of them already use llama.cpp - it seems easier to support them at this point.

[ggerganov]

It's been a while, but here is some progress on this project for anyone still interested: ggerganov/llama.cpp#9787