LlaMa

[slavakurilyak]

It would be great to see LangChain integrate with LlaMa, a collection of foundation language models ranging from 7B to 65B
parameters.

LlaMa is a language model that was developed to improve upon existing models such as ChatGPT and GPT-3. It has several advantages over these models, such as improved accuracy, faster training times, and more robust handling of out-of-vocabulary words. LlaMa is also more efficient in terms of memory usage and computational resources. In terms of accuracy, LlaMa outperforms ChatGPT and GPT-3 on several natural language understanding tasks, including sentiment analysis, question answering, and text summarization. Additionally, LlaMa can be trained on larger datasets, enabling it to better capture the nuances of natural language. Overall, LlaMa is a more powerful and efficient language model than ChatGPT and GPT-3.

Here's the official repo by @facebookresearch. Here's the research abstract and PDF, respectively.

Note, this project is not to be confused with LlamaIndex (previously GPT Index) by @jerryjliu.

[hwchase17]

@conceptofmind i believe you said you were working on this?

[conceptofmind]

@conceptofmind i believe you said you were working on this?

Yes actively working on this with a group of peers. We have successfully deployed inference with the 65B models. Working on a LangChain wrapper now.

[conceptofmind]

Would have to think about how to handle the sizes of different models though. I could see this becoming an issue for the end user.......

[Electomanic]

There is some ongoing work to use GPTQ to compress the models to 3 or 4 bits in this repo. Also a discussion going on over at the oobabooga repo.

Not sure if this is going to work but might be something to keep an eye on. If it works out it could be possible to run the larger models on a single consumer grade GPU.

The original paper is available here on arxiv.

[conceptofmind]

4 bit may be plausible. 8 bit should be fine. The weights are already in fp16 from my understanding. I would have to evaluate this further.

[jooray]

Yes, the weights are fp16. You can convert and run 4-bit using https://github.com/ggerganov/llama.cpp. I think 30B with full precision might be at least on par to 65B 4-bit in case of results. Llama.cpp runs on CPU, including Apple Silicon, which might be a good choice for developers with recent Macbooks, they could develop and run experiments locally with langchain without a need of GPUs.

[fblissjr]

There is some ongoing work to use GPTQ to compress the models to 3 or 4 bits in this repo. Also a discussion going on over at the oobabooga repo.

Not sure if this is going to work but might be something to keep an eye on. If it works out it could be possible to run the larger models on a single consumer grade GPU.

The original paper is available here on arxiv.

Confirmed working on a single consumer grade 4090 here with 13B. Waiting on the 30B 4 bit weights - failed at trying to run them at fp16. :)

[conceptofmind]

I am aware of all these alternatives. We are waiting to hear back from Huggingface before the decision is made. Once we have a concrete answer from them we will proceed from there.

I have some concerns about Llama.cpp since the author seems to have noted he has no interest in maintaining it. And there are other things to factor in when adding dependencies that can not be easily installed. It needs to be a relatively effortless setup for the best user experience.

[gururise]

Using GPTQ 4-bit quantized 30B model, outputs are (as far as I can tell) very good. Hope to see GPTQ 4-bit support in LangChain. The GPTQ quantization appears to be better than the 4-bit RTN quantization (currently) used in Llama.cpp

4-bit 30B model confirmed working on an OLD Tesla P40 GPU (24GB).

[DamascusGit]

Any info on running 7B model with Langchain?

[niansa]

Yes, the weights are fp16. You can convert and run 4-bit using https://github.com/ggerganov/llama.cpp. I think 30B with full precision might be at least on par to 65B 4-bit in case of results. Llama.cpp runs on CPU, including Apple Silicon, which might be a good choice for developers with recent Macbooks, they could develop and run experiments locally with langchain without a need of GPUs.

It'd be really neat if that's going to be an option 😄
Sure it's slow but hey you can run it on a literal laptop.

[conceptofmind]

Llama has been added to Huggingface: huggingface/transformers#21955

The only reason to add a specific wrapper would be to include the perf improvements from cpp or gptq