Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add LoRA support #820

Merged
merged 15 commits into from
Apr 17, 2023
Merged

Add LoRA support #820

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
f52101e
Add lora support
slaren Apr 6, 2023
ac3fbe4
Export lora A matrix pre-transposed
slaren Apr 8, 2023
7136ada
Add support for quantized models
slaren Apr 8, 2023
dc65707
Use the work buffer instead to fix MSVC build
slaren Apr 8, 2023
87c518b
Update exporter and support scaling
slaren Apr 8, 2023
c920f00
Add compatibility with #801
slaren Apr 10, 2023
c45868b
Support more layer types, fix memory and generation issues
slaren Apr 11, 2023
57627f0
Rebase to master
slaren Apr 13, 2023
c150e1b
Add support for using a different base model
slaren Apr 15, 2023
fc89916
Fix windows build
slaren Apr 15, 2023
14858ba
Show warning when using a quantized base model
slaren Apr 15, 2023
3df343b
ggml_cpy: use the work buffer instead of alloca when quantizing
slaren Apr 15, 2023
63da54e
Only attempt to use mmap for the lora base model if it is supported
slaren Apr 16, 2023
0a6d5ad
Reuse definitions from convert.py
slaren Apr 16, 2023
8d37db3
ggml_add: Add more checks
slaren Apr 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
ggml_cpy: use the work buffer instead of alloca when quantizing
  • Loading branch information
@slaren
slaren committed Apr 16, 2023
commit 3df343b4f0ed9699dec4593f15d4be54f095962d
12 changes: 9 additions & 3 deletions ggml.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5466,8 +5466,7 @@ static void ggml_compute_forward_dup_f16(
size_t id = 0;
uint8_t * dst_ptr = (uint8_t *) dst->data;
size_t dst_row_size = nb0 * (ne00 / GGML_BLCK_SIZE[dst->type]);
// todo: use work buffer
float * src0_f32 = (float *) alloca(ne00 * sizeof(float));
float * src0_f32 = (float *) params->wdata;

for (int i03 = 0; i03 < ne03; i03++) {
for (int i02 = 0; i02 < ne02; i02++) {
Expand Down Expand Up @@ -10227,9 +10226,17 @@ void ggml_graph_compute(struct ggml_context * ctx, struct ggml_cgraph * cgraph)
struct ggml_tensor * node = cgraph->nodes[i];

switch (node->op) {
case GGML_OP_CPY:
case GGML_OP_DUP:
{
node->n_tasks = 1;

size_t cur = 0;
if (node->type == GGML_TYPE_Q4_0 || node->type == GGML_TYPE_Q4_1) {
cur = GGML_TYPE_SIZE[GGML_TYPE_F32] * node->ne[0];
}

work_size = MAX(work_size, cur);
} break;
case GGML_OP_ADD:
{
Expand Down Expand Up @@ -10322,7 +10329,6 @@ void ggml_graph_compute(struct ggml_context * ctx, struct ggml_cgraph * cgraph)
{
node->n_tasks = n_threads;
} break;
case GGML_OP_CPY:
case GGML_OP_CONT:
case GGML_OP_RESHAPE:
case GGML_OP_VIEW:
Expand Down