Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

llama : per-layer KV cache #4309

Merged
merged 15 commits into from
Dec 7, 2023
Merged

llama : per-layer KV cache #4309

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
e9bcf66
per-layer KV
slaren Oct 3, 2023
55f2f2f
remove unnecessary copies
slaren Oct 3, 2023
f4f9367
less code duplication, offload k and v separately
slaren Oct 6, 2023
c294c78
Merge branch 'master' into per-layer-kv
ggerganov Dec 3, 2023
986b3da
llama : offload KV cache per-layer
ggerganov Dec 3, 2023
f3dbfb9
llama : offload K shift tensors
ggerganov Dec 3, 2023
3d3e6bd
llama : offload for rest of the model arches
ggerganov Dec 3, 2023
1fa91a4
llama : enable offload debug temporarily
ggerganov Dec 3, 2023
c44bc1e
llama : keep the KV related layers on the device
ggerganov Dec 3, 2023
c80b8a2
llama : remove mirrors, perform Device -> Host when partial offload
ggerganov Dec 3, 2023
e262947
common : add command-line arg to disable KV cache offloading
ggerganov Dec 3, 2023
66aaac9
llama : update session save/load
ggerganov Dec 3, 2023
1a1a1c3
llama : support quantum K cache (#4312)
ggerganov Dec 6, 2023
680a99e
Merge branch 'master' into gg/per-layer-kv
ggerganov Dec 7, 2023
fc5f334
readme : add API change notice
ggerganov Dec 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
llama : enable offload debug temporarily
  • Loading branch information
@ggerganov
ggerganov committed Dec 3, 2023
commit 1fa91a4833915214df68cf6b950657e7cc8e7b7e
2 changes: 1 addition & 1 deletion llama.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5508,7 +5508,7 @@ static struct ggml_cgraph * llama_build_graph(
//
// TODO: will be removed with backend v2

//#define LLAMA_OFFLOAD_DEBUG
#define LLAMA_OFFLOAD_DEBUG

if (!do_offload) {
return;
Expand Down