Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

speculative : add tree-based sampling example #3624

Merged
merged 18 commits into from
Oct 18, 2023
Merged

speculative : add tree-based sampling example #3624

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
5261aee
sampling : one sequence per sampling context
ggerganov Oct 12, 2023
4de5a2d
speculative : add tree-based sampling support
ggerganov Oct 14, 2023
32a67cb
speculative : reuse the n_parallel CLI param
ggerganov Oct 15, 2023
4a7f43f
speculative : refactor sampling
ggerganov Oct 15, 2023
7e48e21
examples : fix build after sampling refactoring
ggerganov Oct 15, 2023
0d96efa
batched : fix n_seq_id
ggerganov Oct 15, 2023
b5554b9
sampling : fix malloc
ggerganov Oct 15, 2023
b8acb6c
swift : fix build
ggerganov Oct 15, 2023
5b34bfa
swift : try to fix build
ggerganov Oct 15, 2023
0059491
prompts : add assistant.txt
ggerganov Oct 16, 2023
360a333
common : add llama_batch_add() and llama_batch_clear() helpers
ggerganov Oct 16, 2023
1c626e2
speculative : minor refactor
ggerganov Oct 16, 2023
373d782
minor : comments + rename
ggerganov Oct 16, 2023
f07cd35
speculative : fix off-by-one for n_drafted
ggerganov Oct 17, 2023
e6dd81f
speculative : fix the n_drafted fix + p constants
ggerganov Oct 17, 2023
010c52e
Merge branch 'master' into speculative-tree
ggerganov Oct 17, 2023
bd9451c
Merge branch 'master' into speculative-tree
ggerganov Oct 17, 2023
ad2727d
Merge branch 'master' into speculative-tree
ggerganov Oct 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
swift : fix build 
ggml-ci
  • Loading branch information
@ggerganov
ggerganov committed Oct 15, 2023
commit b8acb6c9b845ab25ca564da3c1cdf9016bd9a586
8 changes: 5 additions & 3 deletions examples/batched.swift/Sources/main.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ for id: llama_token in tokens {

print("\n")

var batch = llama_batch_init(max(Int32(tokens.count), Int32(n_parallel)), 0)
var batch = llama_batch_init(max(Int32(tokens.count), Int32(n_parallel)), 0, 1)
defer {
llama_batch_free(batch)
}
Expand All @@ -80,7 +80,8 @@ batch.n_tokens = Int32(tokens.count)
for (i, token) in tokens.enumerated() {
batch.token[i] = token
batch.pos[i] = Int32(i)
batch.seq_id[i] = 0
batch.n_seq_id[i] = 1
batch.seq_id[i][0] = 0
batch.logits[i] = 0
}

Expand Down Expand Up @@ -169,7 +170,8 @@ while n_cur <= n_len {
// push this new token for next evaluation
batch.token[Int(batch.n_tokens)] = new_token_id
batch.pos[Int(batch.n_tokens)] = n_cur
batch.seq_id[Int(batch.n_tokens)] = Int32(i)
batch.n_seq_id[Int(batch.n_tokens)] = 1
batch.seq_id[Int(batch.n_tokens)][0] = Int32(i)
batch.logits[Int(batch.n_tokens)] = 1

i_batch[i] = batch.n_tokens
Expand Down