Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core][2/N] Model runner refactoring part 2. Combine prepare prefill / decode to a single API #4681

Merged
merged 49 commits into from
May 15, 2024
Merged
Changes from 1 commit
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
c9fcb26
first checkpoint done
rkooo567 May 8, 2024
de61dbb
refactoring subquery
rkooo567 May 8, 2024
159ea2f
.
rkooo567 May 8, 2024
5833cbb
ip
rkooo567 May 8, 2024
7614ce0
working
rkooo567 May 8, 2024
6744eff
Merge branch 'main' into model-runner-refactoring-coelsce
rkooo567 May 9, 2024
7de7f63
.
rkooo567 May 9, 2024
e8a4ea3
working with flash attn
rkooo567 May 9, 2024
64e8fd4
rocm and sdpa
rkooo567 May 9, 2024
ceec66d
working with flash infer
rkooo567 May 9, 2024
1851d59
add flash infer to pipeline
rkooo567 May 9, 2024
5cf1d3e
.
rkooo567 May 9, 2024
21a612a
working.
rkooo567 May 9, 2024
61dec37
fix spec decoding
rkooo567 May 9, 2024
6cad7bc
Fixed model runner test
rkooo567 May 9, 2024
e20a29e
fixed
rkooo567 May 9, 2024
94964ab
fix intel test
rkooo567 May 9, 2024
ff99251
Merge branch 'main' into model-runner-refactoring-coelsce
rkooo567 May 10, 2024
1c77e2d
.
rkooo567 May 10, 2024
f929edd
done
rkooo567 May 10, 2024
74683a1
.
rkooo567 May 10, 2024
546735a
fix circular reference.
rkooo567 May 10, 2024
89e5df2
Merge branch 'main' into model-runner-refactoring-coelsce
rkooo567 May 10, 2024
d7b2743
working
rkooo567 May 10, 2024
0ed4160
Merge branch 'circular-dep' into model-runner-refactoring-coelsce
rkooo567 May 10, 2024
f5af730
fixed spec decoding
rkooo567 May 10, 2024
7e39882
working
rkooo567 May 10, 2024
e02bc5d
Merge branch 'main' into model-runner-refactoring-coelsce
rkooo567 May 13, 2024
dd48c00
fix embedding meta
rkooo567 May 13, 2024
bba70f1
ip
rkooo567 May 13, 2024
f76b9ea
improve assert
rkooo567 May 13, 2024
cf1dbbb
Merge branch 'main' into model-runner-refactoring-coelsce
rkooo567 May 13, 2024
a281d97
done
rkooo567 May 13, 2024
f6afb05
lint
rkooo567 May 13, 2024
35f64ac
done
rkooo567 May 14, 2024
cc5df57
Merge branch 'main' into model-runner-refactoring-coelsce
rkooo567 May 14, 2024
ccf937c
.
rkooo567 May 14, 2024
0951715
works except spec decoding
rkooo567 May 14, 2024
2b2423d
.
rkooo567 May 14, 2024
f1c12f3
.,
rkooo567 May 14, 2024
8a01746
.
rkooo567 May 14, 2024
bf7959a
.
rkooo567 May 14, 2024
b42b43d
.
rkooo567 May 14, 2024
e9a973e
ip
rkooo567 May 14, 2024
4e733e2
Merge branch 'main' into model-runner-refactoring-coelsce
rkooo567 May 14, 2024
237e939
Merge branch 'main' into model-runner-refactoring-coelsce
rkooo567 May 14, 2024
35e98a0
Merge branch 'main' into model-runner-refactoring-coelsce
rkooo567 May 15, 2024
426d99a
.
rkooo567 May 15, 2024
1271556
done
rkooo567 May 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
improve assert
  • Loading branch information
rkooo567 committed May 13, 2024
commit f76b9eaea4db92ff56f830f8e2c28fc801a53dab
7 changes: 4 additions & 3 deletions vllm/worker/model_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,9 @@ def _prepare_model_input(
decode_only = False
prefill_seq_lens.append(seq_len)
else:
assert query_len == 1
assert query_len == 1, (
"seq_len: {}, context_len: {}, query_len: {}".format(
seq_len, context_len, query_len))
num_decode_tokens += query_len
decode_seq_lens.append(seq_len)

Expand Down Expand Up @@ -382,7 +384,6 @@ def _prepare_model_input(
slot_mapping.append(slot)

batch_size = len(input_tokens)
print(f"SANG-TODO {query_lens=}")
max_query_len = max(query_lens)
max_prefill_seq_len = max(prefill_seq_lens, default=0)
max_decode_seq_len = max(decode_seq_lens, default=0)
Expand Down Expand Up @@ -425,7 +426,7 @@ def _prepare_model_input(
dtype=torch.int,
device=self.device,
)
assert max_query_len > 0
assert max_query_len > 0, ("query_lens: {}".format(query_lens))

context_lens_tensor = torch.tensor(context_lens,
dtype=torch.int,
Expand Down
Loading