Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor dead code - Removing all flash_xxx.py files. #2166

Merged
merged 22 commits into from
Jul 5, 2024
Merged

Refactor dead code - Removing all flash_xxx.py files. #2166

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
b28946d
Refactor dead code.
Narsil Jul 2, 2024
69cb084
First working step.
Narsil Jul 2, 2024
ed34cf0
Remove a lot of duplicated code.
Narsil Jul 2, 2024
7d96b1a
More dead code.
Narsil Jul 2, 2024
ce913b8
More cleanup.
Narsil Jul 2, 2024
db9acc4
Fix Santacoder test.
Narsil Jul 2, 2024
298500a
Fixing the simple tests.
Narsil Jul 2, 2024
b2fb845
Fixing sharding.
Narsil Jul 2, 2024
43ef526
Fixes for VLM.
Narsil Jul 2, 2024
dbf9292
Fixing santacoder (num_kv_heads hardcoded).
Narsil Jul 2, 2024
24bbd7b
Removing more dead code.
Narsil Jul 2, 2024
e8ff76f
Fixing `config.n_head`.
Narsil Jul 2, 2024
2259d2f
Stopping earlier because of `<end_of_utterance>` in idefics2.
Narsil Jul 3, 2024
9cc58d1
Addresses comments.
Narsil Jul 3, 2024
fbf38c9
Removing the dead code.
Narsil Jul 3, 2024
f5ff9b5
Fuse back mistral into FlashCausalLM.
Narsil Jul 3, 2024
e2edf2b
Finish removal.
Narsil Jul 3, 2024
8ecee72
Fixing docs + causal_lm `batch_class`.
Narsil Jul 4, 2024
fc5bfa0
Fixing docs + causal.lm.
Narsil Jul 4, 2024
425f348
Add default to Gemma Causality.
Narsil Jul 4, 2024
4aa0642
Default value for gemma/gemma2.
Narsil Jul 4, 2024
25c9611
Wrong default.
Narsil Jul 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Removing more dead code.
  • Loading branch information
@Narsil
Narsil committed Jul 4, 2024
commit 24bbd7b822dbbb987d7270352e24ec49a5f65e78
9 changes: 5 additions & 4 deletions server/text_generation_server/models/causal_lm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -627,10 +627,11 @@ def fallback(
def batch_type(self) -> Type[CausalLMBatch]:
return CausalLMBatch

def decode(self, generated_ids: List[int]) -> str:
return self.tokenizer.decode(
generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
# This is not used anymore
# def decode(self, generated_ids: List[int]) -> str:
# return self.tokenizer.decode(
# generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
# )
OlivierDehaene marked this conversation as resolved.
Show resolved Hide resolved

def forward(
self, input_ids, attention_mask, position_ids, past_key_values: Optional = None
Expand Down
1 change: 1 addition & 0 deletions server/text_generation_server/models/flash_causal_lm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -827,6 +827,7 @@ def __init__(
aliases=None,
# Used for Santacoder override of config
num_kv_heads=None,
skip_special_tokens: bool = True,
):
self.process_group, rank, world_size = initialize_torch_distributed()
if torch.cuda.is_available():
Expand Down
9 changes: 5 additions & 4 deletions server/text_generation_server/models/seq2seq_lm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -668,10 +668,11 @@ def fallback(
def batch_type(self) -> Type[Seq2SeqLMBatch]:
return Seq2SeqLMBatch

def decode(self, decoder_ids: List[int]) -> str:
return self.tokenizer.decode(
decoder_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
# Not used anymore
# def decode(self, decoder_ids: List[int]) -> str:
# return self.tokenizer.decode(
# decoder_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
# )
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Not used anymore
# def decode(self, decoder_ids: List[int]) -> str:
# return self.tokenizer.decode(
# decoder_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
# )

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't thtat surprising that it's not used anymore ?

Shouldn't we have those flags be used somewhere for those models ? Do we have test that cover the raison d'être of this code ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this code was to allow models to specify the value of skip_special_tokens. For example for santacoder, you needed the fill in the middle special tokens to correctly display the outputs.


def forward(
self,
Expand Down