Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#34 causal logprobs #35

Merged
merged 8 commits into from
Feb 21, 2024
Merged

#34 causal logprobs #35

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
8c12b10
Properly testing batched scoring
ClementRomac Feb 21, 2024
042765e
Applying log_softmax in generate
ClementRomac Feb 21, 2024
d222a70
Left padding for causal LMs
ClementRomac Feb 21, 2024
76e31cc
Properly testing batched scoring
ClementRomac Feb 21, 2024
87a5e5c
Applying log_softmax in generate
ClementRomac Feb 21, 2024
fee5531
Left padding for causal LMs
ClementRomac Feb 21, 2024
f9d2648
Merge remote-tracking branch 'origin/#34-causal-logprobs' into #34-ca…
ClementRomac Feb 21, 2024
d14f59d
Adding doctring in generate
ClementRomac Feb 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Applying log_softmax in generate
  • Loading branch information
@ClementRomac
ClementRomac committed Feb 21, 2024
commit 042765e699ac0151e002918f219bd16c12e67481
16 changes: 10 additions & 6 deletions lamorel/src/lamorel/server/llms/hf_llm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import torch
from math import ceil
from torch.nn.functional import log_softmax

import logging
lamorel_logger = logging.getLogger('lamorel_logger')
Expand Down Expand Up @@ -214,20 +214,24 @@ def generate(self, contexts, return_logprobs=False, **kwargs):

_generated_texts = self._LLM_tokenizer.batch_decode(generated_sequences, skip_special_tokens=True)
if return_logprobs:
logp = torch.stack(results.scores, dim=1)
texts_logp = torch.gather(logp, 2, generated_sequences[:, :, None]).squeeze(-1)
_scores = texts_logp.sum(-1)
logp = log_softmax(torch.stack(results.scores, dim=1), dim=-1)
scores = torch.gather(logp, 2, generated_sequences[:, :, None]).squeeze(-1)
aggregated_scores = scores.sum(-1)
else:
probabilities = torch.stack(results.scores, dim=1).softmax(-1)
texts_probabilities = torch.gather(probabilities, 2, generated_sequences[:, :, None]).squeeze(-1)
_scores = texts_probabilities.prod(-1)
scores = torch.gather(probabilities, 2, generated_sequences[:, :, None]).squeeze(-1)
aggregated_scores = scores.prod(-1)

generations.append([
{
"text": _text,
"score": _score.detach().cpu().numpy()
"text_probability" if not return_logprobs else "text_logprob": _agg_score.detach().cpu().numpy(),
"tokens_probability" if not return_logprobs else "tokens_logprob": _scores.detach().cpu().numpy()
}
for _text, _score in zip(_generated_texts, _scores)
for _text, _tokens, _scores, _agg_score in
zip(_generated_texts, generated_sequences, scores, aggregated_scores)
])

if "sequences_scores" in results: # Useful for Beam search
Expand Down