Skip to content

Avoid some pipeline tasks to use use_cache=True #24893

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 19, 2023
Merged

Avoid some pipeline tasks to use use_cache=True #24893

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9c6d979
fix
ydshieh Jul 18, 2023
0d8d808
fix
ydshieh Jul 18, 2023
3d141b3
fix
ydshieh Jul 19, 2023
753727b
fix
ydshieh Jul 19, 2023
757c493
fix
ydshieh Jul 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions src/transformers/pipelines/question_answering.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import inspect
import types
import warnings
from collections.abc import Iterable
Expand Down Expand Up @@ -510,6 +511,10 @@ def preprocess(self, example, padding="do_not_pad", doc_stride=None, max_questio
def _forward(self, inputs):
example = inputs["example"]
model_inputs = {k: inputs[k] for k in self.tokenizer.model_input_names}
# `XXXForSequenceClassification` models should not use `use_cache=True` even if it's supported
model_forward = self.model.forward if self.framework == "pt" else self.model.call
if "use_cache" in inspect.signature(model_forward).parameters.keys():
model_inputs["use_cache"] = False
output = self.model(**model_inputs)
if isinstance(output, dict):
return {"start": output["start_logits"], "end": output["end_logits"], "example": example, **inputs}
Expand Down
5 changes: 5 additions & 0 deletions src/transformers/pipelines/text_classification.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import inspect
import warnings
from typing import Dict

Expand Down Expand Up @@ -179,6 +180,10 @@ def preprocess(self, inputs, **tokenizer_kwargs) -> Dict[str, GenericTensor]:
return self.tokenizer(inputs, return_tensors=return_tensors, **tokenizer_kwargs)

def _forward(self, model_inputs):
# `XXXForSequenceClassification` models should not use `use_cache=True` even if it's supported
model_forward = self.model.forward if self.framework == "pt" else self.model.call
if "use_cache" in inspect.signature(model_forward).parameters.keys():
model_inputs["use_cache"] = False
return self.model(**model_inputs)

def postprocess(self, model_outputs, function_to_apply=None, top_k=1, _legacy=True):
Expand Down
5 changes: 5 additions & 0 deletions src/transformers/pipelines/zero_shot_classification.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import inspect
from typing import List, Union

import numpy as np
Expand Down Expand Up @@ -221,6 +222,10 @@ def _forward(self, inputs):
candidate_label = inputs["candidate_label"]
sequence = inputs["sequence"]
model_inputs = {k: inputs[k] for k in self.tokenizer.model_input_names}
# `XXXForSequenceClassification` models should not use `use_cache=True` even if it's supported
Copy link
Collaborator

@sgugger sgugger Jul 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use self.framework here instead of checking for attributes? Something like

model_forward = self.model.forward if self.framework == "pt" ekse self.model.call
if "use_cache" in set(inspect.signature(model_forward).parameters().keys():
    model_inputs["use_cache"] = False

Copy link
Collaborator Author

@ydshieh ydshieh Jul 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure!

model_forward = self.model.forward if self.framework == "pt" else self.model.call
if "use_cache" in inspect.signature(model_forward).parameters.keys():
model_inputs["use_cache"] = False
Copy link
Collaborator Author

@ydshieh ydshieh Jul 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So far only in src/transformers/pipelines/zero_shot_classification.py, but can extend to some other pipeline tasks.

Copy link
Collaborator Author

@ydshieh ydshieh Jul 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, there is no backward compatibility issue here:

  • model_inputs in this task (so far) never receive use_cache
  • this pipeline's output doesn't contain past_key_values (despite model gives it to pipeline)

Copy link
Contributor

@Narsil Narsil Jul 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never a fan of inspecting the signature, but 16% perf improvement is worth it !

outputs = self.model(**model_inputs)

model_outputs = {
Expand Down