Skip to content

Conversation

@michaelfeil
Copy link
Contributor

@michaelfeil michaelfeil commented Apr 12, 2024

Adding some suggested fixes:

  • adding HF_TRANSFER via infinity requirements. It will ship soon as default, is always installed with [all]
  • lowering the batch_size: Has no significant performance improvments. 32 or 64 is suggested to not OOM for large batch sizes.
  • HF_HOME is respected to download models.

@alpayariyak

@michaelfeil
Copy link
Contributor Author

@alpayariyak Please review, thanks!

@michaelfeil
Copy link
Contributor Author

Hey @alpayariyak, coming by your SF meetup in-person on Friday night, maybe we can fix this together.

@michaelfeil
Copy link
Contributor Author

Close in favor of #4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant