Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial setup for export benchmarking suite and some models #2596

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

tugsbayasgalan
Copy link
Contributor

@tugsbayasgalan tugsbayasgalan commented Mar 6, 2025

Export team would like to build a benchmarking suite for rising popular models. This diff is the first basic infra we will use to setup the models. Models added:

  1. Qwen2
  2. Kokoro
  3. MiniCPM (only the ConditionalChatTTS part)

Results:

{'environ': {'pytorch_git_version': '555bc8349d67a8f5ccd5325d1a7f46c3619b1769'},
 'metrics': {'errors': {'hf_Qwen2': 'Unsupported input type <class '
                                    "'transformers.cache_utils.DynamicCache'>",
                        'hf_minicpm': 'cannot mutate tensors with frozen '
                                      'storage',
                        'kokoro': '.numpy() is not supported for tensor '
                                  'subclasses.'},
             'success_rate': 0.0},
 'name': 'export_new_models'}

@xuzhao9
Copy link
Contributor

xuzhao9 commented Mar 6, 2025

The code LGTM, but it will break our CUDA CI which is 40GB A100. Is this expected?
If we can't shrink the model to single GPU, please add this model to test bypass list like https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/hf_T5_large/metadata.yaml#L11

@tugsbayasgalan tugsbayasgalan changed the title Initial setup for export benchmarking suite and Qwen model Initial setup for export benchmarking suite and some models Mar 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants