Qualcomm AI Engine Direct - Streaming Mimi Enablement

[winskuo-quic]

Summary

Stateless(Static) Streaming Mimi Decoder Enablement.

Scripts to run

1. Run examples/models/moshi/mimi/install_requirements.sh, manually change pip install moshi==0.2.3 for this since we are currently building static mimi decoder on top of 0.2.3 version

2. Run the example script

• CPU Encoder with QNN Decoder
  python examples/qualcomm/oss_scripts/moshi/mimi.py -b build-android -s $DEVICE -m SM8650 --use_cpu_encoder

• QNN Encoder/Decoder
  python examples/qualcomm/oss_scripts/moshi/mimi.py -b build-android -s $DEVICE -m SM8650

Stats for SM8650

• SQNR Scores Improvement after Streaming Mode Enablement
  • CPU Encoder with QNN Decoder: 4.7 -> 7.8
  • QNN Encoder/Decoder: 0.46 -> 0.88
• Encoder: 3.0ms/chunk
• Decoder: 5.9ms/chunk

Follow ups

• Add README
• UT and Community CI enablement
• Inference Speed Optimization
• Accuracy Improvement

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10570

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit f373833 with merge base 95a1db5 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[winskuo-quic]

Hi @cccclai, @iseeyuan,
This PR is to enable Stateless (Static) Mimi.
Execution commands and some of the stats are shared in the Summary section.
Also, it would be appreciated if you could confirm the license for this PR is written correctly.
As discussed previously, it requires source code modification in order to enable the model.
The following 4 files listed below involves source code modification.

• examples/qualcomm/oss_scripts/moshi/model/static_conv.py
• examples/qualcomm/oss_scripts/moshi/model/static_convtr.py
• examples/qualcomm/oss_scripts/moshi/model/static_mimi.py
• examples/qualcomm/oss_scripts/moshi/model/static_seanet_decoder.py

Please let me know if you believe something is not done correctly.
Thanks

[cccclai]

Hey there seems to be lint error but those are not related to your changes? Mind rebasing and they should be gone.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Thank you!