Qualcomm AI Engine Direct - Unify Llama2&Llama3 and Small Accuracy Improvement.

[winskuo-quic]

Summary

• Initially, to expedite the development process, we created the llama3 script and runner instead of reusing llama2's. Multiple scripts and runners seem to confuse community users and appears unnecessary. This PR unifies llama2 and llama3 flow to make them easier to maintain and more readable.
• Fix hybrid mode accuracy issue in llama hybrid mode.
• Copied KV mode output quant_attrs and apply it to Prefill mode. We aim to align hybrid mode's result with single kv/prefill model. Below is the testing results using stories llama, which we can see the results generated by hybrid mode is relatively similar to single kv/prefill model after copying quant_attrs from KV mode to Prefill mode.

Test plan

• Updated the CI to use unified llama script.

• TODO: Add more UT for llama

• Ensure 1B model inference speed does not drop with 128prefill/512kv
  

• Before and After for Stories Llama 16a4w 32prefill/128kv, Prompt: "Once upon a time"

  Before(Current Mainline)

  prefill:
  Once upon a time, there was a little girl named Lily. She loved to play with her toys and her favorite toy was a big,

  kv:
  Once upon a time, there was a little girl named Lily. She loved to play with her toys and her favorite toy was a big, red ball. One day, Lily's mom asked her to help her with the laundry. Lily was happy to help and she put all the clothes in the washing machine. After the clothes were washed, Lily's mom asked her to help her hang them up to dry. Lily saw a big, black rake and asked her mom what it was. Her mom told her it was a rake and that it helps to

  hybrid:
  Once upon a time, there was a little girl named Lily. She loved to play with her toys and her friends. One day, Lily's mommy told her that they were going to visit her grandma who lived far away. Lily was excited to see her grandma, but she was also a little scared because she had never been on a long trip before. When they arrived at grandma's house, Lily saw a big, scary dog. She was afraid and didn't want to go near it. But her mommy told her that the dog was friendly and just

  After (With this PR)

  prefill:
  Once upon a time, there was a little girl named Lily. She loved to play with her toys and her favorite toy was a big,

  kv:
  Once upon a time, there was a little girl named Lily. She loved to play with her toys and her favorite toy was a big, red ball. One day, Lily's mom asked her to help her with the laundry. Lily was happy to help and she put all the clothes in the washing machine. After the clothes were washed, Lily's mom asked her to help her hang them up to dry. Lily saw a big, black rake and asked her mom what it was. Her mom told her it was a rake and that it helps to

  hybrid:
  Once upon a time, there was a little girl named Lily. She loved to play with her toys and her favorite toy was a big, red ball. One day, Lily's mom asked her to help her with the laundry. Lily was happy to help and she put all the clothes in the washing machine. After the clothes were washed, Lily's mom asked her to help her hang them up to dry. Lily saw a big, black sheet hanging on the line and she wanted to help. She grabbed the sheet and tried to hang it up

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7618

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit b383d09 with merge base 3f9324c ():

NEW FAILURE - The following job has failed:

• Check Labels / Check labels (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[winskuo-quic]

Hi @cccclai,
This PR is mainly to:

1. Merge llama2 and llama3 scripts.
2. Fix hybrid mode runner issue in mainline.
3. Apply KV mode output quant_attrs to prefill mode, which we can see the results for hybrid mode is relatively similar to single kv/prefill mode after this PR (Refer to summary section for the test results).

Please have a look at this PR and let me know if anything is unclear.
Thanks.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Thanks! Sorry miss this one

[cccclai]

Sorry need to revert this PR and re-land it. Because the internal failure can't be bypassed.

[winskuo-quic]

Sorry need to revert this PR and re-land it. Because the internal failure can't be bypassed.

Hi @cccclai,
No worries. I think I forgot to update the CI yml files to the new path:

executorch/.github/workflows/android-perf.yml

Line 276 in 9a0b51c

python -m examples.qualcomm.oss_scripts.llama3_2.llama -- \

Unsure if that is the root cause of internal failure.
Also, I am working on CI for static stories llama: #7884.
Will continue on this once it is re-land. Thanks.