Android app crash with stories110m model

[iceychris]

🐛 Describe the bug

Hey!

I tried to run torchchat on my low-end Android device and got this app crash in logcat with the stories110m model:

Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 17142 (Thread-3), pid 17085 (torch.torchchat)
Cmdline: org.pytorch.torchchat
pid: 17085, tid: 17142, name: Thread-3  >>> org.pytorch.torchchat <<<
      #01 pc 0000000001913aec  /data/app/~~5lVO-wgDBy1pCoR1QhuHsg==/org.pytorch.torchchat-aRLplpZtZODSuxkK1AvhSw==/base.apk!libexecutorch_llama_jni.so (offset 0x1ab000) (et_pal_abort+8) (BuildId: 825150fdc280f1dd9cdaefab2ca01549b499eac3)
      #02 pc 0000000001913a40  /data/app/~~5lVO-wgDBy1pCoR1QhuHsg==/org.pytorch.torchchat-aRLplpZtZODSuxkK1AvhSw==/base.apk!libexecutorch_llama_jni.so (offset 0x1ab000) (torch::executor::runtime_abort()+8) (BuildId: 825150fdc280f1dd9cdaefab2ca01549b499eac3)
      #03 pc 00000000018cef60  /data/app/~~5lVO-wgDBy1pCoR1QhuHsg==/org.pytorch.torchchat-aRLplpZtZODSuxkK1AvhSw==/base.apk!libexecutorch_llama_jni.so (offset 0x1ab000) (torch::executor::Tiktoken::load(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&)+1772) (BuildId: 825150fdc280f1dd9cdaefab2ca01549b499eac3)
      #04 pc 00000000018c263c  /data/app/~~5lVO-wgDBy1pCoR1QhuHsg==/org.pytorch.torchchat-aRLplpZtZODSuxkK1AvhSw==/base.apk!libexecutorch_llama_jni.so (offset 0x1ab000) (torch::executor::Runner::load()+892) (BuildId: 825150fdc280f1dd9cdaefab2ca01549b499eac3)
      #05 pc 00000000001753dc  /data/app/~~5lVO-wgDBy1pCoR1QhuHsg==/org.pytorch.torchchat-aRLplpZtZODSuxkK1AvhSw==/base.apk!libexecutorch_llama_jni.so (offset 0x1ab000) (facebook::jni::detail::MethodWrapper<int (executorch_jni::ExecuTorchLlamaJni::*)(), &(executorch_jni::ExecuTorchLlamaJni::load()), executorch_jni::ExecuTorchLlamaJni, int>::dispatch(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch_jni::ExecuTorchLlamaJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>)+60) (BuildId: 825150fdc280f1dd9cdaefab2ca01549b499eac3)
      #06 pc 0000000000175304  /data/app/~~5lVO-wgDBy1pCoR1QhuHsg==/org.pytorch.torchchat-aRLplpZtZODSuxkK1AvhSw==/base.apk!libexecutorch_llama_jni.so (offset 0x1ab000) (facebook::jni::detail::FunctionWrapper<int (*)(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch_jni::ExecuTorchLlamaJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>), facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch_jni::ExecuTorchLlamaJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*, int>::call(_JNIEnv*, _jobject*, int (*)(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch_jni::ExecuTorchLlamaJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>))+60) (BuildId: 825150fdc280f1dd9cdaefab2ca01549b499eac3)
      #12 pc 0000000000001bec  [anon:dalvik-classes4.dex extracted in memory from /data/app/~~5lVO-wgDBy1pCoR1QhuHsg==/org.pytorch.torchchat-aRLplpZtZODSuxkK1AvhSw==/base.apk!classes4.dex] (org.pytorch.torchchat.MainActivity.setLocalModel+0)
      #17 pc 000000000000172c  [anon:dalvik-classes4.dex extracted in memory from /data/app/~~5lVO-wgDBy1pCoR1QhuHsg==/org.pytorch.torchchat-aRLplpZtZODSuxkK1AvhSw==/base.apk!classes4.dex] (org.pytorch.torchchat.MainActivity.access$700+0)
      #22 pc 000000000000138c  [anon:dalvik-classes4.dex extracted in memory from /data/app/~~5lVO-wgDBy1pCoR1QhuHsg==/org.pytorch.torchchat-aRLplpZtZODSuxkK1AvhSw==/base.apk!classes4.dex] (org.pytorch.torchchat.MainActivity$1.run+0)

Rough repro:

# setup python environment
python3 -m venv venv
source venv/bin/activate
export TORCHCHAT_ROOT=${PWD}
./scripts/install_et.sh

# android lib setup
wget https://ossci-android.s3.amazonaws.com/executorch/main/executorch-llama-tiktoken-rc3-0719.aar
mkdir android/torchchat/app/libs/
mv executorch-llama-tiktoken-rc3-0719.aar android/torchchat/app/libs/executorch.aar

# model export
python3 torchchat.py download stories110m
python3 torchchat.py export stories110m --output-pte-path stories110m.pte

# copy to device
adb shell mkdir -p /data/local/tmp/llama
adb push $HOME/.torchchat/model-cache/stories110M/tokenizer.model /data/local/tmp/llama
adb push stories110m.pte /data/local/tmp/llama

# run app
cd android/torchchat && ./gradlew installDebug

Versions

Executorch commit: b2c3f2697e2bbcda1b51f0c85adfaba27b900857

PyTorch version: 2.5.0.dev20240716
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.5 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.3.9.4)
CMake version: version 3.30.1
Libc version: N/A

Python version: 3.12.4 (main, Jun  6 2024, 18:26:44) [Clang 15.0.0 (clang-1500.3.9.4)] (64-bit runtime)
Python platform: macOS-14.5-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] executorch==0.4.0a0+c757499
[pip3] numpy==1.26.4
[pip3] torch==2.5.0.dev20240716
[pip3] torchao==0.3.1
[pip3] torchao==0.3.1
[pip3] torchaudio==2.4.0.dev20240716
[pip3] torchsr==1.0.4
[pip3] torchvision==0.20.0.dev20240716
[conda] Could not collect

[Jack-Khuu]

Hi Christian, thanks for testing out the repo!!

Stories uses a sentencepiece tokenizer, while the .aar provided in README is for a tiktoken tokenizer (e.g. Llama 3)
In general, other tokenizers can be build via the ExecuTorch script

Note: The AAR file listed in Step 1 has the tiktoken tokenizer, which is used for Llama 3. To tweak or use a custom tokenizer and runtime, modify the ExecuTorch code and use this script to build the AAR library.

If you want, you should give the script a try, but we're working on generating one for sentencepiece too and will have that uploaded

[Jack-Khuu]

Hi @iceychris, here's the aar with BPE sentencepiece. Should work with stories

https://ossci-android.s3.amazonaws.com/executorch/main/executorch-llama-bpe-rc3-0719.aar

We'll be adding it to the README in a bit