Difficult to change max seq len in android llama demo app

[CHNtentes]

Hi,
I tried android llama demo app. While it works, I don't know how to make generated text longer.
I already set max length to 4096 when exporting pte model, and when I use llama_main cli I can set seq_len to 4096.
How do I modify this value in demo app code?

[afwcole]

Hey I ran into this issue aswell. You have to edit the sequence length in the jni_layer_llama.cpp file. can be found in the following folder "extension/android/jni/jni_layer_llama.cpp" for the generate function.

It is set to 128 but you will have to change it 4096
jint generate( facebook::jni::alias_ref<jstring> prompt, facebook::jni::alias_ref<ExecuTorchLlamaCallbackJni> callback) { runner_->generate( prompt->toStdString(), **4096,** [callback](std::string result) { callback->onResult(result); }, [callback](const Runner::Stats& result) { callback->onStats(result); }); return 0; }

Here is the link to the line you have to change here

[larryliu0820]

@afwcole thanks for your reply! Also cc @kirklandsign

[chienhuikuo]

Hey I ran into this issue aswell. You have to edit the sequence length in the jni_layer_llama.cpp file. can be found in the following folder "extension/android/jni/jni_layer_llama.cpp" for the generate function.

It is set to 128 but you will have to change it 4096 jint generate( facebook::jni::alias_ref<jstring> prompt, facebook::jni::alias_ref<ExecuTorchLlamaCallbackJni> callback) { runner_->generate( prompt->toStdString(), **4096,** [callback](std::string result) { callback->onResult(result); }, [callback](const Runner::Stats& result) { callback->onStats(result); }); return 0; }

Here is the link to the line you have to change here

Hi, @afwcole, I encountered this issue too.
However, the jni_layer_llama.cpp file in the master branch no longer includes the line of code that sets seq_len to 128.
Does anyone know how to increase the maximum sequence length?

[kirklandsign]

Hi @chienhuikuo this is passed from Java layer
https://github.com/pytorch/executorch/blob/main/extension/android/src/main/java/org/pytorch/executorch/LlamaModule.java#L75

[chienhuikuo]

Hi @kirklandsign do you mean that if I want to set max_seq_len to 512, I should modify it here (https://github.com/pytorch/executorch/blob/main/examples/demo-apps/android/LlamaDemo/app/src/main/java/com/example/executorchllamademo/MainActivity.java#L758) by passing the max_seq_len value (512) via the generate() to the Java layer?

I tried doing so, but nothing changed—the error still persists.
If there’s anything wrong with my approach, I would appreciate your guidance.

[kirklandsign]

Hi @chienhuikuo this should be passed to

executorch/extension/android/jni/jni_layer_llama.cpp

Line 211 in 30d5a11

seq_len,

and then to llama runner.

Have you tried on linux machine?

[infil00p]

This seems related to this issue I had when getting torchchat to work. pytorch/torchchat#1395

[chienhuikuo]

@kirklandsign I would prefer running this on Android rather than on a Linux machine.

@infil00p Yes, it seems to be the same issue. Based on the comments in pytorch/torchchat#1395, could this be a bug?

[infil00p]

Well, where is the max_seq_len even being set? When I compile it to C++, I'm also stuck with that hard limit and can't generate anything past 128 tokens.

[infil00p]

OK, so in theory, we should be pulling it from the metadata in the pte, but I noticed that the Qualcomm code has it hardcoded to 1024 and the example code has it hardcoded to 128. It doesn't matter what the sequence number is, it's not going to affect the max_seq_len, which will be stuck at 128 forever.

https://github.com/pytorch/executorch/blob/main/examples/models/llama/runner/runner.cpp#L51

I don't know how to pull it from the metadata, so I'm just going to bump the hardcoded value to 4096 to get around the issue. I'm going to do some quick tests on my end to confirm that fixes it.

As for why this doesn't break on iOS, is it actually sending the whole conversation?