Error exporting Llama 3 8B instruct model for mobile

[hietalajulius]

Hey!

When running export Llama 3 8B instruct

python -m examples.models.llama2.export_llama --checkpoint <consolidated.00.pth> -p <params.json> -kv --use_sdpa_with_kv_cache -X -qmode 8da4w --group_size 128 -d fp32 --metadata '{"get_bos_id":128000, "get_eos_id":128001}' --embedding-quantize 4,32 --output_name="llama3_kv_sdpa_xnn_qe_4_32.pte"

as instructed here results in

...
	size mismatch for layers.4.attention_norm.weight: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([64]).
	size mismatch for layers.4.ffn_norm.weight: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([64]).
	size mismatch for norm.weight: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([64]).
	size mismatch for output.weight: copying a param with shape torch.Size([128256, 4096]) from checkpoint, the shape in current model is torch.Size([512, 64]).

Is there an example of a params.json file / should there be a different export script?

Thanks in advance!

[hietalajulius]

ah nevermind, the params.json comes with the downloaded model. However now I get

File "/my_local_executorch/examples/models/llama2/model.py", line 119, in __init__ model_args: ModelArgs = ModelArgs( TypeError: ModelArgs.__init__() got an unexpected keyword argument 'rope_theta'

[iseeyuan]

@hietalajulius the rope_theta issue should have been resolved in #3147. Please make sure you are checking out the updated code from main.

[hietalajulius]

Thanks! It seems to be exporting fine now. However my iPhone 12 Pro is probably too old, the app crashes after a while after prompting, the logs in Xcode don't look very helpful

The view service did terminate with error: Error Domain=_UIViewServiceErrorDomain Code=1 "(null)" UserInfo={Terminated=disconnect method}
The view service did terminate with error: Error Domain=_UIViewServiceErrorDomain Code=1 "(null)" UserInfo={Terminated=disconnect method}
I 00:03:26.206198829211000000 executorch:runner.cpp:54] Creating LLaMa runner: model_path=/private/var/mobile/Containers/Data/Application/EE83BA14-2248-4C99-8196-214FF502796A/Documents/llama3_kv_sdpa_xnn_qe_4_32.pte, tokenizer_path=/private/var/mobile/Containers/Data/Application/EE83BA14-2248-4C99-8196-214FF5$
I 00:03:26.198829 executorch:runner.cpp:54] Creating LLaMa runner: model_path=/private/var/mobile/Containers/Data/Application/EE83BA14-2248-4C99-8196-214FF502796A/Documents/llama3_kv_sdpa_xnn_qe_4_32.pte, tokenizer_path=/private/var/mobile/Containers/Data/Application/EE83BA14-2248-4C99-8196-214FF5$
Unable to simultaneously satisfy constraints.
	Probably at least one of the constraints in the following list is one you don't want. 
	Try this: 
		(1) look at each constraint and try to figure out which you don't expect; 
		(2) find the code that added the unwanted constraint or constraints and fix it. 
	(Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints) 
(
    "<NSAutoresizingMaskLayoutConstraint:0x300c108c0 h=-&- v=--& SwiftUI.UIKitNavigationBar:0x137e53f00.minY == 0   (active, names: '|':UILayoutContainerView:0x137e6c500 )>",
    "<NSAutoresizingMaskLayoutConstraint:0x300c10910 h=-&- v=--& SwiftUI.UIKitNavigationBar:0x137e53f00.height == 56   (active)>",
    "<NSLayoutConstraint:0x300c101e0 V:[SwiftUI.UIKitNavigationBar:0x137e53f00]-(0)-[UIFocusContainerGuide:0x30108a760'UINavigationControllerContentFocusContainerGuide']   (active)>",
    "<NSLayoutConstraint:0x300c102d0 UIFocusContainerGuide:0x30108a760'UINavigationControllerContentFocusContainerGuide'.bottom == UILayoutContainerView:0x137e6c500.bottom   (active)>",
    "<NSLayoutConstraint:0x300c10aa0 'UIView-Encapsulated-Layout-Height' UILayoutContainerView:0x137e6c500.height == 0   (active)>"
)

Will attempt to recover by breaking constraint 
<NSLayoutConstraint:0x300c101e0 V:[SwiftUI.UIKitNavigationBar:0x137e53f00]-(0)-[UIFocusContainerGuide:0x30108a760'UINavigationControllerContentFocusContainerGuide']   (active)>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKitCore/UIView.h> may also be helpful.
interruptionHandler is called. -[FontServicesDaemonManager connection]_block_invoke
[C:2] Error received: Connection interrupted.
[C:2-1] Error received: Connection interrupted.
-[RTIInputSystemClient _configureConnection:withMachName:]_block_invoke  Client connection to service was interrupted: <NSXPCConnection: 0x301eacaa0> connection to service with pid -1 named (null)
<PKDaemonClient: 0x302f02820>: XPC error talking to pkd: Connection interrupted
[C:2] Error received: Connection interrupted.
[C:2-1] Error received: Connection interrupted.
-[RTIInputSystemClient _configureConnection:withMachName:]_block_invoke  Client connection to service was interrupted: <NSXPCConnection: 0x301eacaa0> connection to service with pid -1 named (null)
[C:2] Error received: Connection interrupted.
[C:2-1] Error received: Connection interrupted.
-[RTIInputSystemClient _configureConnection:withMachName:]_block_invoke  Client connection to service was interrupted: <NSXPCConnection: 0x301eacaa0> connection to service with pid -1 named (null)