Have any plan to Implement function llama_state_set_data and llama_state_get_data ?

[webstatic]

I want this feature to restore a very long context history.

With normal setChatHistory, especially for my CPU LLM, it will take hours to reload.

But with "llama_state_get_data" and "llama_state_set_data" functions, it will take only few minute or even seconds to reload.

I already try to implement in AddonContext.cpp, I think the concept is work, it can remember something from reload file, but somehow the chat is broke.

I think there are states in javascript part that also need to be saved. But I don't knows what part and how to.

I hope that this project will implement this feature, it really useful for many use cases.

[webstatic]

from this example https://github.com/ggml-org/llama.cpp/blob/master/examples/save-load-state/save-load-state.cpp

[giladgd]

Yes!
It's in the works as part of a bigger feature that will make using node-llama-cpp even easier, but it's not ready yet.

I might release a version with the support for these functions separately earlier once I can assure the interface for this is stable and won't break in a future version.

Keep in mind that if your chat is longer than the context size, then a context shift will have to occur and reevaluate most of the context state either way.

[giladgd]

I've added this feature in the latest release (version 3.8.0): https://node-llama-cpp.withcat.ai/guide/chat-session#save-and-restore-with-context-sequence-state

[webstatic]

thank you