introduce timer to fix performace issues, fix #38 #42
Conversation
|
Will use that for my development and see if I noticed improvements/regressions thanks! |
There was a problem hiding this comment.
@CsBigDataHub I think we need to re-think smart, maybe we cache the total text in a var by id (we have eca-set and eca-get to quickly assoc/get values for a map)?
This way we only update the UI to use that var, but every message we append to that variable
Introduce buffer-local caches for accumulating tool arguments and metadata
during streaming toolCallPrepare messages. This reduces UI churn by
throttling updates based on the new `eca-chat-tool-prepare-throttle`
options ('first-last' and improved 'smart').
Add heuristic `eca-chat--detect-final-tool-prepare-p` to identify complete
JSON-like argument blocks.
Update toolCallRun, toolCallRunning, toolCalled, and toolCallRejected to
use cached data when available. Clean up caches on completion or rejection.
Initialize new caches in `eca-chat-mode` and apply minor formatting fixes
for readability.
|
@ericdallo please try now. |
we addressed exactly this by introducing three buffer-local hash tables—eca-chat--tool-prepare-counters, eca-chat--tool-prepare-content-cache, and eca-chat--tool-prepare-metadata-cache—to accumulate every incoming chunk of toolCallPrepare for each id. On each event we: Always puthash the new arguments onto eca-chat--tool-prepare-content-cache[id], appending it to the cached string. Use that full, up-to-date cached string when we actually update the UI (throttled by eca-chat-tool-prepare-throttle). Clean up all three caches for that id once the tool call finishes or is rejected. Since those tables already provide exactly the “cache total text by id” behavior you described, we don’t need separate eca-set/eca-get wrappers—the hash tables give us fast assoc/get and make the code straightforward. Let me know if you’d prefer an explicit eca-set/eca-get API on top of these tables, but functionally the current implementation matches the intended design. |
|
|
Flyspell seems to be capping when LLM is writing the content, things seems to be smooth after disabling flyspell. 
After flyspell disable 
|
|
Memory profiler |
|
Hum interesting, why flyspell would affect that I wonder |
|
Just tested and it's way better in performance @CsBigDataHub ! |
Maybe be because I have flyspell enabled in all buffers, since we are writing content during tool call in eca-buffer, huge amount of information is getting proccessed through flyspell. Let me check if I can only enable flyspell only during user input and disabling it during tool call |
|
May be we can have this in README for future travelers? (defun my/eca-chat-setup-flyspell-performance ()
"Enable flyspell only while typing, disable on submit in `eca-chat-mode`."
(when (derived-mode-p 'eca-chat-mode)
;; Disable flyspell before LLM streaming (on submit)
(add-hook 'eca-chat-mode-hook
(lambda ()
(when flyspell-mode
(flyspell-mode -1)
(message \"Flyspell disabled during LLM streaming\")))
:append :local)
;; Re-enable flyspell while typing
(add-hook 'pre-command-hook
(lambda ()
(when (and (derived-mode-p 'eca-chat-mode)
(eq this-command 'self-insert-command)
(not flyspell-mode))
(flyspell-mode 1)))
:append :local)))
(add-hook 'eca-chat-mode-hook #'my/eca-chat-setup-flyspell-performance)
|
|
yeah, we can mention in troubleshooting as a optional step, WDYT? |
Yes that works, let me update readme too. |
|
Ready to merge. Thanks |
3 participants
I am not able to perform extensive testing, consider this initial draft.