Very slow *editing* of large files when tree-sitter is used

[kchibisov]

Summary

Certain markdown files are very slow to edit. In particular when they have a lot of lists.
For example a CHANGELOG.md file from the helix repo is a enough to trigger it.

The problem is likely in grammar here, since neovim with tree-sitter works fine on it.

While this particular problem could be solved with slightly optimized, but less accurate grammar. However the issue manifests itself on any large file of any other type, just requires much larger file than ~600LoC(I've used 50K LoC C++ that I have around).

To avoid delay and input lag in those particular scenarios it would make sense to either run tree-sitter only on a part of the buffer (I'm not familiar how it works) or update it after the typing ended. Since right now you have a latency of like 2 seconds on 50K lines file of C++, which is impossible to work with.

Reproduction Steps

Try to edit CHANGELOG.md from the helix repo or any large file.

Helix log

No response

Platform

Linux

Terminal Emulator

alacritty

Helix Version

helix 22.05 (3cced1e)

[MDeiml]

Author of the markdown grammar here. It seems that at the moment language injection cannot be parsed incrementally. The markdown grammar uses injection a lot so it becomes quite apparent for it. This is what I get from line 846 here:

helix/helix-core/src/syntax.rs

Lines 841 to 848 in 8681fb6

	let layer = self
	.layers
	.iter_mut()
	.find(|(_, layer)| {
	layer.depth == depth && // TODO: track parent id instead
	layer.config.language == config.language && layer.ranges == ranges
	})
	.map(|(id, _layer)| id);

Notice how a layer can only be reused if ranges did not change. But this is often the case when editing a file. The test here could probably be laxer.

I didn't look to deep, but (from my knowledge of treesitter) any layer should be fine here to "reuse" as long as none is reused twice. So something akin to

1. Choosing the layer whose ranges "match best" in some way
2. Changing those ranges to the new ones

should work.

[archseer]

We actually follow tree-sitter-cli/atom logic to update all the markers according to user edits beforehand: https://github.com/helix-editor/helix/blob/8681fb6d9ee06610fac2baafb4ca34f93f9c7f4d/helix-core/src/syntax.rs#L644-L726=

So the injection layers are tracked reliably.

We also seem to only use HTML injections & code blocks: https://github.com/helix-editor/helix/blob/master/runtime/queries/markdown/injections.scm

[MDeiml]

Ah ok, I guess you are still using the old version of my parser then (the main branch). You could try updating to the split_parser branch. That one has some major performance improvements, but uses injection very extensively.

[archseer]

Ah, good point! I'll see if I can update us to the new branch 👍

[asb]

On an 8.4MiB markdown file (I keep daily notes in a single big document), helix 22.05 takes > 10s to open it and is used 1GiB of memory after doing so. I'm assuming this is tree-sitter related and the same issue described here, but let me know if you think not.

[asb]

The following script generates a ~5MiB file that takes >5 seconds to open with Helix. After opening, memory usage (reported via ps_mem) is ~660MiB.

#!/bin/sh

for i in $(seq 10000); do
  cat <<EOF >> big_markdown.md
# Title
## Subtitle
Some notes about things.

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis
nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore
eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt
in culpa qui officia deserunt mollit anim id est laborum

* A list
  * With a sub-item
* Item 2
* Item 3
* Item 4
* Item 5
  * Item 5.1

EOF
done

[kchibisov]

@asb if you disable grammar for markdown does that happen?

[asb]

@kchibisov I'm not sure what option to change to disable the grammar (I was trying out helix for the first time). Certainly, if mv big_markdown.md big markdown.txt and open the version of the file with a .txt extension, it opens instantly and helix consumes a more reasonable 18MiB RAM.

[MDeiml]

I think the simplest way to fix this is to only run the grammar for injected languages if they are visible (i.e. run it lazily).

Markdown is split into two grammars: block and inline. The block-level grammar is quite fast and produces the ranges on which the inline-level grammar is injected. So the computational cost mostly grows with the amount of inline content parsed.

I'd guess that the memory cost is also from the parsed inline-level trees, so that problem would also be solved in the same way, but I'd have to check that.

[MDeiml]

I'm not sure how to best implements something like this tough. Currently the parsing is handled by the Document impl, but for lazy parsing we need information from the Views that point to that document.

[Yevgnen]

Having trouble when open rust.org.gz 3000 line .org file and never finish. Renaming it to abc.xyz to avoid syntax highlight then the process is instant.