Opening largish files is crazy slow due to SMIE

[bitwalker]

I have an Elixir file with ~4k lines in it, more than a few of them actually - and every time I open them, it takes ~10-20 seconds for the file to open. Its honestly driving me nuts.

I ran the profiler to see what is going on, and it looks like the only thing taking any meaningful time at all is elixir-mode's use of SMIE, in particular the elixir-smie--semi-ends-match function and its use of looking-back. It so utterly dominates the samples in the profiler, that fixing this would completely fix the problem I'm seeing (and presumably others are seeing as well, its wild to me that this isn't reported already).

It seems to me that a much more efficient method than is currently used in elixir-smie--semi-ends-match should be possible, or at least use some heuristics to avoid backtracking nearly as much as it does currently. The documentation for looking-back specifically states to avoid using it wherever possible, and we're using it twice to first detect elixir-smie--block-operator-regexp but then reject .+fn.+. At a minimum it would be faster to add a specialized regex that does that match in one go. If I knew more about how all the pieces fit together, I'd take a stab at rewriting it, but wanted to get some feedback first. Have you seen this issue? If so, how have you handled it in your own config?

[axelson]

I have noticed this too, and we were having a discussion about it in #emacs on the Elixir Slack the other day. Some improvements to the performance of the syntax detection would be very welcome.

[victorolinasc]

Hi @bitwalker ! It certainly can be improved... PRs are welcome! I myself haven had to deal with such a large file yet and I am running the native-comp branch currently so haven't seen this kind of slowdowns yet. Anyway I agree the current algorithm can be improved upon.

[lambdadog]

If anyone has an example of such a large elixir file that could be tested against, that might be helpful in addressing this issue.

[bitwalker]

I can’t post the originals that trigger this in my work project, but I’ll put together a repro based on them and post back here.

In general though, you can probably also just take any typical Elixir module, clone the contents of it until you have at least a couple thousand lines of code, save it and reopen the file to reproduce. If you run the profiler you should observe an exponential increase in the time spent opening the buffer.

[maxlorenz]

You can simply do this:

iex(1)> {:ok, file} = File.open("demo.exs", [:write])
iex(2)> Enum.each(1..5000, fn _ -> IO.binwrite(file, "def sq(x), do: x * x\n") end)
iex(3)> File.close(file)

The resulting file will take ~20-30 seconds to open, vs ~1s for the same amount of lines of Python or Ruby on my machine

[korsmakolnikov]

Up