Loading files is very slow.

[kevincox]

Hello, I was loading a large directory and loading it took more then 10 minutes. I wasn't particularly surprised until I noticed that find or git ls-files took a small number of seconds to complete. It seems surprising to me that there is so much overhead.

I assume that most of this can be eliminated by moving the scanner to C. Would you be opposed to a patch that allowed some scanners to be implemented in C? (Of course I would check that the performance is actually significantly improved)

[wincent]

I wouldn't be opposed. I love speed. Most of the cost, however, is probably not in the scanner itself but in the calls to Vim's expand() function, and that won't get any faster by rewriting the other parts in C. You could test this hypothesis by temporarily removing the part of the code that calls expand().

Also note that if you're running on a really huge directory you should probably just use the Watchman scanner, which will toast even a C implementation. (Watchman is itself written in C, maintains a persistent, up-to-date cache of file-system state, and the code in Command-T for talking to Watchman is already written in C, and talks to Watchman using an efficient binary protocol.)

[kevincox]

I wouldn't be opposed. I love speed. Most of the cost, however, is probably not in the scanner itself but in the calls to Vim's expand() function, and that won't get any faster by rewriting the other parts in C. You could test this hypothesis by temporarily removing the part of the code that calls expand().

Awesome to know. I obviously can't promise more performance but I'm willing to dig in. I have been browsing the code and am suspicious of a couple of bits (including the expand) and I'll see what I can do about them, if anything.

Also note that if you're running on a really huge directory you should probably just use the Watchman scanner

Unfortunately this isn't possible for all my setups but it does sound like a nice solution :)

[wincent]

One thing I just remembered: if you're doing anything major you should build on top of the next branch and not the master branch. Due to the way most people tend to install Vim plug-ins (often by pointing at HEAD of master), the master branch must be kept very stable and slow moving; all real feature work happens on next and gets merged back in once its sufficiently baked.

I'd forgotten about this but, I landed 4b94a60 back in May which significantly speeds up the wildignore stuff anyway (for more context see 589f450). Anything you do on master would conflict with that work, so next is the place to work from.

[kevincox]

Thanks. That sounds like it could make a huge difference (mabye 10x based on my observations). I'm still working on a C scanner API which would allow reading and matching filelist without the data ever entering Ruby (just the results). It looks like it is going to have another 2-4x improvement. However that would have been almost in the noise if we still had to go to VIM to filter so these two together sound really promising. The original plan was to keep the current scanners "as is" for the time being but if the filter can be reduced to a regex it will be quite easy to pull that into C.

[dylan-chong]

I'm not a vim expert, but would the async feature of vim/neovim be helpful here. I.e. could you queue up a task to load files when the user opens vim so that it you don't have to wait?

[kevincox]

That would definitely possible but it would be orthogonal to this change.

[dylan-chong]

Ah I see. Nevermind me

[wincent]

@dylan-chong:

I'm not a vim expert, but would the async feature of vim/neovim be helpful here. I.e. could you queue up a task to load files when the user opens vim so that it you don't have to wait?

For that, see #220

[wincent]

Given the big rewrite for v6.0.x, I'm closing all older issues as there is unlikely to be anything significant happening on the 5-x-devel branch from here on¹. Feedback issue for 6.0.x is here:

• 🆕 Command-T 6.0.0 release — compendium issue #393

Footnotes

1. Patches and PRs would be welcome, but my personal efforts are going to be directed towards main. ↩