counsel-projectile-find-file extremely sluggish in larger repositories.

[dminuoso]

When switching the project to https://github.com/NixOS/nixpkgs (~25,000 files), or calling M-x counsel-projectile-find-file inside the same repository, typing a few letters (say "postfix") causes emacs/ivy to hang for a few seconds on my laptop until the ivy minibuffer refreshes and displays the filtered strings. In comparison, projectile-find-file is much more snappy, which seems to respond within a hundred milliseconds or so.

I also tried to set counsel-projectile-find-file-more-chars to 2 to no avail.

My counsel/ivy/projectile config is completely left at default.

Here's the output of CPU and memory profilers:

CPU profiler

- command-execute                                                4889  57%
 - call-interactively                                            4889  57%
  - funcall-interactively                                        4770  55%
   - counsel-projectile-switch-project                           4719  55%
    - ivy-read                                                   4717  55%
     - ivy-call                                                  4622  53%
      - counsel-projectile-switch-project-action                 4622  53%
       - counsel-projectile-switch-project-by-name               4622  53%
        - #<lambda 0xf428ca0>                                    4620  53%
         - counsel-projectile                                    4620  53%
          - ivy-read                                             4617  53%
           - read-from-minibuffer                                4521  52%
            - ivy--queue-exhibit                                 4471  52%
             - ivy--exhibit                                      4471  52%
              - ivy--update-minibuffer                           4464  52%
               - ivy--filter                                     3017  35%
                + counsel-projectile--matcher                    1719  20%
                + ivy--recompute-index                           1297  15%
                  ivy--regex-plus                                   1   0%
               + ivy--format                                     1447  16%
              + ivy--insert-minibuffer                              6   0%
            + minibuffer-inactive-mode                              2   0%
            + timer-event-handler                                   2   0%
              undo-auto--add-boundary                               1   0%
           + ivy--reset-state                                      64   0%
          + projectile-project-p                                    1   0%
          + projectile-prepend-project-name                         1   0%
        + hack-dir-local-variables-non-file-buffer                  2   0%
     + read-from-minibuffer                                        75   0%
     + ivy--reset-state                                             5   0%
    + projectile-project-p                                          1   0%
    + projectile-project-root                                       1   0%
   + execute-extended-command                                      51   0%
  + byte-code                                                     119   1%
- ...                                                            3673  42%
   Automatic GC                                                  3673  42%
+ evil-repeat-post-hook                                             1   0%
+ evil--jump-hook                                                   1   0%
+ direnv--maybe-update-environment                                  1   0%
+ redisplay_internal (C function)                                   1   0%
+ timer-event-handler                                               1   0%
  undo-auto--add-boundary                                           1   0%

Memory profiler

- command-execute                                       3,380,453,797  99%
 - call-interactively                                   3,380,453,797  99%
  - funcall-interactively                               3,377,042,321  99%
   - counsel-projectile-switch-project                  3,373,287,970  99%
    - ivy-read                                          3,373,284,421  99%
     - ivy-call                                         3,373,145,049  99%
      - counsel-projectile-switch-project-action        3,373,145,049  99%
       - counsel-projectile-switch-project-by-name      3,373,145,049  99%
        - #<lambda 0xf428ca0>                           3,373,122,051  99%
         - counsel-projectile                           3,373,122,051  99%
          - ivy-read                                    3,373,111,234  99%
           - read-from-minibuffer                       3,313,649,682  98%
            - ivy--queue-exhibit                        3,313,561,434  98%
             - ivy--exhibit                             3,313,561,434  98%
              - ivy--update-minibuffer                  3,313,181,018  98%
               + ivy--format                            1,742,675,931  51%
               - ivy--filter                            1,570,505,087  46%
                - ivy--recompute-index                  1,502,558,612  44%
                 - ivy--completing-fname-p              1,502,557,556  44%
                  - counsel-projectile--project-buffers-and-files 1,502,557,556  44%
                   - projectile-current-project-files   1,498,212,421  44%
                    - projectile-project-files          1,498,035,757  44%
                     - projectile-dir-files-alien       1,498,035,757  44%
                      - projectile-files-via-ext-command 1,463,755,470  43%
                       - shell-command                  1,400,129,372  41%
                          call-process-shell-command    1,400,084,492  41%
                          make-temp-file                        3,152   0%
                        + delete-file                           3,048   0%
                       + split-string                      61,078,085   1%
                         string-trim                           94,191   0%
                         generate-new-buffer                   28,821   0%
                      + projectile-get-sub-projects-files       134,115   0%
                      + projectile-project-vcs                 16,276   0%
                    + projectile-acquire-root                 176,664   0%
                   + counsel-projectile--project-buffers     3,170,679   0%
                     remove                                   946,176   0%
                   + projectile-project-root                  183,804   0%
                     file-relative-name                        44,476   0%
                 + ivy--preselect-index                         1,056   0%
                + counsel-projectile--matcher              67,944,371   2%
                + ivy--regex-plus                               2,104   0%
              + ivy--insert-minibuffer                        318,648   0%
              + ivy-set-text                                   59,672   0%
                ivy--input                                      2,096   0%
            + redisplay_internal (C function)                  36,424   0%
            + command-execute                                   6,128   0%
            + direnv--maybe-update-environment                  3,168   0%
           + ivy--reset-state                              59,435,965   1%
           + ivy--update-prompt                                 1,024   0%
          + projectile-prepend-project-name                     8,779   0%
          + projectile-project-p                                1,488   0%
          + projectile-maybe-invalidate-cache                     550   0%
        + hack-dir-local-variables-non-file-buffer             21,929   0%
     + read-from-minibuffer                                   105,273   0%
     + ivy--reset-state                                        29,600   0%
     + ivy--update-prompt                                       1,024   0%
      abbreviate-file-name                                      2,048   0%
    + projectile-prepend-project-name                             585   0%
    + projectile-project-p                                        458   0%
    + projectile-project-root                                     458   0%
   + execute-extended-command                               3,754,351   0%
  + byte-code                                               3,411,476   0%
+ redisplay_internal (C function)                              18,740   0%
  evil-repeat-pre-hook                                          8,188   0%
  ...                                                               0   0%

[dminuoso]

The immediate problem appears to be

counsel-projectile/counsel-projectile.el

Line 1632 in 06b03c1

#'counsel-projectile--project-buffers-and-files

This collection function is called each time a new character is added to the input (but not when characters are removed, I guess ivy-read simply caches the filtered results).

In a non-trivial repository this can incur several hundred milliseconds on a single character stroke, so if one rapidly types in foo.nix, this will call the function 6 times in rapid succession and hanging for a few seconds.

[huangfeiyu]

I am experiencing the same issue. I use counsel-projectile to open file, when I type the file name, the issue occurs. But if I just paste the file name without typing, I get the result quickly.

[ericdanan]

Hi, thanks for reporting. I guess you are right that the issues comes from the use of a collection function. I didn't notice it because I don't use large repositories. It is probably worth trying to change this, but unfortunately it will probably take a while until I can look at it, sorry about that (PRs welcome of course).

[nicodebo]

Hello,
I just want to say that I'm having the exact same issue.
The counsel-projectile command is very laggy with a folder of about 200 files while projectile-find-file feels very responsive.

[laustbn]

Hi,

I also see this issue on macOS. If I watch active processes on my machine whilst running counsel-projectile-find-file, counsel-projectile is causing Projectile to re-index the project for every(?) keystroke I do in the mini-buffer. When Projectile indexes a project it will call external commands like git, find, etc. and I believe this is where the slowness comes from. The CPU profile above doesn't reveal this, probably because Emacs is simply waiting for external commands to finish. shell-command does show up in the memory profile, however. I would guess the root cause is the constant re-indexing, but I didn't look into the code to figure out why that happens.

As a workaround I found that if I enable caching of project files in Projectile (with (setq projectile-enable-caching t)), it stops calling external commands and responsiveness becomes similar to projectile-find-file, at least for the projects I work on (few thousand files).

[DerGuteMoritz]

@laustbn

As a workaround I found that if I enable caching of project files in Projectile (with (setq projectile-enable-caching t)), it stops calling external commands and responsiveness becomes similar to projectile-find-file, at least for the projects I work on (few thousand files).

Great tip, thanks for sharing! I recommend reading the docs on caching to understand how/when you would need to manually purge the cache.

Another thing I noticed that it was still sluggish when switching between projects with counsel-projectile-switch-project. Turns out that this is due to the default action being counsel-projectile-switch-project-action which also adds buffers to the set of completion candidates. Setting it to counsel-projectile-switch-project-action-find-file instead makes it snappy, too.