Files panel is empty when (many) new/changed files exist

[vlcinsky]

Describe the bug
While git status shows that changed/new files exist, lazygit panel "Files" panel is empty.

This happens when number of new/changed files is really large (e.g. 90 thousands) - lazygit is aware of their existence but it takes 9 seconds (on my MacBook Pro M1) until anything is printed into Files panel. Until then, user feels like lazygit ignores the changes as there is no sign, lazygit is still trying to show somehting.

To Reproduce
Steps to reproduce the behavior:

1. Prepare new repository with simple initial commit:

$ mkdir manylazyfiles
$ cd manylazyfiles
$ touch README.md
$ git init
Initialized empty Git repository in manylazyfiles/.git/
$ git add README.md
$ git commit -m "Initial commit"
[master (root-commit) 45d7d58] Initial commit
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 README.md

2. Prepare a folder with 90000 files

mkdir data
seq 90000 |xargs -I {} touch data/file-{}.txt

3. Check git status:

$ git status
On branch master
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	data/

nothing added to commit but untracked files present (use "git add" to track)

4. Open lazygit and watch content of the Files panel

lazygit

The panel is empty and lazygit seems ignoring the newly created files until like 9 seconds and then the files are shown properly.

Typical developer will give up after few seconds closing lazygit with unfair remark "ignorant".

Expected behavior
lazygit would print initially "Reading files..." into Files panel and would then replace it with whatever final result would finally get there.

This could happen always or (optimized variant) only if the content of the panel is not populated within a second.

Screenshots
Initial presentation with empty Files (during rendering large number of new items):

And after cca 9 seconds the files are finally shown:

Version info:

$ lazygit --version
commit=, build date=, build source=homebrew, version=0.43.1, os=darwin, arch=arm64, git version=2.45.2
$ git --version
git version 2.45.2

[stefanhaller]

lazygit would print initially "Reading files..." into Files panel and would then replace it with whatever final result would finally get there.

Before we do that, we should see if it's possible to optimize the code so that it doesn't take so long. There's really no reason why it should take several seconds to populate the files panel with a few thousand files.

A cursory profiling shows that almost all the time is spent inside the BuildTreeFromFiles function, so it looks like we only need to speed up that function. Looking at that function (with squinted eyes), it does look like it has quadratic behavior when there are lots of files in the same directory, but I don't have time to investigate this further right now. Do you feel like looking into this?

[parthokunda]

Can I take a loot at it?

[vlcinsky]

Great.
Guys @stefanhaller , @parthokunda you did excellent job.

In my testing repository "manylazyfiles" where it took 9 seconds it now takes fraction of a second (I would say 0.3).

Me and my colleague also enjoy your approach - simply optimizing the process without introducing complexities with "loading ...".

Well done.