add _readdirx for returning more object info gathered during dir scan

[IanButterworth]

Note: This has been converted into an internals only function for 1.11 as a backport for internal performance gains. In 1.12 it should be exported with a bikeshedded name.

---

Based on #53153 (comment) @vtjnash

The uv_fs_scandir that readdir uses gathers the object types, where known, but we currently do no use them.

This introduces an internal _readdirx and the FileKind DirEntry object (names tbd) to give the opportunity for faster isfile, isdir etc. by using that info. Also the objects retain the dir so both their name and path can be retained.

Hoping to help WSL, but it's not clear how common it is to have a filesystem that just returns UV_DIRENT_UNKNOWN.

On MacOS:

julia> @time count(isfile, readdir("/Users/ian/Downloads", join=true))
  0.001515 seconds (1.20 k allocations: 98.406 KiB)
219

julia> @time count(isfile, readdirx("/Users/ian/Downloads"))
  0.000237 seconds (245 allocations: 33.344 KiB)
219

julia> readdirx()
45-element Vector{Base.Filesystem.DirEntry}:
 Base.Filesystem.DirEntry(".", ".DS_Store", 1)
 Base.Filesystem.DirEntry(".", ".buildkite-external-version", 1)
 Base.Filesystem.DirEntry(".", ".clang-format", 1)
 Base.Filesystem.DirEntry(".", ".clangd", 1)
 Base.Filesystem.DirEntry(".", ".codecov.yml", 1)
 Base.Filesystem.DirEntry(".", ".devcontainer", 2)
...

julia> for obj in Base.Filesystem._readdirx(pwd())
           if isfile(obj) && obj == "README.md"
               @info "readme found: $(joinpath(obj))"
           end
       end
[ Info: readme found: /Users/ian/Documents/GitHub/julia/README.md

[fatteneder]

Hoping to help WSL, but it's not clear how common it is to have a filesystem that just returns UV_DIRENT_UNKNOWN.

Just a side comment:
I think it would possible to detect the file system for which rawtype gives useful information using uv_fs_statfs https://docs.libuv.org/en/v1.x/fs.html#c.uv_fs_statfs.
The relevant fs type constants which are supported according to https://docs.libuv.org/en/v1.x/fs.html#c.uv_fs_scandir_next would be

BTRFS_SUPER_MAGIC     0x9123683e
BTRFS_TEST_MAGIC      0x73727279
EXT2_OLD_SUPER_MAGIC  0xef51
EXT2_SUPER_MAGIC      0xef53
EXT3_SUPER_MAGIC      0xef53
EXT4_SUPER_MAGIC      0xef53

But I think this is not of much help, and using instead cached stat, lstat fields should provide cheap enough isfile, isdir, ... calls in case rawtype is unknown.

[IanButterworth]

I think readdirx shouldn't ever call stat, because it's too expensive given the user may not even need to stat a file that, for instance, doesn't have a name that matches a pattern.

This is more about not throwing information away, rather than always being complete

[topolarity]

On WSL2, this gives me about a ~300x speedup (as expected) 🎉

julia> @time count(isfile, readdir("/mnt/c/Windows/system32", join=true))
 11.344859 seconds (25.32 k allocations: 1.850 MiB)
4913

julia> @time count(isfile, readdirx("/mnt/c/Windows/system32"))
  0.034974 seconds (5.08 k allocations: 545.961 KiB)
4913

[topolarity]

Also provides a nice 50x speed-up (and avoids an EACCES error) on native Windows:

julia> @time count(f->try isfile(f) catch _ false end, readdir("/Windows/system32", join=true))
  0.230555 seconds (144.26 k allocations: 6.848 MiB, 3.29% gc time, 11.45% compilation time)
4913

julia> @time count(isfile, readdirx("/Windows/system32"))
  0.004718 seconds (5.08 k allocations: 664.711 KiB)
4913

[IanButterworth]

I haven't come up with clearly better names so far, but some thoughts..

• readdirx: readdirinfo makes sense but might be a bit awkward to read?

• FileKind: Entry might make more sense as a generic filesystem entry, but maybe too generic without the filesystem context. It could be non-exported, so always show as Filesystem.Entry I think?

[topolarity]

My vote would be to rename FileKind to DirEntry (esp. since this is the only place we expect such a struct to appear)

I actually like readdirx but maybe Unix has corrupted me. readdirinfo makes me feel like I'm doing a stat on a directory or similar.

[IanButterworth]

If we backport this to 1.11 it can be used to speed up tab completion hint path scans (estimates of by ~300x on WSL, ~50x on Windows).

I'll mark it for triage to discuss whether that's reasonable.

[LilithHafner]

Triage thinks that it would be good to name this _readdirx and keep it internal, then merge and backport, and add this as a public feature in 1.12 (with a different name, possibly readdirx, but also possibly a different name, I and Oscar personally don't love the readdirx and don't want that bikeshed to block these perf improvements reaching tab completion).

[IanButterworth]

@topolarity in view of the triage review that this should go in as an internal into 1.11 so we can figure out the proper name API etc. for 1.12, can you give a final review please and thanks

[topolarity]

Thanks for putting this together!

Greatly looking forward to the performance improvements this will enable

[IanButterworth]

Just an idea for the bikshedded names readdirentries and DirEntry. Ties the two together