long file name support by default?

[davidziman]

Seems long file names may not work in the Windows x64 build of ripgrep 0.4.0.

In cmd on windows 10 (Version 1607 OS Build 14393.693) do the following:

cd c:\
mkdir ar
cd ar
echo "foobar" > abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs

Execute rg:

rg "foobar"

Result:
./abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs: The system cannot find the path specified. (os error 3)

Try again:

rg "foobar" \\?\%cd%

Result:
\\?\c:\ar\abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs 1:"foobar"

It would be nice to have long file name support by default without having to add \\?\%cd% .

[BurntSushi]

Someone will have to explain how to achieve to this in enough detail for me to do it, or someone else will need to submit a PR.

[JFLarvoire]

I have a solution:
I've published on https://github.com/JFLarvoire/SysToolsLib/tree/master/C/MsvcLibX a Microsoft C library extension, with support for long file names >> 260 characters, UTF-8 file names, symbolic links, etc.
By building ripgrep with MsvcLibX, you should automatically inherit all these capabilities.
Note that MsvcLibX currently has support for many file and directory access functions, but not all. I've never tried it with programs using memory map functions like mmap(). Anyway, it should be relatively easy to add any missing function.

[BurntSushi]

@JFLarvoire Thanks for the pointer!

Allow me to be clearer: I am not a Windows programmer. I know almost nothing about it or its ecosystem. Anything that complicates the build process is something I'm unlikely to want to add. I would rather not add additional C libraries (for Windows or Linux).

If standard Windows tooling doesn't support long file paths, then I'm not sure what to do. Someone will need to explain the actual problem in depth. If it's fixable in Rust, then we should do that. Pointing to a giant C library doesn't really help me understand. Sorry. :-( Maybe you could point specifically to where long file paths are handled?

[JFLarvoire]

A bit of history:
The WIN32 API was created for Windows 95. In Windows 95, the FAT32 file system was limited to 260 characters paths. When they introduced the NTFS file system in NT, even though this file system had no such limitation anymore, for backwards compatibility reasons, they kept that 260-character limitation in the WIN32 APIs. (Educated guess: Some old version of Word broke due to fixed size buffers overflowing when longer pathnames were encountered?)
Still, to allow long-path-aware programs to use pathname longer than that, they documented that these programs should prefix pathnames with the special string "\\?\".

This is a real problem: I have many systems at work with deep directory trees, with paths longer than 260 characters. Most publicly available WIN32 ports of Unix tools (like find or ag) break on these systems. (And as an aside, they usually also break on pathnames containing Unicode characters > \u00FF; And misbehave if there are symbolic links.)

My MsvcLibX library redefines open(), fopen(), exec(), etc, and prepends that special "\\?\" string to all pathnames it receives, before passing it on the Windows' CreateFile(). It also defines Standard C library directory access routines missing in MS libc.
I use it for all my system management tools, and this ensures they work reliably on all our systems at work. Including the ones with deep directory trees. Including the ones localized in Russian or Chinese. Including the new ones with symbolic links. Including the old XP systems that do NOT support the symbolic link APIs. Etc.

Give me a few days: I'll try to build ripgrep with MsvcLibX, and I'll report about the difficulties and benefits.

[BurntSushi]

@JFLarvoire Thanks for the explanation! It sounds like the key problem will be that ripgrep uses Rust's standard library for file system stuff, so you can't "just" hook up fopen/exec/etc. I do believe that ripgrep should be able to handle symbolic links and Unicode characters in paths on Windows.

and prepends that special \\?\ string to all pathnames it receives

Interesting. I wonder if this is something Rust's standard library should be doing. @retep998 thoughts?

[retep998]

MsvcLibX won't fix anything automatically for Rust because Rust doesn't use libc for IO so redefining libc functions is worthless. What would fix things is detecting non-verbatim paths that are over MAX_PATH and automatically converting them into verbatim paths, which would necessitate calling GetFullPathNameW because paths have to be normalized first (naively prepending \\?\ would break on relative paths or paths with .. or . in them). There's already an open issue for that rust-lang/rust#32689

[JFLarvoire]

Well good luck fixing that in Rust's library. This will be really useful. I'm looking forward to getting a ripgrep version for Windows with the fix.
Note that Rust is not alone: I've just started to work with Python, and despite its widespread use, guess what, the Windows version fails to open files in deep paths too!

[BurntSushi]

FWIW, I probably won't be fixing this myself any time soon. If there were a simple way to patch ripgrep for some cases, then I'd be willing to accept a PR for that, even if it doesn't correctly handle every case. But I can't think of anything simple off the top of my head.

[retep998]

Fortunately since Windows 10 version 1607 you can now enable long path support for all paths, not just verbatim paths, as long as you're calling certain W functions. All you need to do is either set a flag in your manifest to enable it for the application or edit your registry to enable it globally. I believe std calls all the right functions, but it may be worth verifying what various crates use.

The allowed functions:

• CopyFile2
• CopyFileExW
• CopyFileW
• CreateDirectoryExW
• CreateDirectoryW
• CreateFile2
• CreateFileW
• CreateHardLinkW
• CreateSymbolicLinkW
• DeleteFileW
• FindFirstFileExW
• FindFirstFileNameW
• FindFirstFileW
• FindFirstStreamW
• FindNextFileNameW
• FindNextFileW
• FindNextStreamW
• GetCompressedFileSizeW
• GetCurrentDirectoryW
• GetFileAttributesExW
• GetFileAttributesW
• GetFinalPathNameByHandleW
• GetFullPathNameW
• GetLongPathNameW
• MoveFileExW
• MoveFileW
• MoveFileWithProgressW
• RemoveDirectoryW
• ReplaceFileW
• SearchPathW
• SetCurrentDirectoryW
• SetFileAttributesW

[JFLarvoire]

@retep998

naively prepending \\?\ would break on relative paths or paths with .. or . in them

Indeed. My subroutine that does it is there, and it handles all the cases you mention plus a few others. (And conversely, if you know about cases I forgot, please tell me!)

Fortunately since Windows 10 version 1607 you can now enable long path support for all paths

This is a workaround for the technical users who know how to enable this option in the registry. For all others who are running Windows 10 with default settings, or Windows 7 or 8 anyway, the \\?\ method is the only one that will work.