TerminalControl: Support MinGW path translation style (C:\ -> C:/)

[abutcher-gh]

Summary of the Pull Request

Support drag-n-drop path translation in the style used by MinGW programs. In particular for usage with shells like ash from busybox (https://frippery.org/busybox/).

Detailed Description of the Pull Request / Additional comments

Provides a new option "mingw" for "pathTranslationStyle".
Shown as "MinGW" with translation documented as (C:\ -> C:/) in the UI.
As per the other modes, this translates \ to / but stops there. There is no prefix/drive translation.

Validation Steps Performed

Run using busybox ash shell. Dragged directories and files from both local disks and network shares onto terminal. All were appropriately single quoted and had their backslashes replaced with forward slashes. They were directly usable by the ash shell.

Language files containing the other options have been updated to include the new one.

PR Checklist

• Closes #xxx
• Tests added/passed
• Documentation updated
  • Docs PR #849
• Schema updated (if necessary)

[abutcher-gh]

@microsoft-github-policy-service agree

[lhecker]

This is entirely optional, but you could merge this with the following if condition if you wanted to:

if (translationStyle != PathTranslationStyle::MinGW && fullPath.size() >= 2 && fullPath.at(1) == L':')

[abutcher-gh]

Sorry for being late in getting back to this.

I thought it was clearer to early-out for the MinGW case to make it obvious that no further processing is required rather than combining the logic with the existing drive handling condition. Readers then know there will be no additional MinGW code further on in the function.

[DHowett]

Thanks so much for doing this! I'll hold it for a little bit in case you take Leonard's suggestion, but we're otherwise clear to merge :)

[DHowett]

/apz run

[DHowett]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[abutcher-gh]

@DHowett: This was merged and closed (along with the docs PR) but I didn't see it in a release/prerelease note yet. There is also a "Needs-Tag-Fix" label attached which I'm not sure about.

jiyuz comment on #18807 is also releavnt.

[DHowett]

Hey, I'm sorry about that. We skipped a whole release cycle to focus on fixing bugs for the current Preview release that is expected to become Stable. :)

[avih]

(I'm an occasional contributor to busybox-w32 and familiar with its codebase, but can't say I know it completely inside out)

For future reference, busybox-w32 works fine with either forward or back slashes in paths, and internally it doesn't do any path translations or modification (with few exceptions like /dev/null, /bin/foo etc). I.e. whatever value the argument happens to have at the command line is exactly what busybox-w32 tries to use with windows APIs.

It's windows itself which supports forward slashes instead of backslashes, and busybox-w32 relies on this fact to expose some values with forward slashes for convenience and compatibility with portable scripts (like most env vars by default - but not COMSPEC because cmd.exe dislikes it, and the value of $PWD, and possibly few more).

So the only issue with "no path translation" in busybox shell in terminal, compared to cmd.exe in terminal, is that the quoting rules are different.

busybox ash is a posix shell, so it has posix sh quoting rules, like unix/cygwin/msys2/etc.

E.g. cat < 'c:\foo\bar' > 'c:\single-'\''-quote' is completely fine in busybox shell.

So the required-for-busybox-shell thing which this new MinGW mode does is the posix shell quoting.

But the main thing it advertises - the back-to-forward shash conversion - is not something busybox-w32 requires or even cares for.

EDIT/ADDITION:
And if, for some reason, something doesn't like the forward slashes - like trying to drag cmd.exe to the busybox shell and then pressing ENTER to run it - then it would fail due to this slash translation by terminal, but would not have failed with only posix sh quoting.

FYI.

[abutcher-gh]

@avih You are right, the quoting is the significant part from the point of view of shell arguments and that is provided by the previously existing posix translation styles. This one just affects how the drive spec is changed (or not changed in this case). There was no existing path translation style that provided the C: drive spec form. I understand that Windows is fine with forward slashes in general (except in certain apps/contexts), and that MinGW and Busybox are (mostly) fine with either path style as a result. The issue with drag-and-drop from explorer was that, by default, it would generate backslashes without hard quoting. These become escape characters and require the user to manually quote the front and back of the path with ' ' to have it make sense. The existing translation modes dropped the colon and potentially added /cygdrive or /mnt prefixes, so would need similar editing. I wanted a mode that would preserve the Windows form but prefer forward slashes.

The main reason for the mode was for convenience to prevent the need for manual editing and to allow a dropped path to be emitted in the forward slash style so that it could be used with things like tab-completion in ash without having to cursor around and fix up the backslashes. Try tab completion with a backslash path in ash (whether hard quoted or not). It doesn't work. It does work with the same path with forward slashes.

Maybe the name for the style is wrong, or I should have been not being so keen to call out Busybox specifically in the PR summary. But it was for usage in Busybox's ash shell that was my motivating use case (as I tend to use that over any other shell when in Windows), so that's why I mentioned it in the PR. Regarding the name: It was meant to reflect the fact that a GNU system would use forward slashes and that would be the "preferred" path style for MinGW apps (even though they would likely support the backslash alternatives just fine).

I had not considered the dropping of paths to be passed to a program that only accepted backslashes as I generally don't run such programs.

Thanks for pointing out the issue with cmd.exe though. That is truly strange and I wouldn't have expected that (even from cmd.exe). In fact, I'm not actually sure it's cmd.exe that's doing it. The same hard quoted command line 'C:/Windows/System32/cmd.exe' running under msys bash works as expected. Whatever it is, it seems that it is parsing the entire command line and picking out distinct tokens /c md .exe from within the program path C:/Windows/System32/cmd.exe and interpreting that as a request to run mkdir with .exe as a directory name then exit! If it is cmd.exe doing that---I know it supports spaceless switches so it could be---I would consider that a bug with cmd.exe itself that it would look to parse its own exe path as a command line to execute within it. I think that's a special case here though, and not something I would expect for someone using this mode. But I take the point on the general issue of passing to programs only accepting backslash paths. That would occur for any of the other POSIX modes though too; they would need manual tweaks in that case.

Perhaps we do want a POSIX non-interpolating quote mode though which just does the ' ' quoting and no path translation. But then the option pathTranslationStyle may need to change to something more generic or just accept that there's a translation mode that just quotes and doesn't actually translate any of the chars.

The mode's drop down in the settings UI should probably be updated to be clear on the fact that all the modes will end up quoting the result rather than just performing the translation. Then, if we did have a POSIX hard quote mode it would make it more obvious; C:\ -> 'C:\'.

[avih]

The issue with drag-and-drop from explorer was that, by default, it would generate backslashes without hard quoting. These become escape characters and require the user to manually quote the front and back of the path with ' ' to have it make sense.

Yes. I didn't say it's not useful for busybox sh. It is useful. "None" does not work at the shell because it's not quoted correctly, and the other translations (WSL/cygwin/msys2) are quoted correctly, but they also would not work for busybox-w32 (or MinGW), or, more specifically, they wouldn't work for Windows itself.

Because busybox-w32 doesn't do any internal paths/arguments/slashes conversions, it's up to Windows itself (and the application which gets executed) to decide what to do with forward slashes in argv or lpCommandLine in general (the argument to CreateProcessW). Usually it's fine, but sometimes it's not fine.

For instance, It's fine for Notepad.exe. This command would work fine in cmd.exe shell (and busybox-w32 shell too), assuming the dir foo exists inside the current dir:

c:/windows/notepad.exe foo/bar.txt

But these do not work in cmd.exe shell, independently (and also not in busybox-w32 shell, for the same reasons):

explorer c:/windows
c:/windows/system32/cmd.exe

(in the first example the user might expect to open explorer with C:\Windows in view, and in the second example the user might expect to start a new cmd.exe shell)

Explorer will get executed, but it expects its argument to use backslashes, so it will not use the c:/windows argument as expected.

cmd.exe will also run, but the newly-executed cmd.exe parses the command line in such a "special" way, that it will actually identify it as a request to create a directory .exe in the current dir - and will actually do it(!). It appears to end up parsing it as if it was cmd /c md .exe. Don't ask. just don't use cmd.exe.

So obviously this new "MinGW" path translation would work for the first case, but wouldn't work for the other two cases in busybox-w32, and the last case will actually be an intrussive failure - it WILL "succeed" and create a .exe dir (or complain it already exists, because you tried it again after the 1st time didn't seem to do anything).

Similarly, there's nothing in current MinGW (MinGW-w64 - which is the headers and a way to build a compiler and tools) which expects or cares about forward slashes, or which does any path translation at any stage at all. MinGW currently also does not include any shell, so the (posix) shell quoting rules are completely irrelevant to "MinGW".

I wanted a mode that would preserve the Windows form but prefer forward slashes.

Then you should have invented some name for it, because there's no system which I'm aware of which requires or expects this form, and I do know of systems which will sometimes do unexpected things with it (Windows/MinGW/Busybox-w32). Maybe you could call it after yourself ;)

Joking aside, to be fair, the landscape of unix-like things for windows is complex, and its history too, and it's easy to confuse terms, behaviors, and requirements between the various options.

So I think the correct thing to do here, assuming the goal is, maybe among others, correctness and usability in busybox-w32 shell:

• Change it to only do posix sh quoting.
• Choose some other name. Maybe POSIX quote (C:\ -> 'C:\') or sh quote (C:\ -> 'C:\').

Do feel free to stop reading here.

The rest is a (pretty big) overview and some history lesson, which might help get a clearer picture of the terminology and requirements. Yeah, I got carried away, but I think it's a good overview.

Historical and current overview of some unix-like things for windows

Click to expand, at your own risk

Here are few things it's worth being aware of, in the context of dragging a path to a shell inside Terminal, and assuming a command takes the form of:

COMMAND [PATH|STRING]...  [REDIRECTIONS]

I.e. COMMAND is the thing to execute, and it takes zero or more independent arguments, where each one is either some arbitrary STRING (like in echo "hello world"), or a PATH which should resolve to some entry in the filesystem (like gcc path/to/file.c), and REDIRECTIONS which may or may not include PATH (e.g. echo foo >/bar/baz or echo foo >&2, also, they can be anywhere at the line - not only the end).

Each STRING/PATH argument, and redirection PATH, may or may not require quoting - in order for the shell to identify its value correctly.

The main path forms are:

• POSIX: root is /, path-component-separator is /, windows drives are in some POSIX path relative to root.
• Windows: the native Windows paths form (\ separator, with or without <drive-letter>: prefix, or UNC).

Cygwin:

• Userland is largely Gnu (gnu ls, gnu make, etc).
• Compiler (gcc) which compiles native-cygwin applications (depend on the posix/cygwin dll).
• Native paths (including ENV vars): POSIX.
• Windows drives: /cygdrive/<drive-letter>/ (not sure about UNC).
• Shell quoting: typically posix (bash by default).
• Command: POSIX paths in COMMAND and PATH arguments.
• Internal automatic path translation: only of COMMAND, and only if it resolves to a Windows-native executable (so /cygdrive/c/windows/system32/cmd.exe [...] would work correctly).
• PATH arguments to windows apps: the user should use cygpath if they want to convert them to windows-native paths (e.g. explorer "$(cygpath -w /cygdrive/c/windows)").
• Internal automatic ENV translation: I think only of a specific set of env vars, from windows-native to posix-paths if a cygwin binary is executed by a windows binary, and the other way around when a cygwin binary (like bash) executes a windows-native binary.

Old/original MinGW (my knowledge is limited, so some details might be incorrect). Its porpose was to allow using gnu compiler to build native win32 applications.

Old/original mingw consists of two parts:

• MinGW: "minimalist gnu (compiler) for windows" - a set of patches for gcc/binutils, and a set of win32 headers, which, together, allows gcc to compile native win32 executables. I believe this MinGW part also includes win32-native gcc/binutils which are enough to compile native win32 apps using gcc, from cmd.exe (where it takes native windows input paths, and the resulting binaries also don't do any path translations). I think it also possible to build cross-gcc/binutils using these patches/headers, e.g. on linux.
• msys: "Minimal (posix) system" - shell and standard posix utilities, which are a useful in conjunction with MinGW - for various build tasks etc. This is a fork of cygwin (same mintty as terminal, same bash as shell and same shell quoting, same generally-gnu userland, and "msys2/cygwin" native gcc too - if needed), but with 2 main changes compared to upstream cygwin (that I know of):
  • No /cygdrive. Instead, windows drives are accessible as e.g. /c/foo. This is a hack and not proper mounting, e.g. ls -al / does not show the drives, but ls /c/ shows the files in the c:\ drive.
  • When running a windows-native executable (like the MinGW gcc): auto-detect and auto-convert PATH arguments from msys/POSIX/cygwin form to windows-native forms. E.g. explorer /c/windows should work, but, more importantly, gcc /c/file.c would work too (that's native-win32 gcc from the MinGW part).
  • This auto-translation can cause some issues, for instance cmd /c echo hello at the the msys env would be executed as cmd c:/ echo hello, which is obviously not as intended. Workarounds do exist.

Old mingw/msys have not been maintained in any meaningful way for probably more than a decade now, and are rarely used today. Instead, today we have these:

Cygwin is still actively maintained, and keeps the same principles detailed above.

MinGW-w64:

• A fork/evolution of the MinGW part of old-mingw.
• Same goal and methods (patches, headers, etc to alow building a compiler which targets win32).
• Enhanced (newer headers, supports x64 too, UCRT, clang, cross compile applications/toolchain, etc).
• win32 binaries which are compiled with MinGW-w64 toolchain still don't do any path translation.
• Does not distribute (toolchain) binaries. It's up to someone else to build the toolchain for their needs. Many linux/BSD distributions do package a mingw-w64 toolchain. cygwin and msys2 do too.
• Does NOT include any shell. It does need a shell to build the toolchain, but it's up to the user to find one.

MSYS2 (I don't follow it closely, feel free to correct inaccuracies):

• A fork/evolution of the msys part of old-mingw, still (default) bash shell with posix sh quoting.
• Enhanced (follows cygwin upstream closely, pacman package manager, with many cutting-edge packages).
• MinGW (w64) is available as package (in various forms - arch, gcc/clang, msvcrt/ucrt, etc).
• When launching one of the MinGW envs, it's equivalent to the old mingw/msys, but everything is better.
• Still maintains the old-msys hacks - /c/foo should be used instead of /cygdrive/c/foo, and auto-detection/guess/translation of anything which looks like a path when executing a windows executable:
  • cmd /c echo hello still executes cmd c:/ echo hello and doesn't work.
  • Same as any other windows-style switches, like clip '/?' (where /? is translated to some absolute path), and many more.
  • Workarounds do still exist.
• Used by other projects, likely most notably it's part of git-for-windows as the posix userland and shell.

Busybox (upsrteam):

• Single small binary (typically less than 1M) which incudes hundreds of posix and linux userland utilities and tools ("applets"). Many times used as a full linux system on embedded systems - like routers, but also in other linux distros. Maybe most notable is in Alpine Linux (which is used a lot for docker images because it's tiny - in part due to the busybox userland).
• Utilities/applets are not gnu-based in any way, and maintained in-house. Most utilities are developed from scratch, though some are imported from elsewhere and adapted. Mainly they follow posix, but sometimes add gnu or bsd flags/features too, beyond the posix requiements.
• Posix shell - 'ash': maintained friendly fork of dash - "Debian Almquist SHell" (Debian's default /bin/sh), which is a friendly fork of NetBSD sh, called "Almquist shell" for its original author name. To this day, NetBSD sh, FreeBSD sh, dash, and busybox-ash are all siblings-descendants of the original ash, evolving closely together and follow POSIX, generally support the same set of features (less than bash), and also perform similarly (better than bash).
• Busybox ash does support some non-POSIX bash features which are not in dash, but it's not supposed to support full bashism.

Busybox-w32

• A friendly fork of upstream busybox to run on Windows, which follows upstream closely, and which includes many/most applets (about 170). Some applets are disabled due to porting issues or due to being irrelevant on windows, and some are added for convenience (like iconv).
• Does NOT emulate posix filesystem, and does not perform any paths translations in any applet (with few exceptions like /dev/null being mapped to NUL, /bin/sh maps to the ash applet, etc). Any value which is supposed to be a path is left to the windows API to interpret it.
• Same ash shell as upstream busybox (ported where needed), same posix sh quoting as upstream.
  • After the shell resolved the command elements values (i.e. according to posix sh rules), the resulting COMMAND, STRING|PATH args, and PATH redirections are used unmodified with windows APIs.
  • For convenience and compatibility with some scripts, some paths are converted to forward slashes in shell:
    • The internal shell variable $PWD (current working directory).
    • Most environment variables. This can be disabled using the -X shell option (e.g. sh -X -l for a login shell with unmodified slashes in env vars).
    • Regardless of -X shell option, paths and arguments in shell commands are still used unmodified with the windows APIs - without any automatic translations.