lib/std/fs/File: enable VT seq support for Windows Console

[KexyBiscuit]

Rationale and changes

• Newer versions of Windows added VT seq support not only in Windows Terminal, but also in the old-fashioned Windows Console (standalone conhost.exe), though not enabled by default.
• Try setting the newer console mode flags provides better experience for Windows Console users.
• lib/std/os/windows/kernel32: add signature for SetConsoleMode
  • From lib/libc/include/any-windows-any/wincon.h#L235
  • See also https://learn.microsoft.com/en-us/windows/console/setconsolemode
  • Also add DISABLE_NEWLINE_AUTO_RETURN constant which will be used by SetConsoleMode in lib/std/os/windows.
• More detailed explanations in code comments.

Examples from BiscuitTin/zig-term-colors

For Windows 10 Enterprise LTSC 2019, the app works as expected in Windows Console. Zig std library checks for (and enables) VT seq support during runtime.

For Windows 10 Enterprise LTSB 2015, Zig std library behaves just like before, no VT seq outputted at runtime.

For Windows 10 Enterprise LTSC 2016, things are a bit more complicated. cmd.exe just works like later Windows versions, but Windows PowerShell 5.1.14393.0 contains a bug, accidentally enabling VT seq support for all applications invoked inside it.

For all other (newer) Windows versions currently Microsoft supports, Windows Console behaves the same as Windows 10 Enterprise LTSC 2019, but Windows Terminal is also available, makes it pointless to guide the user opening Windows Console.

[squeek502]

Nice thorough writeup!

Note, though, that this sort of thing was closed without merging before in #15206:

To make a more general comment about the PR, I'm unsure if setting ENABLE_VIRTUAL_TERMINAL_PROCESSING is something Zig should be doing unconditionally like this. This is similar to #7600 (see also #12400 and #14411) in that it affects all processes run in the console session, not just the current process, so it's not clear cut if attempting to force ENABLE_VIRTUAL_TERMINAL_PROCESSING to be enabled is always the 'right' thing to do. If we can get away with just #15282, it seems like that might be the 'safer' way to go.

Note also that the API in std.io.tty is what's used by Zig for coloring, and it falls back to SetConsoleTextAttribute if File.supportsAnsiEscapeCodes returns false.

So perhaps this logic might be better suited as a helper function in std.io.tty? Something like forceEnableAnsiEscapeCodes(file: File)? And then it can be left up to the user to call that function if they want/need to?

[KexyBiscuit]

To make a more general comment about the PR, I'm unsure if setting ENABLE_VIRTUAL_TERMINAL_PROCESSING is something Zig should be doing unconditionally like this. This is similar to #7600 (see also #12400 and #14411) in that it affects all processes run in the console session, not just the current process, so it's not clear cut if attempting to force ENABLE_VIRTUAL_TERMINAL_PROCESSING to be enabled is always the 'right' thing to do. If we can get away with just #15282, it seems like that might be the 'safer' way to go.

This is exactly what PowerShell/PowerShell#2991 does, application should save original console mode and restore it before exiting.

This pull request currently doesn't implement that, since it was originally a PoC. If save/load approach is acceptable, we are glad to implement it in File and tty logic. After discussion, we think this kind of logic should be implemented by the application or library themselves, rather than in standard library, aligned with Microsoft's original design goal.

No matter what, I think 0d2a328 is safe to merge, since it enables the user to implement save/load logic, or just ignore the risk and enable VT seq support themselves. Should I open a new PR for that?

[KexyBiscuit]

Superseded by #18715, containing only 0d2a328.

[squeek502]

This is similar to #7600 (see also #12400 and #14411) in that it affects all processes run in the console session, not just the current process

I was wrong about this, apologies for not checking my assumptions. I've cherry picked the relevant commit from this PR into #20172, which is a revivial of the spirit of this PR.