proposal: runtime: don't enable process-wide long path support on Windows

[qmuntal]

Proposal Details

Context

The Go runtime and os package currently implements two levels of long path (more than 260 chars) support on Windows:

• Setting an undocumented PEB bit (source) tricking Windows into enabling process-wide support for long paths without following the official way of doing that (documented here).
• In case the previous hack doesn't work because Windows is older than version 1607, then the os package does some path manipulation (source) to prepend the extended-path prefix, which is a documented approach to make Windows APIs support long paths. Note that the current implementation is not complete, as it does not support relative paths.

These are some issues related to long paths on Windows: #41734, #21782, #36375.
I've recently mailed some CLs to improve the path manipulation approach: CL 570995 and CL 574695.

Proposal

We (the Go team at @microsoft) recommend removing this undocumented PEB hack.

The main concern is that long path support might be internally reimplemented and no longer use the PEB bit used by the Go runtime, in which case that bit could be reused for other purposes. If that happens, then the chances of Go corrupting the PEB are high.

This would be a behavior breaking change. Even if we manage to properly support all long path cases in os, that fix won't apply to interactions with the Windows API outside the os package, i.e. using syscall or cgo. Those affected users will have to enable long path support following the official way.

Given that using undocumented Windows APIs might have security implications, I propose not to add a GODEBUG setting to get the old behavior.

Additionally, using undocumented Windows API goes against Microsoft policies, and the Microsoft Go fork has been flagged for doing this PEB long path support hack. We have taken the action to remove it from our fork regardless of the outcome of this proposal. On the other hand, we are working with the Windows team to define and document APIs which will make long path support easier to achieve.

@golang/windows

[alexbrainman]

We (the Go team at @microsoft) recommend removing this undocumented PEB hack.

I am against your proposal. Your proposal will just break existing code. And there are no benefits in your change.

The main concern is that long path support might be internally reimplemented and no longer use the PEB bit used by the Go runtime, in which case that bit could be reused for other purposes. If that happens, then the chances of Go corrupting the PEB are high.

Microsoft have always been pretty good at supporting existing user code. I suspect they will support Go hack until there is better solution than Windows users fiddling with the registry. And if I am wrong, we can remove the hack when it breaks.

... we are working with the Windows team to define and document APIs which will make long path support easier to achieve.

That would be nice. Thank you.

Alex

[qmuntal]

I am against your proposal. Your proposal will just break existing code. And there are no benefits in your change.

Thanks for your feedback. I wouldn't say it has no benefits. Only using documented, scrutinized and well supported features is a major benefit, even more in these days where everything is prone to be used by bad actors to gain a foothold.

Microsoft have always been pretty good at supporting existing user code. I suspect they will support Go hack until there is better solution than Windows users fiddling with the registry. And if I am wrong, we can remove the hack when it breaks.

That might or might not happen. It is a cost-benefi tradeoff for Microsoft. One data point: AFAIK, Go is the only important runtime/framework that does the PEB trick to enable process-wide long path support.

And if I am wrong, we can remove the hack when it breaks.

Fair point, although I would still prefer to do the "right" and future-proof thing.

[rsc]

Go has a history of making sure the right things work. Sometimes that means looking at undocumented APIs until documented ones come along. Is there an official way to enable long paths on a per-process basis on Windows? The "official way" you linked is a global registry key that affects all processes, not just the current process. Enabling for the current process makes sense because the current process can declare "I know how to handle these long paths" without making that claim for everything else on the machine.

Also, old Go binaries changing this bit exist and cannot be deleted from the world. Given the Windows emphasis on backwards compatibility (I still can run x86 binaries I compiled in the 1990s), it seems impossible that this bit could be repurposed in the future anyway.

It seems like the right path forward would be for Windows to add a per-process official way. Then Go could use that when running on newer Windows. Until then, it doesn't seem like it harms much to keep Go programs running on current and older Windows working with long paths.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[qmuntal]

Is there an official way to enable long paths on a per-process basis on Windows? The "official way" you linked is a global registry key that affects all processes, not just the current process.

The global registry key is not sufficient to enable long path support on all processes, but it is a required setting. When that registry key is set, the process will support long paths only if it has an application manifest that includes the longPathAware setting (all these settings are documented here).

It seems like the right path forward would be for Windows to add a per-process official way.

There is already a per-process official way to do so: registry key + application manifest. I doubt that Win32 will ever provide an API to enable long path support from user code, as it will be an incomplete solution which wouldn't apply to code that runs before that call, even if it's in the executable entry point. That includes:

• The Windows loader wouldn't be able to load statically imported DLLs whose path are longer than MAX_PATH.
• Code in DLLMain from DLLs that load before the executable entry point wouldn't be able to use long path.

This is why the Windows team decided to implement this feature via application manifest.

Then Go could use that when running on newer Windows. Until then, it doesn't seem like it harms much to keep Go programs running on current and older Windows working with long paths.

Good point. We should then revisit #17835, which proposes adding a default hardcoded application manifest to Go binaries.

[rsc]

We want Go binaries to work out of the box with long paths on unmodified Windows systems. Are you saying that's possible using application manifests?

If the Go binaries add a note to an application manifest, does that override the registry key? Does it work on default systems where the registry key has not been modified from what Windows sets it during installation?

If it is possible to arrange for Go binaries to work with long paths out of the box with no other configuration needed on the part of the user (or system adminstrator) using some Microsoft-approved method, that's fine: we can switch to that. But it needs to work as well as the current bit flipping (meaning in all the cases where the bit flip works today).

[qmuntal]

Thanks for the clarity @rsc. Let's put this on hold until I return from paternity leave this September.

[rsc]

Placed on hold.
— rsc for the proposal review group

[qmuntal]

Removing the Proposal-Hold label. I've retaken the conversations with the Windows team to see how we move these forward.

[aclements]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.

[aclements]

@qmuntal , any updates from your discussions with the Windows team?

[qmuntal]

I don't have any relevant update. The Windows team doesn't plan to add any other way to enable long path support in the near future. I won't retract this proposal, though, I still think we shouldn't enable long path support using an undocumented method.

[ianlancetaylor]

I don't understand why Windows requires both a registry key and an application manifest entry to get long path support. Why is it not sufficient to just have an application manifest?

[qmuntal]

MAX_PATH is unfortunately hardcoded in too many places, making a lot of common/popular Windows applications not working well with long path.

Long paths are disabled by default to avoid applications creating files that can't be opened by the rest of the ecosystem.