path/filepath: Walk on very long paths fails on Windows

[sandreas]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.9

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\mediacenter\go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

What did you do?

Executing the sample code on a current windows 10 machine fails with an error - seems that running filepath.Walk on very long paths fails on windows in general.

Sample code, which works on play: https://play.golang.org/p/wMS5gti4SD
But it DOES NOT WORK on windows systems!

What did you expect to see?

correctly walk over..../package.json

What did you see instead?

ERROR ON PATH node_modules\babel-preset-es2015\node_modules\babel-plugin-transform-es2015-block-scoping\node_modules\babel-traverse\node_modules\babel-code-frame\node_modules\chalk\node_modules\strip-ansi\node_modules\ansi-regex\package.json: GetFileAttributesEx node_modules\babel-preset-es2015\node_modules\babel-plugin-transform-es2015-block-scoping\node_modules\babel-traverse\node_modules\babel-code-frame\node_modules\chalk\node_modules\strip-ansi\node_modules\ansi-regex\package.json: Das System kann den angegebenen Pfad nicht finden.

Can anyone confirm this behaviour?

[mvdan]

This seems similar to #20829. Perhaps there's another part of the standard library that needs this fix for long paths.

[odeke-em]

/cc @alexbrainman @hirochachacha

[alexbrainman]

This fails in os.Lstat(path), and path is relative path here. The os.Lstat is using os.fixLongPath, which has been created just for such purpose. But os.fixLongPath does not handle relative paths, and for a good reason - relative paths cannot be longer than 255 characters (see https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath for details). I do not know how to fix this. Perhaps @quentinmit has some suggestions - he wrote os.fixLongPath.

Alex

[sandreas]

I'm not sure, if this is possible, but i think os.Lstat returns the same value for absolute and relative paths. Is that correct?

If so, why not convert all paths to absolute paths (e.g. with filepath.Abs) in os.Lstat on windows before using the syscall?

[alexbrainman]

why not convert all paths to absolute paths (e.g. with filepath.Abs) in os.Lstat on windows before using the syscall?

Maybe we'll do that. I am not sure yet.

Alex

[sandreas]

As a short feedback: I implemented a simple filesystem abstraction based on afero (https://github.com/spf13/afero) and the use filepath.Abs did the job for me.

I used following logic:
If the path is not absolute and it is longer than 200 chars and it is a windows system, apply filepath.Abs before any other operation.

Why 200 chars, not 255?
Because a relative path containing 227 chars led to an error, while less than 200 did not. I don't know the reason but 255 did not work. To exclude unicode errors, i tried to use special chars in filename (äöü€), but the 227 bound was still valid. The slash-count is 14, 255-14 = 241, so diff is also not the amount of slashes of the relative path.

The problem is, that in later testing i noticed, that there errors occurred with some paths lower than 227 chars, so i corrected it down to 200. Would be nice to know the exact bound, but i didn't find out.

The path with less than 255 chars, which caused the first error was:
relative:
node_modules\babel-preset-es2015\node_modules\babel-plugin-transform-es2015-block-scoping\node_modules\babel-traverse\node_modules\babel-code-frame\node_modules\chalk\node_modules\strip-ansi\node_modules\ansi-regex\package.json

absolute:
C:\inetpub\wwwroot\irgendwas_4.0\node_modules\babel-preset-es2015\node_modules\babel-plugin-transform-es2015-block-scoping\node_modules\babel-traverse\node_modules\babel-code-frame\node_modules\chalk\node_modules\strip-ansi\node_modules\ansi-regex\package.json

[quentinmit]

why not convert all paths to absolute paths (e.g. with filepath.Abs) in os.Lstat on windows before using the syscall?

Unfortunately, it's very nontrivial to convert relative paths to absolute paths on Windows. I already went down that road and gave up on trying to make it work. syscall.FullPath, which is what filepath.Abs calls, is also only supported on short paths. What makes this hard is that Windows has a separate working directory on each drive, but getcwd and friends only return one of them. e.g. "d:foo.txt" is a relative path that's relative to whatever the current directory is on the D drive.

If you need to Walk on long paths, you should just call Walk with an absolute path. I believe everything inside Walk should work correctly in that case.

[sandreas]

I already fixed it and for me it is working prepending a \\?\. You said that filepath.Abs only will work for short paths. In my case it does work perfectly with long paths. I'm not saying it is working in every case, but for now i did not find a path (including some node modules directories), that did not work. Code sample (taken from my project https://github.com/sandreas/graft/blob/master/filesystem/osfs_windows.go):

func makeAbsolute(name string) string {
	absolutePath, err := filepath.Abs(name)
	if err == nil {
		if strings.HasPrefix(absolutePath, `\\?\UNC\`) || strings.HasPrefix(absolutePath, `\\?\`) {
			return absolutePath
		}

		if strings.HasPrefix(absolutePath, `\\`) {
			return strings.Replace(absolutePath, `\\`, `\\?\UNC\`, 1)
		}

		return `\\?\` + absolutePath
	}
	return name
}

[alexbrainman]

I already went down that road and gave up on trying to make it work. syscall.FullPath, which is what filepath.Abs calls, is also only supported on short paths.

@quentinmit I just tried to test syscall.FullPath:

c:\Users\Alex>type main.go
package main

import (
        "fmt"
        "log"
        "os"
        "path/filepath"
        "syscall"
)

func makeLongPath() string {
        const file = "abcdefghij1234567890"
        var s string
        for i := 0; i < 13; i++ {
                s = filepath.Join(s, file)
        }
        return s
}

func testPath(path string) {
        fullpath, err := syscall.FullPath(path)
        if err != nil {
                log.Fatal(err)
        }
        fmt.Printf("\nFullPath(%q) is %v\n", path, fullpath)
}

func main() {
        err := os.Chdir(`U:\`)
        if err != nil {
                log.Fatal(err)
        }
        longpath := makeLongPath()
        testPath(`C:` + longpath)
        testPath(`C:\` + longpath)
        testPath(`U:` + longpath)
        testPath(`U:\` + longpath)
        testPath(longpath)
}

c:\Users\Alex>go run main.go

FullPath("C:abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is C:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

FullPath("C:\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is C:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

FullPath("U:abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is U:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

FullPath("U:\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is U:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

FullPath("abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is U:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

c:\Users\Alex>

and it seems to be working. It works on both Windows 10 and Windows XP.

Alex