panic: runtime error: slice bounds out of range

[nyarly]

This seems to be the same as #171, but based on the most recent release.

What version of dep are you using (dep version)?

v0.3.1

What dep command did you run?

dep init

panic: runtime error: slice bounds out of range

goroutine 20 [running]:
github.com/golang/dep/internal/gps.(*gitSource).listVersions(0xc420228000, 0xb18980, 0xc42022c000, 0xc42019b7a0, 0x410bd8, 0x10, 0x873ea0, 0xc42019b701)
        /home/travis/gopath/src/github.com/golang/dep/internal/gps/vcs_source.go:224 +0x1082
github.com/golang/dep/internal/gps.maybeGitSource.try.func1(0xb18980, 0xc42022c000, 0xb18680, 0xc4200653c0)
        /home/travis/gopath/src/github.com/golang/dep/internal/gps/maybe_source.go:94 +0x71
github.com/golang/dep/internal/gps.(*supervisor).do(0xc420067080, 0xb186c0, 0xc420012278, 0x8de8f5, 0xc, 0x1, 0xc42019b8f0, 0xc420226000, 0x0)
        /home/travis/gopath/src/github.com/golang/dep/internal/gps/source_manager.go:597 +0xd5
github.com/golang/dep/internal/gps.maybeGitSource.try(0xc4201b6500, 0xb186c0, 0xc420012278, 0xc42015af00, 0x1b, 0xb1b400, 0xc420208040, 0xc420067080, 0x0, 0x0, ...)
        /home/travis/gopath/src/github.com/golang/dep/internal/gps/maybe_source.go:92 +0x1fe
github.com/golang/dep/internal/gps.maybeSources.try(0xc42018a580, 0x4, 0x4, 0xb186c0, 0xc420012278, 0xc42015af00, 0x1b, 0xb1b400, 0xc420208040, 0xc420067080, ...)
        /home/travis/gopath/src/github.com/golang/dep/internal/gps/maybe_source.go:46 +0x104
github.com/golang/dep/internal/gps.(*maybeSources).try(0xc4201b03a0, 0xb186c0, 0xc420012278, 0xc42015af00, 0x1b, 0xb1b400, 0xc420208040, 0xc420067080, 0x30, 0x30, ...)
        <autogenerated>:1 +0xbb
github.com/golang/dep/internal/gps.(*sourceGateway).require(0xc420218000, 0xb186c0, 0xc420012278, 0x1, 0x40, 0x0, 0x0)
        /home/travis/gopath/src/github.com/golang/dep/internal/gps/source.go:569 +0x210
github.com/golang/dep/internal/gps.(*sourceGateway).sourceURL(0xc420218000, 0xb186c0, 0xc420012278, 0x0, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/golang/dep/internal/gps/source.go:537 +0xb5
github.com/golang/dep/internal/gps.(*sourceCoordinator).getSourceGatewayFor(0xc42016fab0, 0xb186c0, 0xc420012278, 0xc4201a8180, 0x1c, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/golang/dep/internal/gps/source.go:205 +0x9f7
github.com/golang/dep/internal/gps.(*SourceMgr).SyncSourceFor(0xc4200670e0, 0xc4201a8180, 0x1c, 0x0, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/golang/dep/internal/gps/source_manager.go:453 +0x82
github.com/golang/dep/internal/gps.(*bridge).SyncSourceFor(0xc42018ce10, 0xc4201a8180, 0x1c, 0x0, 0x0, 0xc4201dc000, 0x3)
        /home/travis/gopath/src/github.com/golang/dep/internal/gps/bridge.go:216 +0x5b
created by github.com/golang/dep/internal/gps.(*solver).selectRoot
        /home/travis/gopath/src/github.com/golang/dep/internal/gps/solver.go:601 +0x614

What did you expect to see?

Not that - I've never used dep on one of my own projects before, but roughly "dependencies resolved!" or something like.

What did you see instead?

The above panic.

[jmank88]

It looks like an assumption that git ls-remote returns lines longer than 41 (commit id first) is not holding:
https://github.com/golang/dep/blob/v0.3.1/internal/gps/vcs_source.go#L224

[nyarly]

If I run git ls-remote I get:

SSH...
From git@github.com:nyarly/versiontool
5e9f87d5d1e5cf45f704264de96b3b6fcdf67b23        HEAD
5e9f87d5d1e5cf45f704264de96b3b6fcdf67b23        refs/heads/master

The leading "SSH..." is a reminder to myself that I will need to activate my U2F token.

[jmank88]

What is the output if you pass the repo explicitly?
git ls-remote git@github.com:nyarly/versiontool.git

Can you elaborate on how you set the SSH... reminder text?

[ean]

I was hit by this today as well, I added some debug output to the top of the listVersions loop to see the lines that it tried to parse, and the cause seems to be a warning from git that causes my panic:
Line: warning: invalid credential line: get
panic: runtime error: slice bounds out of range

goroutine 50 [running]:
github.com/golang/dep/internal/gps.(*gitSource).listVersions(0xc42048a3a0, 0xb65f40, 0xc420308320, 0xc4200bd3b8, 0x4123e8, 0x10, 0x8a4dc0, 0xc4200bd301)
/go/src/github.com/golang/dep/internal/gps/vcs_source.go:225 +0x129d
github.com/golang/dep/internal/gps.maybeGitSource.try.func1(0xb65f40, 0xc420308320, 0xb65c40, 0xc42026e080)
/go/src/github.com/golang/dep/internal/gps/maybe_source.go:97 +0x71
github.com/golang/dep/internal/gps.(*supervisor).do(0xc42027e1e0, 0xb65c40, 0xc42031a780, 0x9144f5, 0xc, 0x1, 0xc4200bd508, 0xc42028e0a0, 0x0)
/go/src/github.com/golang/dep/internal/gps/source_manager.go:615 +0xd5
github.com/golang/dep/internal/gps.maybeGitSource.try(0xc420096c80, 0xb65c40, 0xc42031a780, 0xc420277880, 0xb, 0xb68960, 0xc420374240, 0xc42027e1e0, 0xc420306400, 0x0, ...)
/go/src/github.com/golang/dep/internal/gps/maybe_source.go:95 +0x1fe
github.com/golang/dep/internal/gps.maybeSources.try(0xc420096d80, 0x7, 0x8, 0xb65c40, 0xc42031a780, 0xc420277880, 0xb, 0xb68960, 0xc420374240, 0xc42027e1e0, ...)
/go/src/github.com/golang/dep/internal/gps/maybe_source.go:45 +0x2b9
github.com/golang/dep/internal/gps.(*maybeSources).try(0xc42024a140, 0xb65c40, 0xc42031a780, 0xc420277880, 0xb, 0xb68960, 0xc420374240, 0xc42027e1e0, 0x30, 0x30, ...)
:1 +0xbb
github.com/golang/dep/internal/gps.(*sourceGateway).require(0xc420061c20, 0xb65c40, 0xc42031a780, 0x1, 0x40, 0x0, 0x0)
/go/src/github.com/golang/dep/internal/gps/source.go:569 +0x210
github.com/golang/dep/internal/gps.(*sourceGateway).sourceURL(0xc420061c20, 0xb65c40, 0xc42031a780, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/golang/dep/internal/gps/source.go:537 +0xb5
github.com/golang/dep/internal/gps.(*sourceCoordinator).getSourceGatewayFor(0xc4202a44d0, 0xb65c40, 0xc42031a780, 0xc420023800, 0x30, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/golang/dep/internal/gps/source.go:205 +0x9f7
github.com/golang/dep/internal/gps.(*SourceMgr).ExportProject(0xc42027e240, 0xb65c40, 0xc42031a780, 0xc420023800, 0x30, 0x0, 0x0, 0xb68660, 0xc42024af80, 0xc420302780, ...)
/go/src/github.com/golang/dep/internal/gps/source_manager.go:474 +0x84
github.com/golang/dep/internal/gps.WriteDepTree.func1.1(0xc42035c060, 0xb65c40, 0xc42031a780, 0xc420023800, 0x30, 0x0, 0x0, 0xb689e0, 0xc4202a3040, 0xc420023980, ...)
/go/src/github.com/golang/dep/internal/gps/result.go:95 +0x382
github.com/golang/dep/internal/gps.WriteDepTree.func1(0x8, 0x932330)
/go/src/github.com/golang/dep/internal/gps/result.go:110 +0xe5
github.com/golang/dep/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc42031a7c0, 0xc4203404e0)
/go/src/github.com/golang/dep/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57
created by github.com/golang/dep/vendor/golang.org/x/sync/errgroup.(*Group).Go
/go/src/github.com/golang/dep/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66

it turned out I had a bad "credential.helper" field set in my git config which was the cause of it.

[nyarly]

@jmank88 Output of git ls-remote git@github.com:nyarly/versiontool.git

SSH...
5e9f87d5d1e5cf45f704264de96b3b6fcdf67b23        HEAD
5e9f87d5d1e5cf45f704264de96b3b6fcdf67b23        refs/heads/master

 ⮀ git config --get core.sshcommand
proud-ssh

 ⮀ cat $(which proud-ssh)
#!/bin/sh

echo "SSH..." >&2
exec ssh "$@"

[jmank88]

I'm not sure of the best way to handle these cases. A dirty solution would be to ignore anything that doesn't look like a commit id (at least don't panic!). Is there a git flag to suppress output we don't want?

[ean]

I think dep is handling this wrong, it parses everything on stderr and stdout, but the git references are only written to stdout, everything on stderr is usually warnings related.

Eiriks-MacBook-Pro:virtual-sensor eirik$ git ls-remote git@github.com:nyarly/versiontool.git 2>/dev/null
5e9f87d5d1e5cf45f704264de96b3b6fcdf67b23 HEAD
5e9f87d5d1e5cf45f704264de96b3b6fcdf67b23 refs/heads/master
Eiriks-MacBook-Pro:virtual-sensor eirik$ git ls-remote git@github.com:nyarly/versiontool.git >/dev/null
SSH...
Eiriks-MacBook-Pro:virtual-sensor eirik$ git config --get core.sshcommand
proud-ssh

dep should only parse stdout, stderr could be forwarded to the users stderr to show the messages, but dep shold not care about them.

[sdboyer]

yeah, it's probably preferable in this case to just grab stdout, not stderr. now, it's not mutually exclusive, but we really ought to just to improve this parser in general - there's no reason we shouldn't be able to just discard lines that don't fit with valid patterns.

i outlined what improved parsing logic would look like over in #1160 (comment) .

[jmank88]

Is there any risk in implicitly assuming that a line which parses is legitimate? It seems unlikely that it would happen on accident (given the strict format), but who knows what people are injecting.

[sdboyer]

@jmank88 i don't think so.

even if we're reading both stdout and stderr, i think the attack vector there requires placing a compromised git binary onto a user's system, then using it to inject output into git ls-remote that either crashes dep (e.g. with an endless stream of bytes without an \n, causing dep to OOM because we read into a buffer rather than streaming) or manipulates it into thinking that certain rev/ref combos exist that actually do not, or some that do actually exist do not. the former is a dead-end attack afaict - there's no opportunity here for privilege escalation, or anything. the latter is possible, but is incapable of spreading to any other system that doesn't have a similarly compromised git binary, as git itself will catch problems with the reported version list, as well as any inconsistencies in the code on disk (and the our eventual verification mechanisms, #121, will doubly guarantee on the latter).

compromising the server side, or a MITM, is a tad more interesting. but even then, git's doing integrity checks all the way up and down via its Merkle DAG. if incorrect revs are reported, we're going to know. the only real "attack" there that i can see being possible is pointing a ref to a rev with compromised code in it - but that's now out of the domain of git ls-remote, which is just accurately reporting a list of versions, one of which happens to be compromised.

[dim13]

Confirm. Run into it with VisualHostKey enabled in ssh_config.

[bastianccm]

Hi,

we have stumbled upon this error because an empty id_rsa.pub file was in the users ~/.ssh folder, thus git ls-remote showed an error message:

project git:(master) ✗ git ls-remote git+ssh://git@gitlab/go/project.git
key_load_public: invalid format
aaaaaaaa21b0a819a72ae4c32edac782aaaaaaaa    HEAD
aaaaaaaa21b0a819a72ae4c32edac782aaaaaaaa    refs/heads/master
aaaaaaaaa3900b65d65cc298b1ece9adaaaaaaaa    refs/heads/branch1

The line key_load_public: invalid format let dep error with the same panic.

Creating an invalid public-key file in your ~/.ssh folder triggers this issue, and can be used for debugging/reproducing the error.