"git.company.com:1234/group/project.git" is not a valid import path

[fabulous-gopher]

From @liuliuhit on April 19, 2017 3:22

I got this error message because of normalizeURI() function.

Notice deduce.go line 75
pathvld = regexp.MustCompile(`^([A-Za-z0-9-]+)(.[A-Za-z0-9-]+)+(/[A-Za-z0-9-_.~]+)*$`)

pathvld seems not support the uri pattern with port.

thanks😀

Copied from original issue: sdboyer/gps#219

[tobert]

One of my deps is on an internal Atlassian Stash/Bitbucket server that requires a port number. I'm hitting this error too with the following dependency.

[[dependencies]]
branch = "master"
name = "source.mycompany.com/core/thing"
source = "ssh://source.mycompany.com:7999/core/thing"

[sdboyer]

@liuliuhit - sorry for the delayed response. The right fix for your case should be to declare a source field in Gopkg.toml, as @tobert has done, and then use an import path in the code that doesn't contain the port. As the spec notes, it's valid for a compiler (which dep is not in general, but in this case is behaving as one) to reject import paths containing :. lmk if that fixes it for you 😄

@tobert so, I suspect you have a separate issue - "failed to deduce repository root." these problems are subtly different, and unfortunately, yours may be the harder one. dep needs to be able to look at an import path and figure out where the root is, from the string alone, in accordance with go get's rules.

Does the private Stash service allow you to specify the repository extension as part of the path? If so, e.g. "source.mycompany.com/core/thing.git" will work, and will be by far the easiest way to solve this problem.

[tobert]

I tried adding the extension and am getting the same results.

`[tobert@gadget scorebot]$ tail -n 4 Gopkg.toml
[[dependencies]]
branch = "master"
name = "source.mycompany.com/core/thing"
source = "ssh://source.mycompany.com:7999/core/thing.git"
[tobert@gadget scorebot]$ dep ensure -v
Root project is "source.mycompany.com/core/scorebot"
22 transitively valid internal packages
30 external packages imported from 11 projects
(0) ✓ select (root)
(1) ? attempt github.com/alexedwards/scs with 2 pkgs; at least 1 versions to try
(1) try github.com/alexedwards/scs@master
(1) ✓ select github.com/alexedwards/scs@master w/2 pkgs
(2) ? attempt github.com/go-sql-driver/mysql with 1 pkgs; at least 1 versions to try
(2) try github.com/go-sql-driver/mysql@master
(2) ✓ select github.com/go-sql-driver/mysql@master w/1 pkgs
(3) ? attempt github.com/aws/aws-sdk-go with 4 pkgs; at least 1 versions to try
(3) try github.com/aws/aws-sdk-go@master
(3) ✓ select github.com/aws/aws-sdk-go@master w/25 pkgs
(4) ? attempt github.com/go-ini/ini with 1 pkgs; at least 1 versions to try
(4) try github.com/go-ini/ini@v1.27.0
(4) ✓ select github.com/go-ini/ini@v1.27.0 w/1 pkgs
(5) ? attempt github.com/juju/errors with 1 pkgs; at least 1 versions to try
(5) try github.com/juju/errors@master
(5) ✓ select github.com/juju/errors@master w/1 pkgs
(6) ← no more versions of source.mycompany.com/core/thing (from ssh://source.mycompany.com:7999/core/thing.git) to try; begin backtrack
(5) ← backtrack: no more versions of github.com/juju/errors to try
(4) ← backtrack: no more versions of github.com/go-ini/ini to try
(3) ← backtrack: no more versions of github.com/aws/aws-sdk-go to try
(2) ← backtrack: no more versions of github.com/go-sql-driver/mysql to try
(1) ← backtrack: no more versions of github.com/alexedwards/scs to try
✗ solving failed

Solver wall times by segment:
b-list-pkgs: 2.39547546s
b-source-exists: 1.512774821s
b-gmal: 556.415717ms
unselect: 20.108505ms
satisfy: 18.202661ms
select-atom: 15.065399ms
select-root: 1.526935ms
new-atom: 396.445µs
b-deduce-proj-root: 99.138µs
backtrack: 73.226µs
other: 7.639µs

TOTAL: 4.520145946s

ensure Solve(): "source.mycompany.com:7999/core/thing.git" is not a valid import path`

[sdboyer]

ensure Solve(): "source.mycompany.com:7999/core/thing.git" is not a valid import path`

Ah, OK, you didn't say this was the error you were getting 😄

First, to make sure we're clear - you're absolutely certain that source.mycompany.com:7999/core/thing.git does NOT appear in any import statements in your codebase, or the codebase of any of your dependencies?

[tobert]

Right, my code has been using a manually vendored copy of the problematic repo for about a year now.

The import is "source.mycompany.com/core/thing" and I manually vendored it into "$GOPATH/src/source.mycompany.com/core/scorebot/vendor/source.mycompany.com/core/thing".

Having the source line working Godeps.toml will make this a much nicer situation :)

[sdboyer]

I'm working up a program to get some output. However, on looking back over,

[[dependencies]]
branch = "master"
name = "source.mycompany.com/core/thing"
source = "ssh://source.mycompany.com:7999/core/thing.git"

This is not what I meant. I meant this:

[[dependencies]]
branch = "master"
name = "source.mycompany.com/core/thing.git"
source = "ssh://source.mycompany.com:7999/core/thing"

And you'll have to change all your import paths that were pointing to source.mycompany.com/core/thing to point to source.mycompany.com/core/thing.git instead.

[sdboyer]

Assuming the above change still doesn't fix the problem, please compile and run this program. Doesn't matter where you run it from.

[Mmmmiracle]

@sdboyer I'm sorry I forget paste my Gopkg.toml

Ah, I declare source field in Gopkg.toml for internal package, and got "xxx is not a valid import path" error.
[[dependencies]]
branch = "master"
name = "company.com/project"
source = "https://git.company.com:1234/group/project.git"

And I resolve this problem in a hack way for temporary.

Notice github.com/sdboyer/gps, deduce.go line 75:
pathvld = regexp.MustCompile(`^([A-Za-z0-9-]+)(.[A-Za-z0-9-]+)+(/[A-Za-z0-9-_.~]+)*$`)

pathvld pattern used in normalizeURI() function where throws "xxx is not a valid import path" error, so I fix pathvld as follows
pathvld = regexp.MustCompile(`^([A-Za-z0-9-]+)(.[A-Za-z0-9-]+)+(?::(\d{1,5}))?(/[A-Za-z0-9-_.~]+)*$`)

[sdboyer]

@liuliuhit i mean...hacking dep is one option, i guess :P but, now that you've done that, my recommended course of action for you is the same as it was for @tobert - include the vcs extension (e.g. company.com/project.git) in BOTH the import paths in your code and in Gopkg.toml, and root deduction should work fine.

[sdboyer]

Closing this one out - in the small scale, this is a won't fix, as we're within the parameters laid down by the Go spec to treat : as an invalid char in import paths.

[tobert]

Adding the .git extension doesn't work and I haven't had a chance to try your program. I will do so after morning meetings.

@liuliuhit's fix is obviously correct. I get that the Go project doesn't support this use case but it doesn't mean dep can't solve a long-standing issue with 'go get'.

[sdboyer]

Adding the .git extension doesn't work

What error do you get?

I get that the Go project doesn't support this use case but it doesn't mean dep can't solve a long-standing issue with 'go get'.

The end goal for dep is to make it into the toolchain. Decisions like these that depart from current toolchain behavior are of the sort that, while they address a pain point in the immediate term, may add little bits of complexity and edge cases to the eventual move, which is already going to be a tremendously complex process.

In this case, the concern with allowing colons would be the subtle Windows incompatibility it creates, where colons in filenames are not allowed. It's even a little hinky on OSX.

[tobert]

I get the same error regardless of how I modify the source field. I tried the variations you suggested as well as some others.

From what I can tell, the 'dependencies.source' field in Gopkg.toml exists to allow redirecting an import (name) to a fully-specified URL so that imports and filesystem paths do not need to be modified.

I took a look at your gps package and see that it is using the same RE for the import path and source checks and yeah, it wouldn't make sense to allow ports there. Could the regexp be split so that there is an import regexp that strictly conforms to the go specs and then something less strict for sources? I suppose if it's calling through to go get's fetching code it won't work.

It appeared that the source field would scratch this itch, but I understand if it needs to be kept limited to the go specs. I already have to hack around those limitations and can continue to do so.

Thanks!