Support for private repositories e.g. Github Enterprise

[mikkeloscar]

I didn't see this addressed in any of the docs, so I'll raise the question
here.

Have you considered support for private repositories?

Many of the existing vendor tools does not handle private repositories very
well or at all. godep and govender can sort of handle it because they get
the dependencies from your $GOPATH. But existing tools that fetches directly
from the remote does not work with private repos in my experience (If someone
knows any tools that does work, please let me know :) ).

It would be great if dep could break this trend and work well with private
repositories from the start.

I'm probably not covering all use cases, but these are the use cases I would
like to see solved.

As a developer (consumer) I want to import dependencies from a private Github
Enterprise repository my.ghe.com.
Assuming I can already clone repositories either via https or ssh, I don't
want to also configure the authentication in dep. Configuring it one time
should be enough e.g. in git (or ssh).

As a developer I want to avoid committing the vendor folder, thus my CI/CD
system must be able to pull in all dependencies (also those from a private
repo) when running a build.

As a developer I don't want to encode the type of repository in my source
code e.i. I don't want to specify a vcs extension in my import: `import
"my.ghe.com/repo/package**.git**"

From my point of view this could be as simple as introducing an environment
variable which maps your private repo URL to a certain protocol. E.g.:

GO_DEP_REPO=my.ghe.com=git+ssh:my.ghe2.com=git+https

Which means whenever dep encounters an import path which starts with
my.ghe.com it will automatically choose git+ssh when fetching the repo. And
since I already have configured an ssh key on my machine it should know how to
authenticate.
Similarly when it encounters an import path with my.ghe2.com it
will choose git+https and ask me for username+password or just work if I
have configured what token to use in .gitconfig.

I'm sure someone could think of a better name for the environment variable and
a better format, but I hope you get my point.

[zevdg]

At my company, we solve this exact problem using go's vanity imports. We have something like https://github.com/dominikh/go-vanity which serves up the appropriate protocol depending on various factors and we use the url of the vanity server instead of our ghe url in our import paths.

[freeformz]

This is part of our spec (https://docs.google.com/document/d/1qnmjwfMmvSCDaY4jxPmLAccaaUI5FfySNE90gB0pTKQ/edit#). Search for "alt location".

Does that work, at least basically for your needs?

With that said it doesn't exist atm.

[sdboyer]

First - solving this with vanity imports via an external system, as @zevdg describes, is far easier. (I didn't know about go-vanity - that's awesome.) That said, vanity imports still have some of the same issues as what's described here.

Have you considered support for private repositories?

We didn't talk about it much directly within the committee as it wasn't an immediate necessity, but I've thought about this quite a lot. It's in the top three nasty problems I've grappled with over the last year.

To be clear - these problems are solvable, but they add a ton of new, potentially obnoxious failure modes, so I've been very cautious about adding support for them in gps. I've also been thinking about them for way too long, and have gotten somewhat lost in the forest - help with weighing tradeoffs is very welcome :)

This is kind of a slapdash writeup, but hopefully it covers the basic bases.

The essential problem is portability: we have to be able to deduce the source/repo for all import paths we encounter. If that's only possible by relying on custom rules, then anyone trying to work with your project without those custom rules won't be able to resolve those import paths. The really nasty part, though, is that we're unable to provide users in that situation with useful error messages, because it's impossible to tell the difference between "invalid import path" and "invalid import path because you're missing some custom deduction rules." How could we know such custom rules exist?

If you're working on a private, internal company system, then this sort of portability isn't really a concern. The problem with a feature like this is that it wouldn't just be used for that - how many people would write their own little custom rule for, say, gitlab (for which we don't currently have a built-in rule)?

We could mitigate this by having the custom rules live in the manifest, alongside constraints/ignores/requires/overrides. composer does this, generally to good effect. But, if you're in a company with a bunch of different projects, then each would need to define the same rules in their manifests. (Pattern definition would have to be a root-manifest-only thing, for basically the same reasons as composer gives). If you have to make any changes in how those rules work, you have to make them in all your projects, and it could potentially make it impossible to use older versions of those projects, because your new rules don't work with the old imports.

But all this works well enough in composer, right? Sure. Things are strictly more difficult for Go, though, because this problem of deducing projects/repositories from import paths is one that composer simply doesn't have. Composer is just dealing with the name of a dep and a location from which to retrieve it and its metadata. Because dep tries hard to be truly idiomatic, we still treat the import graph, not the manifest, as the canonical expression of what code is required (c.f. writeup; we follow option 2). So, rather than just naming a dep and where to get it from, we have to specify a pattern that can be used to match import paths that we encounter and map them back to their repository root.

Plus, composer constrains names to have two parts, but we have no such guarantee - launchpad.net/someproject has two, but github.com/golang/dep has three. Variable-length root names make it impossible to designate a general rule that, say, "all custom rules must have two path parts," which would make ensuring that we don't have any conflicting rules much easier.

Also, as one final kick in the pants, because we're ultimately talking about a system to decide what names (import paths) mean, it would have to be incorporated into the input hashing system we use to determine when you need to regenerate your lock. (That's the memo field in your lock.json - this is a very experimental idea that we may drop)

Again, I do think there are reasonable solutions here. But I have yet to figure out a good, balanced compromise that ratchets down control to avoid rendering the entire system unsafe; I've been enmeshed in this problem for a long time, and have lost a lot of the perspective necessary to weigh tradeoffs. Feedback is super welcome :)

[sdboyer]

Oh! I forgot to mention...

All of this ugliness is my own single strongest motivation for creating a registry (#175): we can impose meta-rules on a subset of import paths. For example, imagine this pattern:

    import "p/gopherhoard.com/user/project"

• The leading p indicates it's a registry source type
• The second part is the base domain where the registry is hosted. This would allow the crucial feature of allowing people to host their own private (or public) registries; we might optionally have a shorter alias for the main, public registry.
• There's a fixed number of trailing elements. (We might also just do project name, without namespacing by user).

This solves so many problems.

• There's no more "unknown rule" problem, because there's a meta-pattern followed by all registries
• Root deduction is dead-simple
• The accuracy of input hashing is no longer at risk
• ...all of the other problems that registries solve, independent of this

[zevdg]

Thanks for the background! Glad I'm not the only person thinking about this.

The problem that vanity imports don't solve for us is having a centralized cache of 3rd party libraries. We use artifactory to do this from nearly every other language, but since go uses the import path to uniquely identify both the package and its source url, the existing go tool simply cannot handle changing the source url without also changing the package's identifier. We could re-write import paths to point to artifactory, but we'd have to also do that inside of every 3rd party package we use as well so that we don't end up referring to the same package with by different names.

The alternate urls described in dep ensure -h looks like exactly what we need. I was assuming that those alternate urls would be stored in your manifest.json individually for every dependency. Something like

{
    "dependencies": {
        "github.com/golang/protobuf": {
            "branch": "master",
            "source": "artifactory.corp-intranet.com/github.com/golang/protobuf"
        },
        "github.com/pkg/errors": {
            "branch": "master",
            "source": "artifactory.corp-intranet.com/github.com/pkg/errors"
        },
        "ghe.corp-intranet.com/myteam/lib": {
            "branch": "master"
        }
    }
}

So while that leaves us with a lot of seemingly redundant text in the manifest file, it keeps any complex deduction rules out of it. Anyone would be able to pull down this project's deps from the right place assuming they had access to the intranet.

That alone seems "sufficient" to solve the problem, but it requires a lot of discipline for devs to add the correct alternate location to each and every 3rd party dependency. So on some level, I do want a "custom rule" of some sort, but more generally, I'm imagining some kind of onManifestEdit hook there I can run some arbitrary script AFTER dep updates the manifest but BEFORE dep uses manifest to resolve dependencies and update the lockfile. This hook would have the opportunity to also add more updates to the manifest and if it did, dep would re-read and use the new info in generating the lock file. Conceptually, this would be similar to setting up your editor to run goimports followed by go build on save.

In particular, the hook I'd personally want would be along the lines of (in psedo code)

for each package p:
    if p is 3rd party:
        p.source = "artifactory.corp-intranet.com/"+p.importpath

So that on a properly configured machine, dep ensure github.com/golang/protobuf would inject my alternate source but dep ensure ghe.corp-intranet.com/myteam/lib would not. On an unconfigured machine, everything would still work fine, but one would have to manually specify alternate sources for any new dependencies they add.

I imagine there would be other legit use cases for having an automated edit to the manifest here. I haven't thought too deeply about the implication of allowing a hook like this, but since the manifest is supposed to be editable by humans, never mind other tools, this doesn't seem too dangerous at first glance.

Also, we may have gone very far off topic from the original issue. Maye this should be separated into a separate issue or 2?

[francknouama]

+1

[sdboyer]

The alternate urls described in dep ensure -h looks like exactly what we need. I was assuming that those alternate urls would be stored in your manifest.json individually for every dependency.

So while that leaves us with a lot of seemingly redundant text in the manifest file, it keeps any complex deduction rules out of it. Anyone would be able to pull down this project's deps from the right place assuming they had access to the intranet.

Yep yep!

That alone seems "sufficient" to solve the problem, but it requires a lot of discipline for devs to add the correct alternate location to each and every 3rd party dependency.

Oh for sure - but I'll go two steps stronger.

Step 1: it's not just that it requires discipline for devs. It's that you effectively leave to individual developers what may, to meet the workflow/legal/whatever reqs of your organization, need to be questions of policy.

Step 2: when you explicitly record the source values in manifest.json, you're creating a new vector for conflicts - all of your projects relying on github.com/golang/protobuf, for example, need to agree that they are sourced from artifactory.corp-intranet.com/github.com/golang/protobuf. If any don't agree, it's considered a conflict by the solver - not a version constraint conflict, but a source conflict - because how could it not be? We can't source one thing from two different places. (I described a little more how this class of conflict works in #191 (comment) - the mechanism is the same there)

This creates a potentially nasty situation: let's say your company gets acquired, and the artifactory base URL has to change from artifactory.corp-intranet.com to artifactory.newcorp-intranet.com, because 🐙. All of your projects will have to independently cut over to the new artifactory URL at the same time, roll new releases with that change, and none of them will be able to work with older versions. That's exactly the kind of high-risk change scenario that we should all be seeking to avoid.

(The source property is rife with possible outcomes like this. That's why I don't really like it that much as-is, and would ideally like to see it replaced with a few, more narrowly-defined properties that accomplish the same sort of goals.)

I'm imagining some kind of onManifestEdit hook there I can run some arbitrary script AFTER dep updates the manifest but BEFORE dep uses manifest to resolve dependencies and update the lockfile.

Yes, I've been imagining something like this since some folks described a similar need in Masterminds/glide#372. A script would probably work well; my plan was that it would take either a project root or a URL (so, github.com/golang/protobuf or https://github.com/golang/protobuf) and transform them into the URL(s) that may be used to interact with the source.

If we went that route, it would be transparent to the manifest, and the onus would be on the implementor of the script to ensure that the mirror is properly reflecting the data from upstream. Failure to do that would compromise the portability of the lock outside of the alternate sourcing universe imposed by the script...which is maybe fine in some cases, but has its risks :)

Also, we may have gone very far off topic from the original issue. Maye this should be separated into a separate issue or 2?

Sure - my thinking on these topics is all pretty jumbled up together, so if you can see a way to tease out more specific issues, please do!

[sdboyer]

Just bumping this to see if you've had any more thoughts on separating out some issues, @zevdg 😄

[mikkeloscar]

I'm glad I'm not the only one who have thought about these things :)

At my company, we solve this exact problem using go's vanity imports.

I had this thought maybe 8 months ago but went away from it for reasons I can't quite remember. I think I imagined it would be inconvenient to use a different import url. But looking at it now I see more benefits than trade-offs.

This is part of our spec (https://docs.google.com/document/d/1qnmjwfMmvSCDaY4jxPmLAccaaUI5FfySNE90gB0pTKQ/edit#). Search for "alt location".

One minor thing that this doesn't solve as far as I can tell is when you want to vendor for the first time with dep init, then you would have to run dep ensure <package> for all the dependencies instead, which is a bit annoying if you already defined them as imports in your code.

If you're working on a private, internal company system, then this sort of portability isn't really a concern. The problem with a feature like this is that it wouldn't just be used for that - how many people would write their own little custom rule for, say, gitlab (for which we don't currently have a built-in rule)?

I can understand wanting to prevent users from misusing a feature like this, but if this would be the only way to get dependencies from gitlab, then it would be better than dep not working with gitlab at all IMO (assuming that not having a built-in rule == it doesn't currently work).

[zevdg]

Ok, so my main concerns around rewrite rules for alternate sources seem to be much less pressing due to the paradigm shift that seems to be happening around #213 . If the manifests will not be edited by dep, then we can just make our own manifest generation tooling that writes to the manifest and not have to worry about conflicting with or hooking into dep. If implemented, #221 could help ensure this tooling is installed correctly. The UX could be improved by making some sort of source rewrite rules a first class citizen in dep, but it's a less pressing issue now and it seems like a contentious feature that could be added in a later version of dep. I'll open a ticket for reference and discussion around this issue.

The other main issue is authentication. Even though github has the right go-import meta tags, they are not accessible for any repository (public or private) in my company's GHE necessitating our vanity import server. If I could configure dep with a github access token or with a git-like credential manager, that wouldn't be necessary. To be clear though, AFAIK, this isn't just an issue for GHE, but also for private repositories on regular old github.com We could continue to use this issue for that, but it's now pretty cluttered with other stuff so I'll open a separate issue for that as well.