Alternate init modes

[sdboyer]

To accommodate the variety of ways people currently manage their Go code and make the transition to dep as easy as possible, we want to give dep init different operational modes. This is a meta-issue for discussing and tracking all these different modes we might implement.

Currently, dep init works by:

1. Analyzing the imports in your project
2. Searching the GOPATH in which your project is located for these imports
3. If not found, it adds the project to a list for gps to solve
4. If found, it reads the vcs version from disk and uses that as the initial constraint in the manifest
5. Step 4 is repeated transitively until we've run out of imports to visit
6. If any imports weren't found on disk, we run a solve to pick versions for those, populating the manifest with versions from the solution (only for missing direct deps, though); they are added to the lock, but omitted from the manifest.

This is a reasonable model for people who've been using GOPATH in the conventional/vanilla way: it's basically just pulling up information from what's on disk into your manifest, so that dep can then replicate everything under vendor. However, this hardly the only way that may make sense for init to work.

Converting existing tools

First, there's converting over from an existing community tool. The basic approach here is to write a converter - something that reads the metadata files from the tool and translates them into the dep equivalent.

There's a bunch of prior art to work from here, fortunately: glide does these sorts of conversions automatically. In the work I was doing to slot gps in as the engine for glide, I updated the converters for gb, godep, gom, and gpm to be gps-style. These write to glide's version of manifests and locks, but those are quite analogous to dep, and should be easy to adapt. There's also an open PR for converting from govendor, though that's to existing glide structs, not the gps-adapted versions of them. (No help for gvt or anything else, unfortunately)

Now, we might ultimately decide we want to keep these tool-specific converters external to dep - maybe in a fully separate converter tool, or just ask the maintainers to add such commands themselves. Either way, though, it doesn't hurt to write them here now.

Just the network, please

Another alternate mode might be to ignore all the existing local information - GOPATH, existing tools, everything - and just have the solver pick out the latest versions it can. This might be surprisingly worthwhile, especially for projects that may have been lagging behind.

Interactive selection

All of these preceding options are more or less separate modes (with "just the network" being a supplemental fallback that can automatically fill in holes left by any others). However, there's another orthogonal option that we may want: prompting the user to select the version they want. I'm imagining two, mutually exclusive options:

1. -prompt-missing, where the tool prompts the user to pick a version available upstream only if the main mode provides no useful constraint
2. -prompt-all, where the user is prompted to choose from among the available upstream versions, and the version determined by the main inferencing mode (if any).

[sdboyer]

Let's do feedback and discussion on the general question of init modes here, but open up spinoff issues/PRs for each of the discrete parts.

[mattn]

I can write the converter of gom (my self) if you want.

[sdboyer]

@mattn sounds great to me :) Open a new PR whenever you're ready, and just link it to this? I'll also probably create a label at some point.

[flibustenet]

Why init doesn't look in vendor folder before GOPATH (like the app) ?

[sdboyer]

@flibustenet init relies on vcs metadata to determine versions. That metadata is usually present on GOPATH, and usually NOT present in vendor (most tools don't preserve it). Inferring versions without having vcs metadata is absurdly difficult in the general case.

More generally, though, init considers vendor to be a volatile - a place where code can be dumped. It does not represent user intent, per this graphic: #119 (comment) . At most, dep's interest in vendor is populating it, or verifying (#121) that it is what it should be.

[flibustenet]

@sdboyer Thanks for your explanation, it is obvious.

[jbrodriguez]

Just the network, please 👍

[nathany]

When running dep init on a project previously managed with gvt, my hope would be to read in the gvt vendor/manifest and use the commit sha from that file for the new lock.json file instead of basing it on SHAs in the GOPATH.

That's the only part that I feel needs importing in this case, matching the lock file to what exists in vendor based on metadata from an existing tool.

I'd actually prefer everything else be based on the dep solver, which may find more/less packages than the vendor/ folder or manifest contain.

It's imperfect, but establishes a good base to continue from.

An alternative to building this into dep init or providing stand-alone tools might be to have the other tools write out a dep manifest (once solidified). e.g. gvt export to write the correct manifest for dep.

[nathany]

On the flip side, if dep init or specific mode of init is focused around using the SHAs from GOPATH or whatever the latest version is from the network, then I think the vendor folder should be updated on init.

The problem I saw was that dep init was locking in commit SHAs that didn't match what was in my vendor folder -- not what was already there, and not updating what was there to match.

"Note: init does NOT vendor dependencies at the moment. See dep ensure." dep init -h

Though I'd personally prefer that the lock file was imported from the legacy tool so I could upgrade dependencies gradually.

[dmitris]

I would love for dep init to have an option to pick up the "source" origin (in Glide - repo field such as repo: git@opensource.mirror.tld.com:golang/net.git) from the current GOPATH and insert it into the manifest file so that you would not have to do it manually for each of the many mirrored repos.

[sdboyer]

@dmitris that's doable, though it would be a bit of work with the way things are currently split up. Would your use case not be better served simply by being able to convert a glide.yaml?

[dmitris]

@sdboyer - that would mean that when starting a new project, I would need to manually add to the manifest the repo or source field to every "external" (github.com, golang.org/x etc.) dependency to point to the mirror (have to do the same thing with glide now as well).

In any case, being able to convert from glide.yaml without loss of information such as repo mappings would of course be very helpful for the existing projects.