[rfc] gb build/test automatically download missing dependencies

[davecheney]

This rather blandly titled issue is a placeholder to discuss the changes to gb for the 0.4 series, which the roadmap hints "improve gb vendor".

Backstory

gb cares about reliable builds, a lot. Giving Go developers the tools they need to achieve repeatable builds was the motivation for developing gb, and the main way gb does this is via the $PROJECT/vendor/src directory. A bit later gb-vendor came along when it was clear that users wanted tooling to help them manage their vendor'd code.

But it also clear that not everyone is comfortable with actually having copies of the source in their tree; they would rather have a manifest file that explains where to get those dependencies on request. This catalyzed around the gb vendor restore command which I now regret accepting as it confused the message that gb's reliable build story is based on vendoring. See #498.

To be very clear, the answer for how to get the most reliable builds with gb is always to copy your dependencies into $PROJECT/vendor/src. But sometimes there is an argument for trading reliability for convenience, this is the motivation for this proposal.

Proposal

I propose that during package resolution (which happens for build and test, but may happen at other stages in the future) once the standard search locations have been exhausted (see below), a per user cache of source code is consulted, and if the package is not found in the cache, we attempt to fetch it using the standard gb vendor fetch lookup rules.

Importantly, this is not a proposal to automatically invoke gb vendor fetch, but a separate mechanism that is invoked once all the per project source locations have been exhausted.

TODO:

• decide if this is opt in or opt out; should there be a flag to disable / enable this behaviour, and if so, what its default state will be? This feature will be opt in, it will be enabled by the presence of a dependency file in the project root.

Updating

Maintaining a cache means establishing rules for when the cache is updated. My proposal for this is updating the cache, ie fetching from upstream if the entry is present in the cache, should be off by default.

This means adding a new flag, probably -u which effectively marks anything in the cache as stale and forces the upstream to be consulted. This could be a potentially very time consuming operation.

TODO:

• decide if a partial update is something that should be supported. The UI for this sounds complicated, so it might be simpler to give the user a command to clear a particular cached entry, which accomplishes the same thing. This feature will be enabled only if the dependency file is present, and the dependency file lists explicitly the version required. If the file is updated to reflect a version which is not present locally, that will be downloaded in due course.

Package cache

The package cache, or more correctly a cache of source code, is a location where dependencies can be stored and referenced for gb projects. This cache is per user not per project.

TODO:

• figure out the appropriate location for the package cache, some operating systems have a notion of a per user cache, others don't.

Search order

The search order for packages is currently:

• $GOROOT/src
• $PROJECT/src
• $PROJECT/vendor/src

With this change, rather than terminating if a package could not be found, the cache will be searched, and if the package is not found, an attempt to fetch it from it's upstream (as defined by gb vendor fetch, which itself is defined by go get's rules) is made, and the cache searched again.

The proposed search order could be considered to be roughly:

• $GOROOT/src
• $PROJECT/src
• $PROJECT/vendor/src
• $HOME/$CACHE/src
• $REMOTE_URL

The key take away is that source code higher up the search order takes precedent. If you need a specific revision of a dependency, you should vendor it to $PROJECT/vendor/src.

One more thing

With this change, I've just reinvented go get, golf clap.

Well, yes, and no. It is correct to say that in the first implementations of this design fetching from upstream would have the same problems as go get has currently. But I believe that gb has all the pieces to improve on this.

In golang/go#12302 I made a proposal for a simple release mechanism for Go packages; use semver tags for package releases.

I propose that gb can use this information to let project owners specify the version (as defined in proposal/design/12302-release-proposal.md) or a version range for a dependent package.

Arrgh, Dave, you've introduced a manifest file; twice!

True, to avoid pulling from head, you'd need to use a manifest file, but just like gb vendor, it's totally optional, if you don't want gb to fetch upstream, make sure the source you need is in $PROJECT/vendor/src.

TODO:

• decide on where and how to store version information in the gb project; I propose a file in the root of the project that lists the import path prefixes and a semver like range expression. It will be recorded in $PROJECT/depfile

[gosuri]

I like the convince of downloading missing packages but bit confused about repeatability for the team.

Explicitly downloading dependency to vendor/src, in my mind, is the single most valuable feature that helped us to share dependencies reliably with the team for a large project. The thing i'm concerned about is how do you communicate when a dependency is in cached locally and not vendored.

[davecheney]

@gosuri yup, that's a good point. It sounds like as a prerequisite to this work I need to add a command to gb that print out detailed information of where each package in a project is resolved from.

Additionally this sounds like, as an existing user, you would prefer this feature to default to off.

[davecheney]

@gosuri but don't think of this as "a missing dependency is going to fall back to an unknown pull from upstream's head", the goal is to drive people towards the design in golang/go#12302. If you could specify in some kind of manifest file the version (see 12302) of a dependency, rather than vendor a specific hash, would that be good enough^tm ?

[tianon]

Since you asked for comments explicitly, I'll add my vote for this fetch/cache to be explicitly opt-in if it's added; I'd be very surprised if I did a "gb build" and it started downloading instead of throwing an error (maybe with suggestions about how to get the dep I'm after).

I can't see myself using the feature described here, and honestly see it as contrary to gb's main value proposition. 😇

[davecheney]

@tianon yup, i get that. IMO I think everyone, myself included, is shell shocked by the belief that pulling from upstream would bring us right back to square one with go get.

But imagine if there was a way to pull a specific release version of a dependency from upstream if it was not already present in the project. I believe all the pieces exist to make that happen, although I admit it won't happen overnight; there will be some arm twisting required.

How about this middle ground. For the initial iterations, while there is no manifest file, and no real support for fetching a version of a dependency, this feature defaults to off. When support for fetching a missing dependency version is implemented, then the default could be changed.

[davecheney]

@tianon

I can't see myself using the feature described here, and honestly see it as contrary to gb's main value proposition. 😇

That's a fair comment, and this is on me for not being clear enough with the message. gb cares about reliable builds, and that's by vendoring into $PROJECT/vendor/src, but I recognise that there is a set of users who object to vendoring for various reasons - and to be honest, after working with gb for six months I count myself among them.

Vendoring is a effective, but crude hammer. You get repeatable builds, you get independence from upstream foibles, but, well, hammer.

This proposal doesn't take away that method of working, $PROJECT/vendor/src will always be supported in gb. This proposal is an attempt to ease people into the gb project model, without the biggest hurdle, which is setting up their vendor/src directory.

[gosuri]

@davecheney We prefer reliability over convenience. Although, use semver very religiously for public projects, there are countless instances where version numbers would be stale internal projects. We resorted to injecting commit hashes in version commands to be explicit. Example:

$ walker version
walker 0.0.3 (8d5f2f4a65e871e8156fb7617f7c8c282f32f544+CHANGES)

[davecheney]

@gosuri i understand that completely. The gb party line is, if you don't trust the upstream, then you should vendor the dependency.

[gosuri]

@davecheney yeah.. you'd be surprised how many very "prominent" projects don't really understand the <breaking>.<feature>.<fix> model.

[davecheney]

@gosuri I understand the reservation about semver, however this really isn't about semver, this is about trusting your upstream to do a good job. If you don't trust them -- vendor; but really you're forking them because, well, you don't trust them.

[gosuri]

@davecheney Exactly my point. We use semver religious once the project reaches publishable state, working towards getting better at early stages as well.

Very unfortunate to see several prominent (don't want to single out) projects ship breaking APIs as "feature" releases. I learned not to trust upstreams by default. My current upgrade flow right now is something like:

$ git checkout -b upgrade-foo
$ gb vendor delete foo
$ gb vendor fetch foo -tag ...
$ ... # test
$ git checkout master
$ git merge upgrade-foo # if checks out
$ git branch -D update-foo 

[ChrisHines]

@davecheney

but I recognise that there is a set of users who object to vendoring for various reasons - and to be honest, after working with gb for six months I count myself among them.

I am interested to hear about this evolution in your thinking.

[dahankzter]

I am also interested in hearing how vendoring came to feel as a hammer for you.
I echo most of the objections but with off by default it might work.

Maybe have it as an "experimental" feature?

Is most of the motivation coming from wanting to promote a standard versioning scheme?
I have previously been very much in favor of it before but have warmed to vendoring. I get nostalgic. It reminds me of sending code as patches and tar balls over email.
Can this help with #49? It relates to versions of dependencies at least in part right? Then why not try it? Or make it more focused and make a special manifest file that specifies this.
Please by all that is holy make it simpler than poms...