Building An App with Go Modules and Athens 🎉

Hey Gophers! We're gonna build an awesome webapp with Gin. Well, it's pretty basic but it shows cat and dog pictures so it's still pretty awesome 😀.

The cool part though? Instead of pulling all my webapp's dependencies directly from version control systems like GitHub (which we've always been doing in the past), we're gonna build it using module proxies and Athens.

In fact, we're actually gonna do the build in two different ways! We'll talk about each as we go. I developed and tested this demo on a Zsh shell on MacOS.

I gave this demo at GoLab 2019 in the session titled "The Athens Project: A Proxy Server for Go Modules".

About the Web App

The web application we're going to build is fairly simple, but not trivial. I built a little server using gin as the framework, and it shows some HTML pages with cat pictures on them.

The cool thing is that the whole codebase is modules-aware. That means it has a go.mod file that keeps track of all my code's dependencies. The built-in Go Modules system reads that file to look up the dependencies it needs to download before it builds my server.

Run The Demo

Below is how how to do the demo yourself. The instructions are for Linux/Mac OS X systems, and I'm assuming you have Go 1.13

First Way: Build With Athens and an Upstream VCS

Athens starts up initially with nothing in its storage. When you run go get, it downloads modules from an "upstream". In this demo, we're configuring it to fetch code directly from the VCS, and then store it forever. You can also configure it to download from module mirrors like proxy.golang.org or gocenter.io.

Run The Server!

We try hard to make it easy to run your own Athens. See here for instructions for running the server a few different ways. Today, we're going to use Docker to run ours.

First, run this to start Athens up:

$ docker run --rm -p 3000:3000 -e GO_ENV=development -e ATHENS_GO_GET_WORKERS=5 gomods/athens:v0.7.0

Next, open a new terminal window and set your GOPROXY environment variable to tell modules to use the local server:

$ export GOPROXY=http://localhost:3000

Also, the Go tool keeps a read-only on-disk cache of every module version you've downloaded for any build. To make it read-only, it stores each file in the cache with -r--r--r-- permissions. Since that's the case, you need to use sudo to clear the cache. Do this in the same terminal window as the previous step:

$ sudo rm -rf $(go env GOPATH)/pkg/mod

And then build and run the server, with verbosity turned on so you can see what's going on!

$ go run -v -x .

After your app is up and running, you can go to http://localhost:3000/catalog to see all the modules that Athens has downloaded into its storage.

Second Way: Use Your Athens While Offline 😱

Like I mentioned in the last demo, Athens stores the dependencies you use forever in its own storage. That means that you can build your code without access to the internet. Let's do that here!

In the previous step, Athens was using in-memory storage, which is strictly for local development / demonstration purposes.

In this demo, we're going to run Athens using its disk storage driver, and pre-load its module database - located in this repository at athens-archive/ with all the dependencies our app needs. This way, Athens will be able to serve all the dependencies that go run requests, and won't ever need to fetch code from the public internet.

First, run Athens with the disk driver, and mount the module database into the Docker container:

$ export ATHENS_ARCHIVE="$PWD/athens-archive"
$ docker run --rm -p 3000:3000 -e GO_ENV=development -e ATHENS_GO_GET_WORKERS=5 -e ATHENS_STORAGE_TYPE=disk -e ATHENS_DISK_STORAGE_ROOT=/athens -v $ATHENS_ARCHIVE:/athens gomods/athens:v0.7.0

Set ATHENS_ARCHIVE to wherever your archive lives. If, for example, it is on a USB key, set it to /Volumes/MyKey (this is where it would most likely live on a Mac)

Next, in a new terminal, clear out your cache again:

$ sudo rm -rf $(go env GOPATH)/pkg/mod

And then, shut down your internet connection 🙈.

You can go to http://localhost:3000/catalog again and see all the modules that Athens has in its storage. Notice that this is the same output that you saw at the end of the first demo. We've pre-loaded all the modules into storage that our app will need!

And finally, run the app again!

$ go run -v -x .

Notice how much faster the build is this time, compared the the previous demo. That's because Athens isn't downloading anything - it's just streaming data directly from disk to the client!

If you want to create your own disk archive

Athens treats storage like a cache, except without the evictions or TTLs. That means you can mount an empty volume, connect to the internet, and execute go run (or similar) against that Athens.

On every "miss", Athens will synchronously download the module and store it. After your go run succeeds, Athens guarantees that it will have stored all your app's dependencies on disk (the same guarantee applies to all other storage drivers too!).

These commands will set up Athens with disk storage and an empty volume:

$ mkdir $ATHENS_ARCHIVE
$ docker run -p 3000:3000 -e GO_ENV=development -e ATHENS_GO_GET_WORKERS=5 -e ATHENS_STORAGE_TYPE=disk -e ATHENS_DISK_STORAGE_ROOT=/athens -v $ATHENS_ARCHIVE:/athens gomods/athens:v0.7.0

After you run them, set GOPROXY and do a go run like we did in the previous demos, and the disk archive will be at $ATHENS_ARCHIVE.

Finally

Thanks for following along! Now you're both a Gopher and an Athenian 💚.

If you want to learn more, check out docs.gomods.io! We'd also love for you to get involved - here are some ways to do so:

• Come star our repo
• Come say hello on the #athens channel in the Gophers Slack
  • This is a great place to come ask for help getting started and ask questions too 😄
• And of course, file bug reports or feature requests and contribute code!

Keep on rockin', Gophers!