cmd/go: clean GOCACHE based on disk usage

[4a6f656c]

The GOCACHE appears to lack a disk size limit - this is a problem in a space constrained environment and/or when running go on a disk that is nearing capacity. For example, on the openbsd/arm builder (which runs on a USB stick), the ~/.cache/go-build directory runs past several GB in a very short time, which then leads to various failures (git clone or go builds). The only option that I currently appear to have is to run go cache -clean regularly, in order to keep the cache at a respectable size. It seems that having a configurable upper bound would be preferable and/or free disk space based checks that prevent writes (e.g. evict then write) to the cache from failing when the disk is full due, partly due to a large GOCACHE:

[gopher@cubox ~ 102]$ du -csh ~/.cache/go-build                                                                                                                                                                                 
2.2G    /home/gopher/.cache/go-build
2.2G    total
[gopher@cubox ~ 103]$ df -h /home
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd1m      9.0G    8.5G   -6.0K   100%    /home
[gopher@cubox ~ 104]$ go build -o /tmp/main /tmp/main.go

/home: write failed, file system is full

/home: write failed, file system is full

/home: write failed, file system is full

/home: write failed, file system is full

/home: write failed, file system is full

/home: write failed, file system is full

/home: write failed, file system is full

/home: write failed, file system is full

/home: write failed, file system is full
[gopher@cubox ~ 105]$ du -csh ~/.cache/go-build          
2.2G    /home/gopher/.cache/go-build
2.2G    total
[gopher@cubox ~ 106]$ df -h /home                        
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd1m      9.0G    8.5G   -6.0K   100%    /home

[jayconrod]

I think this would be a good thing to have, especially since the cache will be required starting in Go 1.12. I don't think there's a fixed maximum cache size that would work for everyone, but maybe we could at least make eviction more aggressive when the disk is low on space.

[mvdan]

We're no longer allowing GOCACHE=off starting in 1.12, so it might be a bit tricky to allow an arbitrary size limit. One could simply swap GOCACHE=off with GOCACHELIMIT=0, for example.

I seem to remember that @rsc designed the cache to automatically evict based on modification time. Seems sound to me to also automatically evict if the disk is full enough that it might realistically hit errors if it builds a few large packages (say, if there's less than 500MB left).

[bitfield]

This sounds like a bit of a can of worms. How do you check for remaining disk space in a cross-platform way? How much free space should trigger a cache eviction? How do you guarantee that, having evicted some cache, the build process won't still run out of disk anyway?

If you're building on a small disk, then wouldn't running go cache -clean before each build be a better solution than trying to build disk space management into the build tool itself?

[mvdan]

I personally don't know how easy it would be to control "does the disk have enough space left" in a sane and portable way. I was simply pointing out that since the current eviction algorithm takes into account timestamps, perhaps it should also use some data from the filesystem or disk. If we can make that work, we could avoid adding more knobs to the go tool.

If you're building on a small disk, then wouldn't running go cache -clean before each build be a better solution than trying to build disk space management into the build tool itself?

That might not be a great solution - what if you're building a very large project like Kubernetes? It might produce many hundreds of megabytes of build cache, so it's not unreasonable to think that it could on its own be enough to fill up some filesystems.

[bitfield]

That might not be a great solution - what if you're building a very large project like Kubernetes? It might produce many hundreds of megabytes of build cache, so it's not unreasonable to think that it could on its own be enough to fill up some filesystems.

Indeed. So this proposal would, at best, kick the problem slightly further down the road.