Interest in switching appveyor to docker

[TimNN]

So apparently docker is available on windows [1] and appveyor [2] [3].

I was wondering if there was any interest in moving the appveyor builds to use docker -- which would make it easier to test windows builds locally without having to fiddle with msys / mingw / etc..

If there is interested (and no one else is interested in doing it) I would be interested in looking into that (after the LLVM 4.0 upgrade is done).

[alexcrichton]

ZOMG YES PLEASE

Or more seriously, I would love to do this. It would:

• Isolate us from changes in the AppVeyor environment (which happens from time to time)
• Allow easy reproduction locally
• Force us to write down what all our real dependencies are

I don't know if AppVeyor has support for this yet, but if they do then I see no reason to not implement this. I'd do it for OSX if I could...

[aidanhs]

Allow easy reproduction locally

Sadly, I suspect not many contributors have access to Windows server 2016 for isolating issues on ;)

(good idea regardless though)

[retep998]

@aidanhs If you have Windows 10 Professional or Enterprise then you can use Windows Containers just fine.

[aidanhs]

Huh, neat. Thanks for pointing that out, I wasn't aware.

[alexcrichton]

I believe AppVeyor has a new Visual Studio 2017 image and their list of updates claims an update to docker images, so maybe this is available now?

@TimNN would you be interested in poking around?

[TimNN]

@alexcrichton: Sure, I'll look into it.

[TimNN]

I've started playing around with docker on windows and can report a first success: x86_64-pc-windows-gnu build and tested successfully_{except for run-make, but we'll fix that in time} in the container.

However one problem is already apparent: The size of the windows base image is ~10GB, though it luckily should be preloaded on windows servers with docker support (unless there are different versions of the image used). This nevertheless means that our current approach to caching docker images is no longer a viable option (at least I don't think so). Additionally just installing the necessary build tools (cmake, python, git, vcbuildtools) increases the image size by another 4GB, which will not be preloaded.

Thus I believe that before we can consider switching appveyor to docker we need to first consider alternative docker caching strategies, so that at least the base image can be excluded from the cache.

---

One possible caching strategy I could imagine:

• hash the contents of the docker build directory
• attempt to pull rust-lang/$image-$hash, where image is for example dist-x86_64-linux
• if pull was unsuccessful build and push the image

Some thoughts on that method:

• This may need some kind of stamp file to allow triggering a manual rebuild of images
• This would probably need some logic to trigger a rebuild if the windows base image on appveyor changed (maybe include the image hash of the base image in our hash?)

[TimNN]

Some other issues I encountered (although I haven't investigated them in depth yet):

• I couldn't get volumes to work -- and even if they should work, some posts I read indicated volumes are implemented using smb / network shares, leading to serious performance problems.
• Various problems with (console) i/o:
  • When attached with tty, some programs don't show their output (MSYS2-based programs don't output anything to console docker/for-win#262)
  • When not attached with tty some output is still not showing (eg. git download progress -- git checkout progress does work...), the terminal sometimes doesn't refresh / doesn't show the last few lines of output unless ENTER is pressed manually
  • Several operations feel kind of slow, eg. when doing a docker build it takes in the order of minutes to get from "the build container has stopped / disappeared from docker ps" to "the image has been created / appeared in docker images" (Although that may be a problem with my environment -- I'm testing in an Azure VM).

While I'll continue to investigate, my intuition says that it's a bit too early to use docker for windows for our ci builds.

[retep998]

some posts I read indicated volumes are implemented using smb / network shares, leading to serious performance problems.

Even if there wasn't a performance problem, Rust is known for its high frequency of spurious failures when working off network drives due to assuming operations finish as soon as the function returns, despite that not actually happening because network drives. For example: #28002

[alexcrichton]

Thanks for all the investigation @TimNN! Sounds like we unfortunately shouldn't jump the gun just yet :(