Add ppc64le-linux platforms (nogate)

[amboar]

Hello!

Rust and cargo are in a state where PowerPC64LE can bootstrap from recent respective betas and nightlies (I'm yet to test BE). I'm keen to get some CI ducks in a row so that when src/stage0.txt is updated we can contribute results to the build matrix.

I think I've deciphered the bits I need from master/master.cfg such that only the platforms of interest are run on our ppc64le buildslaves, which I intend to name ppc64*-linux.

Andrew

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @brson (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[alexcrichton]

Awesome! I think we'll also need to accompany this with a change to slave-list.txt. Could you gist what you've been using locally for that?

[amboar]

@alexcrichton Apologies for the late reply. I'm getting back to testing my setup. I'm running with the following in slave-list.txt:

ppc64le-linux asdf max_builds=2 snap=true dist=true trigger=true

Now given the patch configures the slave as no-gate we should probably drop the dist parts. Though as I understand should just be enough to not specify dist=true in slave-list.txt?

[alexcrichton]

Oh so as an update @amboar in the meantime we've been rethinking how we do our buildbots and how we orchestrate our releases. Our current thinking is to actually move away from buildbot entirely and move to Travis and AppVeyor entirely for producing our releases. That in turn will make it much more difficult to slot in custom hardware (such as a powerpc builder). It won't be impossible, just more infrastructure that needs to be written.

In light of that, do you still want to pursue this in the meantime? One way forward would be to at least get emulation testing working where we could, on Travis, run a QEMU instance which runs tests inside that small container. We've often wanted to do this but haven't ever gotten around to fleshing out the support for it, but now may be the chance!

[amboar]

Oh so as an update @amboar in the meantime we've been rethinking how we do our buildbots and how we orchestrate our releases. Our current thinking is to actually move away from buildbot entirely and move to Travis and AppVeyor entirely for producing our releases.

Ah, I caught wind of this yesterday when I was trawling through cargo commits.

That in turn will make it much more difficult to slot in custom hardware (such as a powerpc builder). It won't be impossible, just more infrastructure that needs to be written.

Well, on that front, @daxtens runs TopHat (Travis CI On Power - Helper for Automated Testing) which we might be able to leverage for Power native builds. Currently TopHat runs all jobs in a docker container but doesn't yet support docker-in-docker, which is problematic for Rust's Travis configuration.

In light of that, do you still want to pursue this in the meantime?

What's the timeline for the Travis/Appveyor transition? If it's many months this pull-req might still be viable, if its weeks lets give it a miss.

One way forward would be to at least get emulation testing working where we could, on Travis, run a QEMU instance which runs tests inside that small container. We've often wanted to do this but haven't ever gotten around to fleshing out the support for it, but now may be the chance!

If we can't get TopHat to work for us QEMU is a decent enough fall-back. If I recall correctly, @japaric has something like this going with the smoke repository?

[japaric]

If I recall correctly, @japaric has something like this going with the smoke repository?

The stuff in the trust repository is easier to reuse. And, yes, it can run ppc64le binaries transparently.

[alexcrichton]

Well, on that front, @daxtens runs TopHat (Travis CI On Power - Helper for Automated Testing) which we might be able to leverage for Power native builds. Currently TopHat runs all jobs in a docker container but doesn't yet support docker-in-docker, which is problematic for Rust's Travis configuration.

Whoa that's awesome! I had no idea this existed. I'd love to hook up rust-lang/rust to something like this, and I think it could work great.

Although on Linux we use docker, the main build does not require Docker as we also run tests on OSX and Windows. I suspect that this would similarly just be "run the right script with the right args" and then we'd be off to the races.

What's the timeline for the Travis/Appveyor transition?

My hope is to enable Travis/AppVeyor for auto next week, although that may be ambitious. In any case though I'm shooting for a time-scale of weeks.

If I recall correctly, @japaric has something like this going with the smoke repository?

Certainly! I'd be totally up for the QEMU approach. With being able to transparently run powerpc binaries it should be even easier as well!

[daxtens]

I haven't exactly publicised TopHat yet - it's still definitely in beta.

Docker support in TopHat will take me until at least early next year, but if there's a script we can run that doesn't require Docker, that may well work now. I'll sync up with @amboar as I'm not really across the rust build process.

[alexcrichton]

Oh yeah a lack of docker support should be fine. This system could just mirror what we do on OSX which is to just configure some env vars followed by running the actual build script.

[amboar]

Oh yeah a lack of docker support should be fine. This system could just mirror what we do on OSX which is to just configure some env vars followed by running the actual build script.

I'm experimenting with this approach at the moment. I'm interested in avoiding wasting cycles running jobs that are based on x86 docker images. Ideally we should be able to specify machine or architecture in the build matrix to help restrict where jobs are run, but as Travis doesn't yet have that capability I'm taking an approach like:

matrix:
  include:
   - env: TRAVIS_ARCH_NAME=ppc64le SRC=.
...
script:
  - >
        if ...
        elif [ `uname -m` == "ppc64le" ]; then
          if [ "$TRAVIS_ARCH_NAME" = `uname -m` ]; then
            git submodule update --init;
            src/ci/run.sh;
          else
            echo skipping job not fit for ppc64le;
          fi
        elif ...
        fi
...

I've invented the TRAVIS_ARCH_NAME env variable as a parallel to TRAVIS_OS_NAME with the idea that eventually it would do a similar thing to os for an arch entry in the matrix. That's a problem beyond the scope of Rust, but we can rename it to make that more obvious.

Anyway, any thoughts on the approach above? If it works out and you think it looks okay I'll consult with @daxtens and likely send a pull-req to rust-lang/rust.

[alexcrichton]

Oh does the TopHat service also read .travis.yml? I figured that we'd have something like .tophat.yml in our repo for separate configuration of TopHat.

If TopHat reads .travis.yml though then that seems like a fine approach to me!

[amboar]

Yep, TopHat uses the configuration in .travis.yml (but relies on travis-ci.org for parsing). @daxtens could provide more detail there, but essentially there's no .tophat.yml.

[alexcrichton]

Ah then yeah I don't mind adding an entry which is a noop on "real travis" but is the only build run on "TopHat travis"

[amboar]

Great. I'll swap the conditions in the script to actually make it a noop on the real travis.

[amboar]

@alexcrichton was the docker approach partly to remove yourself from ancient Ubuntu environments on Travis? Rustbuild Cargo reports:

... cargo: /lib64/ld64.so.2: version `GLIBC_2.22' not found ...

I'll speak with @daxtens to see if we can sort that out as TopHat is running Ubuntu 14.04 with eglibc 2.19.

[alexcrichton]

@amboar oh that was to just give us control over the environment, we didn't have those issues with x86_64.

Note that we haven't done much glibc compatibility business with our powerpc releases, so it's likely that powerpc is built with such a new toolchain to require a newer glibc, but we can bump that requirement back by compiling our own powerpc toolchain with an older glibc.

[amboar]

@alexcrichton just letting you know we're still here! the "definitely in beta" nature of TopHat has caused us some issues. We still intend to get Rust up and running on it, we just need some more time and resources dedicated to it. We're trying to resolve those issues now.

[alexcrichton]

Ok cool, sounds good to me!

[edunham]

@amboar Thanks for the work setting this up! Builders are now configured in https://github.com/rust-lang/rust/blob/master/.travis.yml so if you'd like to cycle back on this when you're ready, please submit a PR there. This repo's contents is now deprecated and only available for historical reference.

[amboar]

Yeah unfortunately I've been heavily involved with other projects so I haven't been able to dedicate any time to the effort. If I do get time (or someone) I'll pick up with the Travis approach. Thanks!