Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a Cargo-based build system to eventually replace make #31123

Merged
merged 10 commits into from
Feb 12, 2016

Conversation

alexcrichton
Copy link
Member

This series of commits adds the initial implementation of a new build system for
the compiler and standard library based on Cargo. The high-level architecture
now looks like:

  1. The ./configure script is run with --enable-rustbuild and other standard
    configuration options.
  2. A Makefile is generate which proxies commands to the new build system.
  3. The new build system has a Python script entry point which manages
    downloading both a Rust and Cargo nightly. This initial script also manages
    building the build system itself (which is written in Rust).
  4. The build system, written in rust and called bootstrap, architects how to
    call cargo and manages building all native libraries and such.

One might reasonably ask "why rewrite the build system?", which is a good
question! The Rust project has used Makefiles for as long as I can remember at
least, and while ugly and difficult to use are undeniably robust as they contain
years worth of tweaking and tuning for working on as many platforms in as many
situation as possible. The rationale behind this PR, however is:

  • The makefiles are impenetrable to all but a few people on this
    planet. This means that contributions to the build system are almost
    nonexistent, and furthermore if a build system change is needed it's
    incredibly difficult to figure out how to do so. This hindrance prevents us
    from doing some "perhaps fancier" things we may wish to do in make.
  • Our build system, while portable, is unfortunately not infinitely portable
    everywhere. For example the recently-introduced MSVC target is quite unlikely
    to have make installed by default (e.g. it requires building inside of an
    MSYS2 shell currently). Conversely, the portability of make comes at a cost of
    crazy and weird hacks to work around all sorts of versions of software
    everywhere, especially when it comes to the configure script and makefiles.
    By rewriting this logic in one of the most robust platforms there is, Rust,
    we get to assuage all of these worries for free!
  • There's a standard tool to build Rust crates, Cargo, but the standard library
    and compiler don't use it. This means that they cannot benefit easily from the
    crates.io ecosystem, nor can the ecosystem benefit from a standard way to
    build this repository itself. Moving to Cargo should help assuage both of
    these needs. This has the added benefit of making the compiler more
    approachable for newbies as working on the compiler will just happen to be
    working on a large Cargo project, all the same standard tools and tricks will
    apply.
  • There's a huge amount of portability information in the main distribution, for
    example around cross compiling, compiling on new OSes, etc. Pushing this logic
    into standard crates (like gcc) enables the community to immediately benefit
    from new build logic.

Despite these benefits, it's going to be a long road to actually replace our
current build system. This PR is just the beginning and doesn't implement the
full suite of functionality as the current one, but there are many more to
follow! The current implementation strategy hopes to look like:

  1. Land a second build system in-tree that can be itereated on an and
    contributed to. This will not be used just yet in terms of gating new commits
    to the repo.
  2. Over time, bring the second build system to feature parity with the old build
    system, start setting up CI for both build systems.
  3. At some point in the future, switch the default to the new build system, but
    keep the old one around.
  4. At some further point in the future, delete the entire old build system.

Alright, so with all that out of the way, here's some more info on this PR
itself. The inital build system here is contained in the src/bootstrap
directory and just adds the necessary minimum bits to bootstrap the compiler
itself. There is currently no support for building documentation, running tests,
or installing, but the implemented support is:

  • Compiling LLVM with cmake instead of ./configure + make. The LLVM
    project is removing their autotools build system, so we'd have to make this
    transition eventually anyway.
  • Compiling compiler-rt with cmake as well (for the same rationale as above).
  • Adding Cargo.toml to map out the dependency graph to all crates, and also
    adding build.rs files where appropriate. For example alloc_jemalloc has a
    script to build jemalloc, flate has a script to build miniz.c, std will
    build libbacktrace, etc.
  • Orchestrating all the calls to cargo to build the standard distribution,
    following the normal bootstrapping process. This also tracks dependencies
    between steps to ensure cross-compilation targets happen as well.
  • Configuration is intended to eventually be done through a config.toml file,
    so support is implemented for this. The most likely vector of configuration
    for now, however, is likely through config.mk (what ./configure emits), so
    the build system currently parses this information.

There's still quite a few steps left to do, and I'll open up some follow-up
issues (as well as a tracking issue) for this migration, but hopefully this is a
great start to get going! This PR is currently tested on all the
Windows/Linux/OSX triples for x86_64 and x86, but more portability is always
welcome!


Future functionality left to implement

  • Re-verify that multi-host builds work
  • Verify android build works
  • Verify iOS build work (mostly compiler-rt)
  • Verify sha256 and ideally gpg of downloaded nightly compiler and nightly rustc
  • Implement testing -- this is a huge bullet point with lots of sub-bullets
  • Build and generate documentation (plus the various tools we have in-tree)
  • Move various src/etc scripts into Rust -- not sure how this interacts with make build system
  • Implement make install - like testing this is also quite massive
  • Deduplicate version information with makefiles

@rust-highfive
Copy link
Collaborator

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

@sfackler
Copy link
Member

🎊

@steveklabnik
Copy link
Member

Hero mode omg 💯 💯 💯

@alexcrichton
Copy link
Member Author

In terms of review, I've tried to split things up in as bite-sized chunks as possible, and I'll avoid rebasing for awhile so the commit comments are preserved

@jonas-schievink
Copy link
Contributor

Yesss ✌️

@nikomatsakis
Copy link
Contributor

Hold the phone.

@aturon
Copy link
Member

aturon commented Jan 22, 2016

@alexcrichton does it again!

@Gankra Gankra added T-tools relnotes Marks issues that should be documented in the release notes of the next release. labels Jan 22, 2016
@retep998
Copy link
Member

Is the python bit optional? As in, if we already had a recent enough rust/cargo nightly, could we just directly build the Rust bootstrap and go from there?

import tarfile

def get(url, path, quiet=False):
if quiet:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if not quiet:

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh this is actually intended (albeit kinda weirdly) where quiet means "just print what you're downloading" and not quiet means "print the command we're running"

@alexcrichton
Copy link
Member Author

@retep998

Is the python bit optional? As in, if we already had a recent enough rust/cargo nightly, could we just directly build the Rust bootstrap and go from there?

There's some discussion here about this as well, but I wouldn't necessarily want it to be a prerequisite to build Rust that you have Rust installed just yet. Maybe down the road we can expect that (I think gcc requires this, right?).

That being said I would love to remove even the python script (as @frewsxcv has discovered I'm awful at writing python) and just have pure Rust from the get-go. For now though we require Python because of the LLVM build anyway, and I tried to keep the Python as tiny as possible. Hopefully we can either minimize the python over time (e.g. get it even smaller than it is today) or possibly even rewrite it in Rust as you mentioned and just assume that rustc is available.

@alexcrichton
Copy link
Member Author

I've started keeping a tentative list of items left-to-implement at the top of this PR as well. I plan on explicitly not implementing them here but rather opening follow-up tracking issues for them.

if code != 0:
if not verbose:
print("failed to run: " + ' '.join(args))
raise Exception("failed to run command")
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RuntimeError

@hanna-kruppe
Copy link
Contributor

Aside: How does one use this build system? I tried ./configure --enable-rustbuild (64 bit Windows in a mingw shell) but that fails in a clean checkout:

configure: looking at LLVM
configure:
configure: configuring LLVM for x86_64-w64-mingw32
configure: configuring LLVM with:
configure: --enable-targets=x86,x86_64,arm,aarch64,mips,powerpc --enable-optimized --disable-assertions --disable-docs --enable-bindings=none --disable-terminfo --disable-zlib --disable-libffi --disable-pthreads --build=x86_64-w64-mingw32                         --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-python=/mingw64/bin/python2.7
./configure: line 1738: <...>/src/llvm/configure: No such file or directory

src/llvm is empty, not even a .git. It seems that --enable-rustbuild disables checking out submodules.

@alexcrichton
Copy link
Member Author

I'm gonna rebase this to get rid of the merge and [rebase] commits, and pray nothing dies anywhere!

This commit is the start of a series of commits which start to replace the
makefiles with a Cargo-based build system. The aim is not to remove the
makefiles entirely just yet but rather just replace the portions that invoke the
compiler to do the bootstrap. This commit specifically adds enough support to
perform the bootstrap (and all the cross compilation within) along with
generating documentation.

More commits will follow up in this series to actually wire up the makefiles to
call this build system, so stay tuned!
During the transition period where we're still using ./configure and makefiles,
read some extra configuration from `config.mk` if it's present. This means that
the bootstrap build should be configured the same as the original ./configure
invocation.

Eventually this will all be removed in favor of only storing information in
`config.toml` (e.g. the configure script will generate config.toml), but for now
this should suffice.
This likely isn't always valid, and subverts auto-detection.
@alexcrichton alexcrichton force-pushed the who-doesnt-want-two-build-systems branch from 96bf1b9 to 1bd4a65 Compare February 11, 2016 18:46
@alexcrichton
Copy link
Member Author

@bors: r=brson 1bd4a65db80b7ce2ab0f3c8801cfaa68009a4133

I guess that wasn't so bad after all

These describe the structure of all our crate dependencies.
This commits adds build scripts to the necessary Rust crates for all the native
dependencies. This is currently a duplication of the support found in mk/rt.mk
and is my best effort at representing the logic twice, but there may be some
unfortunate-and-inevitable divergence.

As a summary:

* alloc_jemalloc - build script to compile jemallocal
* flate - build script to compile miniz.c
* rustc_llvm - build script to run llvm-config and learn about how to link it.
  Note that this crucially (and will not ever) compile LLVM as that would take
  far too long.
* rustdoc - build script to compile hoedown
* std - script to determine lots of libraries/linkages as well as compile
  libbacktrace
Have all Cargo-built crates pass `--cfg cargobuild` and then add appropriate
`#[cfg]` definitions to all crates to avoid linking anything if this is passed.
This should help allow libstd to compile with both the makefiles and with Cargo.
This will allow it to be used as a crate in a Cargo-based build
Refactor a bit to have less repetition and #[cfg] and try to bury it all inside
of a macro.
This commit adds a `--enable-rustbuild` option to the configure script which
will copy a different `Makefile.in` into place to intercept all `make`
invocations.

Currently this makefile only has one target, but it's expected to be filled out
quite a bit over time!
When building with Cargo we need to detect `feature = "jemalloc"` to enable
jemalloc, so propagate this same change to the build system to pass the right
`--cfg` argument.
@alexcrichton alexcrichton force-pushed the who-doesnt-want-two-build-systems branch from 1bd4a65 to 55dd595 Compare February 11, 2016 19:12
@alexcrichton
Copy link
Member Author

@bors: r=brson 55dd595

@bors
Copy link
Contributor

bors commented Feb 11, 2016

⌛ Testing commit 55dd595 with merge 2300556...

@bors
Copy link
Contributor

bors commented Feb 11, 2016

💔 Test failed - auto-mac-64-opt

@alexcrichton
Copy link
Member Author

@bors: retry

On Thu, Feb 11, 2016 at 2:16 PM, bors notifications@github.com wrote:

[image: 💔] Test failed - auto-mac-64-opt
http://buildbot.rust-lang.org/builders/auto-mac-64-opt/builds/8008


Reply to this email directly or view it on GitHub
#31123 (comment).

@bors
Copy link
Contributor

bors commented Feb 12, 2016

⌛ Testing commit 55dd595 with merge 78a5d5b...

bors added a commit that referenced this pull request Feb 12, 2016
…, r=brson

This series of commits adds the initial implementation of a new build system for
the compiler and standard library based on Cargo. The high-level architecture
now looks like:

1. The `./configure` script is run with `--enable-rustbuild` and other standard
   configuration options.
2. A `Makefile` is generate which proxies commands to the new build system.
3. The new build system has a Python script entry point which manages
   downloading both a Rust and Cargo nightly. This initial script also manages
   building the build system itself (which is written in Rust).
4. The build system, written in rust and called `bootstrap`, architects how to
   call `cargo` and manages building all native libraries and such.

One might reasonably ask "why rewrite the build system?", which is a good
question! The Rust project has used Makefiles for as long as I can remember at
least, and while ugly and difficult to use are undeniably robust as they contain
years worth of tweaking and tuning for working on as many platforms in as many
situation as possible. The rationale behind this PR, however is:

* The makefiles are impenetrable to all but a few people on this
  planet. This means that contributions to the build system are almost
  nonexistent, and furthermore if a build system change is needed it's
  incredibly difficult to figure out how to do so. This hindrance prevents us
  from doing some "perhaps fancier" things we may wish to do in make.

* Our build system, while portable, is unfortunately not infinitely portable
  everywhere.  For example the recently-introduced MSVC target is quite unlikely
  to have `make` installed by default (e.g. it requires building inside of an
  MSYS2 shell currently). Conversely, the portability of make comes at a cost of
  crazy and weird hacks to work around all sorts of versions of software
  everywhere, especially when it comes to the configure script and makefiles.
  By rewriting this logic in one of the most robust platforms there is, Rust,
  we get to assuage all of these worries for free!

* There's a standard tool to build Rust crates, Cargo, but the standard library
  and compiler don't use it. This means that they cannot benefit easily from the
  crates.io ecosystem, nor can the ecosystem benefit from a standard way to
  build this repository itself. Moving to Cargo should help assuage both of
  these needs. This has the added benefit of making the compiler more
  approachable for newbies as working on the compiler will just happen to be
  working on a large Cargo project, all the same standard tools and tricks will
  apply.

* There's a huge amount of portability information in the main distribution, for
  example around cross compiling, compiling on new OSes, etc. Pushing this logic
  into standard crates (like `gcc`) enables the community to immediately benefit
  from new build logic.

Despite these benefits, it's going to be a long road to actually replace our
current build system. This PR is just the beginning and doesn't implement the
full suite of functionality as the current one, but there are many more to
follow! The current implementation strategy hopes to look like:

1. Land a second build system in-tree that can be itereated on an and
   contributed to. This will not be used just yet in terms of gating new commits
   to the repo.
2. Over time, bring the second build system to feature parity with the old build
   system, start setting up CI for both build systems.
3. At some point in the future, switch the default to the new build system, but
   keep the old one around.
4. At some further point in the future, delete the entire old build system.

---

Alright, so with all that out of the way, here's some more info on this PR
itself. The inital build system here is contained in the `src/bootstrap`
directory and just adds the necessary minimum bits to bootstrap the compiler
itself. There is currently no support for building documentation, running tests,
or installing, but the implemented support is:

* Compiling LLVM with `cmake` instead of `./configure` + `make`. The LLVM
  project is removing their autotools build system, so we'd have to make this
  transition eventually anyway.

* Compiling compiler-rt with `cmake` as well (for the same rationale as above).

* Adding `Cargo.toml` to map out the dependency graph to all crates, and also
  adding `build.rs` files where appropriate. For example `alloc_jemalloc` has a
  script to build jemalloc, `flate` has a script to build `miniz.c`, `std` will
  build `libbacktrace`, etc.

* Orchestrating all the calls to `cargo` to build the standard distribution,
  following the normal bootstrapping process. This also tracks dependencies
  between steps to ensure cross-compilation targets happen as well.

* Configuration is intended to eventually be done through a `config.toml` file,
  so support is implemented for this. The most likely vector of configuration
  for now, however, is likely through `config.mk` (what `./configure` emits), so
  the build system currently parses this information.

There's still quite a few steps left to do, and I'll open up some follow-up
issues (as well as a tracking issue) for this migration, but hopefully this is a
great start to get going! This PR is currently tested on all the
Windows/Linux/OSX triples for x86\_64 and x86, but more portability is always
welcome!

---

Future functionality left to implement

* [ ] Re-verify that multi-host builds work
* [ ] Verify android build works
* [ ] Verify iOS build work (mostly compiler-rt)
* [ ] Verify sha256 and ideally gpg of downloaded nightly compiler and nightly rustc
* [ ] Implement testing -- this is a huge bullet point with lots of sub-bullets
* [ ] Build and generate documentation (plus the various tools we have in-tree)
* [ ] Move various src/etc scripts into Rust -- not sure how this interacts with `make` build system
* [ ] Implement `make install` - like testing this is also quite massive
* [x] Deduplicate version information with makefiles
@aturon
Copy link
Member

aturon commented Feb 12, 2016

🍻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
relnotes Marks issues that should be documented in the release notes of the next release.
Projects
None yet
Development

Successfully merging this pull request may close these issues.