Simplify build process

[pg83]

Hi.

Please consider using cmake, or similar open source tool, for building carbon.

Many exciting google projects are not in widespread use, just because they use Bazel as build system.

Bazel is huge, slow, and depends on Java.

[jojolatina]

Having the need for a build system is really bad, the compiler should be able to build the project from the sources directly. All the build configuration should be doable with the language itself, not by using an external tool. This is a new language, let's forget about old legacy limitations

[janwilmans]

A build system like cmake is able to many things, such as create packages, run tests, reconfigure the build for different platforms. The same way you should not create God classes that do everything, we should also not build God tools that do everything.

So the compiler should compile, and a different tool should build, a nice separation of concerns.

[chandlerc]

I think there is some confusion here...

As mentioned in #1436 -- whatever build system we use internally to build Carbon tools and components has no effect on what build systems we support for eventually building Carbon code. We will of course support users with CMake as well as most other build systems.

As for the tools we use to build our internal tools and infrastructure, those are motivate by pragmatic concerns. The current setup with Bazel gives us really high quality test management and careful control over the exact build mode, for example enabling fuzz testing and ASan out of the box.

We think we've made it pretty low-overhead to grab Carbon and start building all the same. And no one else should need to interface with us at this point, it is much much too early. So for now, it seems reasonable to just focus on what works well for the team. Does that make sense?

[laurentlb]

fyi, you don't need to install Java for using Bazel. The Bazel binary should be self-contained (and it's around 50MB).

Build systems are pretty much required as soon as builds become complex (with multiple languages, distributed compilation, remove cache, etc.).

[pg83]

fyi, you don't need to install Java for using Bazel. The Bazel binary should be self-contained (and it's around 50MB).

Build systems are pretty much required as soon as builds become complex (with multiple languages, distributed compilation, remove cache, etc.).

yeea...

pg-> wget https://github.com/bazelbuild/bazel/releases/download/6.0.0-pre.20220706.4/bazel-6.0.0-pre.20220706.4-linux-x86_64
...
(objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 52039598 (50M) [application/octet-stream]
Saving to: ‘bazel-6.0.0-pre.20220706.4-linux-x86_64’

bazel-6.0.0-pre.20220706. 100%[==================================>]  49.63M  8.01MB/s    in 8.5s    

2022-07-22 15:41:18 (5.81 MB/s) - ‘bazel-6.0.0-pre.20220706.4-linux-x86_64’ saved [52039598/52039598]

pg-> chmod +x bazel-6.0.0-pre.20220706.4-linux-x86_64 
pg-> ./bazel-6.0.0-pre.20220706.4-linux-x86_64 
bash: ./bazel-6.0.0-pre.20220706.4-linux-x86_64: No such file or directory

I am using custom statically linked musl-based distro.

[pg83]

you don't need to install Java for using Bazel

Many of us think that running precompiled binaries from Internet is not a very good idea.

keywords: guix, reproducible builds, pure builds, bootstrap :)

[pg83]

We think we've made it pretty low-overhead to grab Carbon and start building all the same. And no one else should need to interface with us at this point, it is much much too early. So for now, it seems reasonable to just focus on what works well for the team. Does that make sense?

Absolutely.

[davidzchen]

Looking at the installation documentation on the Bazel website, while Bazelisk is the recommended way to install, Bazel is also already available on many popular package managers across the popular Linux distros, macOS (via Homebrew), as well as Windows (via Chocolatey and Scoop). Fetching a binary from the internet using wget is not required.

[pg83]

Looking at the installation documentation on the Bazel website, while Bazelisk is the recommended way to install, Bazel is also already available on many popular package managers across the popular Linux distros, macOS (via Homebrew), as well as Windows (via Chocolatey and Scoop). Fetching a binary from the internet using wget is not required.

I am proud owner of one of those distributions, called stal/IX - https://github.com/pg83/ix (from STAtically LInked LInuX)

So, it is my duty to provide Java, then Bazel, then Carbon, to my users.

And, actually, it is very hard to achive, because java need java to build, in static environment.

https://bootstrappable.org/projects/java.html

[davidzchen]

This can probably be merged with #1544, which discusses the same issue.

It seems that the main complaints with Bazel is 1) it's implemented in Java; and 2) it is slow. These seem to be issues with the tool itself and not so much the build language.

These issues are understandable. Bazel is more optimal for large codebases. After the initial bootstrapping of the first build, subsequent builds are fast and fully incremental, and clean builds are never needed. The slowness is a result of having to start up the JVM for the Bazel daemon that runs in the background. There is an alternative universe somewhere out there where Bazel was implemented in C++ instead of Java, and it would have been wonderful if that were the case here.

The JVM issue aside, a pipe dream of mine is that it would be cool to put together enough resources to reimplement Bazel in Carbon. I would expect the code to even be simpler and cleaner, especially with using fewer Java patterns.

I think Bazel's build file and Starlark extension language are some of its strong points. I've found it to be much easier to grok the dependencies between different parts of the codebase, understand what the build is doing, and debugging build breakages, especially compared to many JVM build systems like Gradle.

There are in fact implementations of Starlark in other languages, including Go and Rust. I am actually pretty keen on writing a Starlark implementation in Carbon once the language is in a ready enough state. :)

Anyway, it seems there are two issues being raised here:

1. As OP stated, many Google C++ libraries are underused because they use Bazel and are difficult to incorporate into codebases using other build systems such as CMake.
2. From cargo or go build alike cross platform build support #1544, Carbon should have its own build system, similar how Go and Rust have their own build systems.

While I certainly agree that ideally, the way Carbon libraries is built should be standardized, which would lend well to having an easy to use package management system. At the same time, (1) and (2) seem somewhat mutually exclusive since having a custom build system for Carbon would make it more difficult to incorporate Carbon libraries into existing codebases using other build systems.

One possible solution I can think of is that Carbon standardizes on a single build language, perhaps that of Bazel or something else. This would fulfill (2) and make it easier to create a package management system. We would also implement a tool that would generate build files for other build systems such as CMake, etc., which would fulfill (1).

[chandlerc]

Regarding comment #1519 (comment) -- again, I think we're pulling the question from #1544 into here and should avoid that. Build system support is a complex topic that is best discussed separately.

So I think this issue should be focused on whether the internal tools of Carbon use Bazel.

I agree that there are downsides, but there are also very significant upsides.

Note that we strongly recommend using bazelisk to handle the installation. This approach uses signed, released versions of Bazel rather than arbitrary things. If folks are having trouble using bazelisk on a platform, would be good to know. My impression that while it isn't perfect it is quite widely available across Linux, BSD, macOS, Windows... If there are super locked down distros as discussed above, they may have to do work to figure out how to make things available, and sorry for that, but I think the benefits of Bazel still outweigh this.

[davidzchen]

Agreed and apologies for the derailment, @chandlerc. I should have moved pulled the discussion of OP's concerns into #1544 rather than the other way around.

And I certainly agree with your other points. It seems that the issue of how Carbon's codebase should be built is pretty much settled then, and we'll help people resolve issues they encounter when they try to build Carbon as they come in.

[vmiheer]

My impression that while it isn't perfect it is quite widely available across Linux, BSD, macOS, Windows

Availability doesn't help on windows? Using cmake can help: #298.

[gshanemiller]

I think one undersold feature of Rust is its build system - no dependency hell. Like GO, RUST I sure do hope Carbon has some support for build dependency management in the compiler at least for those modules in the build that use it. It's also worth pointing out bootstrapping the build to make the compiler need not meet this goal; it's for builds once the tool chain is up and running. I'll echo earlier comments: I gave up on Bazel for building C/C++ code with a C/C++ compiler.

[davidzchen]

The main discussion on the build system for Carbon projects (as opposed to the Carbon compiler codebase itself) is in #1544.

Definitely agree that there needs to be good tooling for working with external dependencies. This is more a job for a separate tool rather than the compiler (think Cargo vs rustc), but it should be as easy to use as possible.

My view is that it is better to make Carbon's tooling integrate extremely well with existing build systems than to create a bespoke build system for Carbon (i.e. what Go and Rust has). This is separate from a good dependency/package management system, which I think Carbon should have.

The biggest problems with Go and Rust's build tooling is that they are not general-purpose, as raised in #1544. Many software projects, especially larger projects, require steps that are not building source code, such as code generation (e.g. protobufs, swig wrappers, lexer/parser generators, etc.). Building and maintaining a general-purpose build system is highly non-trivial. Since one of the main features of Carbon is seamless integration with existing C++ projects, which enable incremental in-place migration to Carbon, I think it is more important to build good integration with existing build systems (including Bazel, CMake, Ninja, and others) both in terms of building source code and pulling in external dependencies.

Let's continue this discussion on #1544.