separate the build runner process from the configure process

[andrewrk]

This is a prerequisite to #14286.

Currently, the build runner process starts from the main() function inside lib/compiler/build_runner.zig and is compiled alongside the user's build.zig logic. The build runner invokes the user's build.zig logic, which is the "configure phase". After the build function returns, the build runner proceeds to execute the build graph, which is the "make" phase.

This issue is to separate the one compilation into two:

• Configure phase executable which contains only build.zig logic and the build.zig logic of its dependencies.
• Build runner

The primary motivation for this is to make zig build faster, both in the sense that it will take less time to compile, because only the user's build.zig logic is being recompiled when it changes, and in the sense that the build runner could be built with optimizations enabled, which is starting to become more valuable now that we have introduced --watch and --fuzz.

--debug-runner-optimize=[mode] should be introduced to allow switching the build runner to be compiled in a different optimization mode; this will be helpful when working on a patch to the build runner itself.

The primary challenge for implementing this will be separating out state from std.Build.Step that corresponds to configure phase and that corresponds to make phase. It will be a welcome change, because it will make the API a lot smaller and prevent users from having access to state that is only meant for use by the build runner. The build runner will probably have its own version of each std.Build.Step with different fields and logic.

The other major thing that needs to be done to make this work is serialization of the build graph. The output of the configure process needs to be communicated to the runner process, ideally by using stdout alone.

This constitutes a breaking change. In particular, custom build steps will stop working. Users should switch to using Run steps instead.

If we can't get this fully done in this release cycle, #14941 will be a good first step along the way.

[mlugg]

This breaking change may also be an opportune time to rename pub fn build to pub fn configure.

[ikskuh]

As you've already stated on discord, this will break certain use cases of custom build steps.

I'd consider #20935 a blocker for this as currently we need to fall back to custom build steps to achieve re-running the step when something in the directory changes. Uses cases here are asset bundling for games/websites/...

Also projects like https://github.com/zig-osdev/disk-image-step will suffer from this because of two reasons:

• Making them do every individual operation as a single run step will bloat the cache directory enormously (disk image size * number of operations), which isn't really great when you have 50 to 100 operations and a disk image size of 1 GB
• Making it a single run step will break on windows where you only have up to 8192 characters for a command line. This means you have a (theoretical) upper limit of 256 files passed assuming no file has a path longer than 32 chars (which is the length of our cache hash identifiers). This means we need to provide support for somehow passing "argument" files or we need to be able to have a step that allows using LazyPath values to be written to a file with custom information (kinda like filling out a file template or something like that)