build system: replace fuzzing UI with build UI, add time report

[mlugg]

This is one mega-commit, sorry; I kinda got carried away and did everything without really committing at any point. To compensate, there's a massive commit message below. But first, here are some demos of the interface this PR implements (with apologies for the poor video quality):

zig_build_fuzzer.mp4

zig_build_time_report.mp4

---

Follow-up tasks:

• Merge std.http writergate changes so that --webui can be enabled on Windows
• Dynamically load more data so that WASM doesn't need to keep a copy of all of it
• Support combining --webui and --watch (this will be easier with std.Io; run the two things concurrently and cancel the watch when we rebuild!)
• Support multiple fuzzer instances in the web UI (requires making the WASM Walk state non-global)
• Display the step hierarchy in the web UI instead of a flat list
• Consider moving time report / fuzz information to be collapsible under the step in the main list/hierarchy
• Exit the fuzzer processes so we can rebuild (this deficiency already existed)
• Fix module assignment in the fuzzer (this bug already existed)
• Fix unsound fuzzer shared memory approach (see comment in std.Build.Fuzz.sendUpdate) (this bug already existed)

---

This commit replaces the "fuzzer" UI, previously accessed with the --fuzz and --port flags, with a more interesting web UI which allows more interactions with the Zig build system. Most notably, it allows accessing the data emitted by a new "time report" system, which allows users to see which parts of Zig programs take the longest to compile.

The option to expose the web UI is --webui. By default, it will listen on [::1] on a random port, but any IPv6 or IPv4 address can be specified with e.g. --webui=[::1]:8000 or --webui=127.0.0.1:8000. The options --fuzz and --time-report both imply --webui if not given. Currently, --webui is incompatible with --watch; specifying both will cause zig build to exit with a fatal error.

When the web UI is enabled, the build runner spawns the web server as soon as the configure phase completes. The frontend code consists of one HTML file, one JavaScript file, two CSS files, and a few Zig source files which are built into a WASM blob on-demand -- this is all very similar to the old fuzzer UI. Also inherited from the fuzzer UI is that the build system communicates with web clients over a WebSocket connection.

When the build finishes, if --webui was passed (i.e. if the web server is running), the build runner does not terminate; it continues running to serve web requests, allowing interactive control of the build system.

In the web interface is an overall "status" indicating whether a build is currently running, and also a list of all steps in this build. There are visual indicators (colors and spinners) for in-progress, succeeded, and failed steps. There is a "Rebuild" button which will cause the build system to reset the state of every step (note that this does not affect caching) and evaluate the step graph again.

If --time-report is passed to zig build, a new section of the interface becomes visible, which associates every build step with a "time report". For most steps, this is just a simple "time taken" value. However, for Compile steps, the compiler communicates with the build system to provide it with much more interesting information: time taken for various pipeline phases, with a per-declaration and per-file breakdown, sorted by slowest declarations/files first. This feature is still in its early stages: the data can be a little tricky to understand, and there is no way to, for instance, sort by different properties, or filter to certain files. However, it has already given us some interesting statistics, and can be useful for spotting, for instance, particularly complex and slow compile-time logic. Additionally, if a compilation uses LLVM, its time report includes the "LLVM pass timing" information, which was previously accessible with the (now removed) -ftime-report compiler flag.

To make time reports more useful, ZIR and compilation caches are ignored by the Zig compiler when they are enabled -- in other words, Compile steps always run, even if their result should be cached. This means that the flag can be used to analyze a project's compile time without having to repeatedly clear cache directory, for instance. However, when using -fincremental, updates other than the first will only show you the statistics for what changed on that particular update. Notably, this gives us a fairly nice way to see exactly which declarations were re-analyzed by an incremental update.

If --fuzz is passed to zig build, another section of the web interface becomes visible, this time exposing the fuzzer. This is quite similar to the fuzzer UI this commit replaces, with only a few cosmetic tweaks. The interface is closer than before to supporting multiple fuzz steps at a time (in line with the overall strategy for this build UI, the goal will be for all of the fuzz steps to be accessible in the same interface), but still doesn't actually support it. The fuzzer UI looks quite different under the hood: as a result, various bugs are fixed, although other bugs remain. For instance, viewing the source code of any file other than the root of the main module is completely broken (as on master) due to some bogus file-to-module assignment logic in the fuzzer UI.

Implementation notes:

• The lib/build-web/ directory holds the client side of the web UI.

• The general server logic is in std.Build.WebServer.

• Fuzzing-specific logic is in std.Build.Fuzz.

• std.Build.abi is the new home of std.Build.Fuzz.abi, since it now relates to the build system web UI in general.

• The build runner now has an actual general-purpose allocator, because thanks to --watch and --webui, the process can be arbitrarily long-lived. The gpa is std.heap.DebugAllocator, but the arena remains backed by std.heap.page_allocator for efficiency. I fixed several crashes caused by conflation of gpa and arena in the build runner and std.Build, but there may still be some I have missed.

• The I/O logic in std.Build.WebServer is pretty gnarly; there are a lot of threads involved. I anticipate this situation improving significantly once the std.Io interface (with concurrency support) is introduced.

[andrewrk]

With respect to --listen, what will be the CLI arg for enabling #615? I.e. the one that an IDE would connect to in order to drive the build system?

[mlugg]

With respect to --listen, what will be the CLI arg for enabling #615? I.e. the one that an IDE would connect to in order to drive the build system?

Good point; I think this functionality might deserve a more precise argument name. Perhaps --webui[=ip] or something? Open to suggestions if you have any.

[VisenDev]

Perhaps --webui[=ip] or something? Open to suggestions if you have any.

Whats the point of even specifying the IP for something running locally? Wouldn't that IP always need to be 127.0.0.1?(Specifying the port makes sense though)

[andrewrk]

No, you can choose for example 0.0.0.0 which allows it to be accessible from outside the host. Or you could specify a different specific IP which allows it to be accessible from that network interface only. Also the syntax provides a standard way for choosing the port.

[mlugg]

Pushing with these changes:

• Use a DebugAllocator instead of smp_allocator in the build runner (but back the arena with page_allocator)
• Rename the --listen option to --webui (also tweak a help string)
• Temporarily ignore client->server messages on Windows due to existing std bug; I'll open a follow-up issue about this before merge

[mlugg]

That last push just rebases, and disables --webui on Windows entirely due to another existing std bug; no significant changes. I'm going to do some testing on macOS, and assuming all is good, hit the shiny green button.

[mlugg]

I can confirm that this appears to work correctly on macOS. I've tested building the compiler itself and running parts of its test suite, with combinations of --webui and --time-report; everything behaved as expected.