Get pacman.zig on the Package Manager

[floooh]

On @andrewrk's suggestion, I'm opening a similar ticket like #16672 to document and track the hickups I encountered when bringing https://github.com/floooh/sokol-zig as package into https://github.com/floooh/pacman.zig.

(NOTE: the package manager support is currently in branches:

• https://github.com/floooh/sokol-zig/tree/package
• https://github.com/floooh/pacman.zig/tree/sokol-package

It might make sense to split this issue into more detailed issues later.

Some background info:

• the sokol-zig repo contains auto-generated Zig bindings for cross-platform C header libraries which consists of:
  • a Zig module with the bindings (https://github.com/floooh/sokol-zig/blob/master/src/sokol/sokol.zig)
  • a static link library compiled from the C code here: https://github.com/floooh/sokol-zig/tree/package/src/sokol/c
• the C code sits on top of platform-specific operating system APIs (and may actually not be pure C code, but also at least ObjC, and maybe even C++):
  • on Windows: d3d11, dxgi, kernel32, user32, ...
  • on Linux: gl, x11, ...
  • on macOS: metal, cocoa, ...
  • for the web platform, the C code makes use of Emscripten features (embedding Javascript into C code, and Emscripten specific APIs)
• the pacman.zig project brings it all together and can be considered a fairly minimal test project for cross-platform code which relies on a C library for wrapping platform specialties that go beyond POSIX.

Currently (without package manager integration) sokol-zig can be integrated in two ways:

• the recommended approach is currently to integrate the bindings as a git submodule, import its build.zig, and call a helper function 'buildLibSokol()' like this: https://github.com/floooh/sokol-zig/blob/561fe5e2a9a99055a0a56e2f68e4da20332e60b3/build.zig#L39-L50
• the other option is to copy the required files into the project, and do everything yourself in the build.zig (this was the approach that pacman.zig used before)

With the package manager integration it now works like this (but this was quite a bit of trial and error):

• expose the Zig bindings code as a module obtainable via std.Build.dependency().module()
• expose the C libary as an artifact obtainable via std.Build.dependency().artifact()

(writing this I guess it would be nice if I could add a linker dependency to a module, so that I only need to expose the Zig module, which would have a dependency on the C library, so that a project would only need to get the Zig module from the dependency, and this would automatically add a library dependency to the top-level project - but this would currently not work with the way I'm handling the web build).

Communicating build parameters to package manager dependencies

I stumbled over two problems:

• First, I was confused that the CrossTarget and OptimizeMode are not communicated automatically to the dependency builder object, instead I need to pass those in the args: anytype parameter of the std.Build.dependency() function. I think it would be better if all -D options are "inherited" automatically, and the parameter would only be used for overriding or adding build options.

• The --sysroot arg is passed as-is to the dependency builder object, if the sysroot path is relative to the root poject, everything works fine in the root project, but breaks down in the dependencies. I would suggest either to reject relative paths in --sysroot, or better, automatically convert relative paths into absolute paths. I currently use this workaround which also works: https://github.com/floooh/pacman.zig/blob/ff3b259f9905f1d16e4fe430552b33c46160d9ab/build.zig#L11-L15

Some confusion about using dependency build.zig as import

I had a little detour when I thought that I'm stuck, and imported the dependency's build zig as module (great feature!).

But calling functions in the import also caused some confusion, but I think this just needs to be documented.

• At first I just called functions in the imported build.zig with the builder object of the root project, instead of the builder object of the Dependency object which I can obtain with std.Build.dependency(). This caused some problems with relative file paths in the dependency build scripts which disappeared after calling the function with the dependency's builder object.

wasm32-freestanding vs wasm32-emscripten

This is a known issue: currently I can't compile the Zig part of pacman.zig with the wasm32-emscripten target because of a problem in the Zig stdlib (see here: #10836 (comment)), this means I need to "patch" the platform .emscripten into the CrossTarget when building the sokol-zig C library (because the C code depends on Emscripten SDK featurs). This means that the Zig parts of the project need to be built with wasm32-freestanding and the C parts as wasm32-emscripten (which works, but feels kinda weird).

Need to provide C include directory on top of Emscripten sysroot

This might be a simple bug, but I noticed that in order for the C compilation to work, I need to provide a separate header search path derived from the sysroot so that the C stdlib headers are found:

https://github.com/floooh/sokol-zig/blob/24a47f10231e45875452a110aacb78aa1a8e9648/build.zig#L110-L112

Without this, common headers like stdlib.h, string.h etc are not found in the C compile step.

Suggestion: given a sysroot path, the header and library search paths should be setup automatically for C compilation.

Suggestion: better integration of external linkers (like the Emscripten linker)

As I described above, the C code of sokol-zig depends on Emscripten platform features (embedding Javascript code into C source files, and using Emscripten specific headers). For the compilation part this works fine with the Zig compiler by using the wasm32-emscripten target, and providing a sysroot which points into the Emscripten SDK, but in the end I need to use the Emscripten linker to get a runnable web build (.html + .js + .wasm files).

Currently I create a system command build step to invoke the Emscripten linker like this: https://github.com/floooh/pacman.zig/blob/b26f4f258ee0aa9889098973edc40df04471d9fc/build.zig#L106-L130

It took a while to figure this out, and it's a bit awkward mainly because I need to make all build outputs available as installed artifacts before the Emscripten linker is called, and I need to make sure that all dependencies are setup correctly.

Maybe it makes sense to provide a ExternalLinkerStep wrapper for "gcc toolchain compatible linkers" which at least takes care of the dependency setup, and adds the build outputs as -l....

Emscripten Integration Wishlist

The following are some ideas for the future, and with the package manager infrastructure I think this doesn't need to be part of the Zig core project:

• the Emscripten SDK could be installable as a package (ideally that package would just consist of scaffolding which git-clones the actual Emscripten SDK installer from https://github.com/emscripten-core/emsdk, and then runs emsdk install [version] and emsdk activate --embedded [version], which would basically make the Emscripten SDK available to Zig projects without "polluting" the user's environment
• this Emscripten SDK package could also "inject" new CompileStep types into the Zig build system - if that's at all possible at the moment (for instance an EmscriptenLinkerStep), or at the very least helper functions which provide the same functionality

...ideally this would be coordinated with other "SDK/toolchain packages", like:

• that standalone Clang toolchain which will replace the current C++/ObjC integration
• maybe parts of the WASI SDK (mainly: https://github.com/WebAssembly/wasi-libc)
• Android SDK/NDK (but that would be a PITA to maintain)
• ...?

build.zig.zon wishlist

It would be nice if I could directly use a git-url and git-ref to provide the package content (.tar.gz for any git commit works via "github magic", but it looks a bit messy):

https://github.com/floooh/pacman.zig/blob/sokol-package/build.zig.zon

This is because sokol-zig doesn't have "semver release tags" so far, they are updated automatically on each commit to the main sokol repository, and a single semver version number also doesn't really make sense, because it's actually a collection of libraries (and technically each library would need its own semver version number).

This is also why I hope that the version number in the build.zig.zon file is just "decorative" and won't be used to check if any caches are uptodate.

[floooh]

Hey, quick update on the current state:

• We switched sokol-zig over to be 'package manager only' now and also updated everything for the latest zig-0.12.0 build system changes.
• We added preliminary support for wasm32-emscripten which is convenient for sokol-zig users to add to their build scripts, but is sort of hacky implemented when it comes to integrating the Emscripten SDK with the Zig build system (I hope to vastly improve the Emscripten linker integration at some later point, also, PRs are always welcome) ;)

Here are some example build.zig scripts how it looks now (there are still 2 separate code paths for 'native vs web':

• pacman.zig: https://github.com/floooh/pacman.zig/blob/6964e94d9c23a28286248435bd661efa1121d515/build.zig
• kc85.zig: https://github.com/floooh/kc85.zig/blob/43e3a1467513c2f3eb79b75c17fdcd114068da11/build.zig

...and this is what the sokol-zig build.zig currently looks like, which also currently contains a lot of Emscripten SDK specific code (this is the one point which could be vastly improved in the future -> move out into a separate emsdk-zig package, and turn the Emscripten linker step into a proper custom build step, instead of a cobbled together run-step:

• https://github.com/floooh/sokol-zig/blob/2cbe853a7c453f1a7c2de9215a72889d4cc9e0d7/build.zig

Overall, the amount of hacks and workarounds for the wasm32-emscripten build could be drastically reduced (for instance it's no longer necessary to provide a separate entry.c file with a C main() function).

Right now I'm happy :)

...ok, apart from a few minor cosmetical/style issues in the "new" build system API, like this:

pacman.root_module.addImport("sokol", dep_sokol.module("sokol"));

...could at least be shorted to this:

pacman.addImport("sokol", dep_sokol.module("sokol"));

It feels a bit less like poking around in the guts of the build system ;)

[peterino2]

I wanted to ask how did you solve this problem?

First, I was confused that the CrossTarget and OptimizeMode are not communicated automatically to the dependency builder object, instead I need to pass those in the args: anytype parameter of the std.Build.dependency() function. I think it would be better if all -D options are "inherited" automatically, and the parameter would only be used for overriding or adding build options.

It seems like to me that if one package generates a build flag there's no way to effectively use that build flag in a nested system without leaking that flag to all future potential dependencies.

[andrewrk]

I think this issue has served its purpose. Thanks, @floooh

[floooh]

Yep indeed. I have converted all my Zig projects to using build.zig.zon and have no complaints :)

The build system also lets me fix weird edge cases like injecting a package dependency into another package dependency from a top-level project, or the somewhat wonky Emscripten linker support I created for sokol-zig projects (not entirely happy with it yet, but it does the job for now).

Cheers!