zml/rules_zig upstreaming

[steeve]

Hi Andreas,

We have a few modifications in https://github.com/zml/rules_zig/commits/zml/ for convenience but it's time to upstream them imho.

Probably the most controversial one is we renamed zig_module to zig_library to be closer to the bazel nomenclature. This is a breaking change but IMHO a worthy one.

I am also proposing the following rules that we are working on in our fork:

• zig_library -> zig_static_library
• zig_cc_binary: builds a static library to uses rules_cc for linking (useful on macos notably where the linker was slow or for extra customisation)
• zig_cc_{static,shared}_library: same for static and shared library (closer to what rules_cc to moving to)

We also have zls completion support, unified C libraries support (aggregating the CcInfos into one namespace and calling translate-c on it, which is useful when types are shared between libraries in the C namespace).

What do you think ?

[steeve]

Actually, I implemented what I think is a better idea in place of zig_cc_ rules, a linker attribute that can either be cc or zig, and then the zig_build rule calls cc_common link.

Benefit: single target, and solves the fat archive problem when used with build-archive

[aherrmann]

Hi Steeve, Thank you for reaching out! Upstreaming changes sounds great!

To get an idea of what changes to look for, how up-to-date is the zml branch of rules_zig on changes that happened upstream? I see that https://github.com/zml/rules_zig/commits/master is at 1fe88a2 (Feb 21) at the moment. A few things have changed since in particular on the CC rule integration side.

Let me address the highlighted points one-by-one:

rules_cc integration
Zig static and shared library targets can be used as dependencies to cc targets already, see #445 and #443. This enables the use of cc rules to link together Zig generated libraries. Would that address the same need as zig_cc_binary and zig_cc_{static,shared}_library or is there more to it?

There's also support for C-header generation, see #447, #446 (though limited on the Zig toolchain side to Zig <=0.13). And control over whether to include the compiler-rt, see #444, which can be needed for CC interop.

So far I've ensured that the cc toolchain is an optional dependency for rules_zig (e.g. here). That's a useful property for pure Zig projects, especially for complex cross-compilation use-cases, where a correct C toolchain setup would be an utter nightmare. I would like to maintain this property.

static and shared library rules
The point about renaming zig_library to zig_static_library sounds well aligned with #285. That issue also references Bazel's static/shared cc library rules. The intention is to be provider compatible with the static/shared cc library rules.

The reverse, i.e. depending on cc_shared_library is tracked in #178. cc_static_library didn't exist then, but the same applies.

zig_module → zig_library
I've thought about that in the past. This seems like a sensible direction once the current zig_library is renamed to zig_static_library. My main concern here is on providing a good migration path. But that's definitely doable. Initially zig_module was called zig_package. That change included a deprecation and upgrade path. The same can be done here.

ZLS support
Neat, that would address #403.

Unified C libraries support

unified C libraries support (aggregating the CcInfos into one namespace and calling translate-c on it, which is useful when types are shared between libraries in the C namespace

Can you go into a bit more detail, or share a link to an example, on what that means? It sounds related to what #167 provides but more sophisticated.

Bazel and Zig versions
A question I have for you: Which Bazel versions, bzlmod/WORKSPACE setting, and Zig versions do you currently need to support? Currently rules_zig still tries to support the relative large set of combinations (Bazel 7 & 8) x (bzlmod & WORKSPACE) x (Zig 0.11.0 .. 0.14.1). That's too much and I've been planning to drop all but the last two Zig versions and also remove WORKSPACE support. If you don't need support for these either, then that could be a good thing to do first, to make the upstreaming easier. (Fewer cases to support on CI, etc.) WORSPACE in particular is a big headache.

[steeve]

Nice!

To get an idea of what changes to look for, how up-to-date is the zml branch of rules_zig on changes that happened upstream? I see that https://github.com/zml/rules_zig/commits/master is at 1fe88a2 (Feb 21) at the moment. A few things have changed since in particular on the CC rule integration side.

Yeah you want to look at the zml branch: https://github.com/zml/rules_zig/commits/zml/

What I was thinking was we start from your current master and incrementally add. Rebasing will be a lot more complex.

rules_cc integration Zig static and shared library targets can be used as dependencies to cc targets already, see #445 and #443. This enables the use of cc rules to link together Zig generated libraries. Would that address the same need as zig_cc_binary and zig_cc_{static,shared}_library or is there more to it?

Yeah, this is what we did at https://github.com/zml/zml/blob/master/bazel/zig.bzl but it turns out it's a bit painful because it creates 2 targets and there are subtleties related to compiler rt and a few footguns.
As it turns out we can directly call cc_common.link and do the linking ourselves directly inside the rule, which is clearner.

This is what i've done in https://github.com/zml/rules_zig/blob/steeve/zig-build/zig/private/common/zig_build.bzl#L368. There is a new setting --@rules_zig//zig/settings:linkmode={zig,cc}.

There is a cursed path for zig_test, but it works great.

There are a few main reasons we want to perform the linking using the linker configured in cc:

• it's not possible to override the linker using zig build-
• SPEED, on macos we go from 40s to 8s linking
• control, not all options are supported and some are even swallowed (zig cc silently ignores -fuse-ld= option ziglang/zig#18357)

There's also support for C-header generation, see #447, #446 (though limited on the Zig toolchain side to Zig <=0.13). And control over whether to include the compiler-rt, see #444, which can be needed for CC interop.

Yeah we add it too, but it doesn't work. @SuperAuguste had started a PR to refresh it at ziglang/zig#20353 but it never landed. FWIW there is not immediate action to make it work in the short term.

So far I've ensured that the cc toolchain is an optional dependency for rules_zig (e.g. here). That's a useful property for pure Zig projects, especially for complex cross-compilation use-cases, where a correct C toolchain setup would be an utter nightmare. I would like to maintain this property.

I'm aligned. We use use_cc_toolchain(mandatory = False).

static and shared library rules The point about renaming zig_library to zig_static_library sounds well aligned with #285. That issue also references Bazel's static/shared cc library rules. The intention is to be provider compatible with the static/shared cc library rules.

The reverse, i.e. depending on cc_shared_library is tracked in #178. cc_static_library didn't exist then, but the same applies.

zig_module → zig_library I've thought about that in the past. This seems like a sensible direction once the current zig_library is renamed to zig_static_library. My main concern here is on providing a good migration path. But that's definitely doable. Initially zig_module was called zig_package. That change included a deprecation and upgrade path. The same can be done here.

ZLS support Neat, that would address #403.

Yeah, there is a target to download/sandbox ZLS and one to create the completion json at https://github.com/zml/rules_zig/blob/zml/zig/private/zls_completion.bzl

There is a custom build runner that use invokes that target to give ZLS the json it wants.

You can actually try it on zml, everything is sandboxed there.

Unified C libraries support

Can you go into a bit more detail, or share a link to an example, on what that means? It sounds related to what #167 provides but more sophisticated.

Suppose you have 2 Zig packages that share C types packages. With a per package cImport, those types are incompatible because each will have its own naming. Yes, ptrCast or bitCast works but it gets old very very fast.

With the deprecation of cImport and even externalization of translate-c, what we do is transitively aggregate the direct CC deps of our Zig libraries, generate a master header that imports the public headers, then call translate-c on it.

This is also in line with what the Zig doc recommends: a centralized cImport file for all the C deps. This essentially does it automatically.

It lives at: https://github.com/zml/rules_zig/blob/zml/zig/private/common/zig_translate_c.bzl

Bazel and Zig versions A question I have for you: Which Bazel versions, bzlmod/WORKSPACE setting, and Zig versions do you currently need to support? Currently rules_zig still tries to support the relative large set of combinations (Bazel 7 & 8) x (bzlmod & WORKSPACE) x (Zig 0.11.0 .. 0.14.1). That's too much and I've been planning to drop all but the last two Zig versions and also remove WORKSPACE support. If you don't need support for these either, then that could be a good thing to do first, to make the upstreaming easier. (Fewer cases to support on CI, etc.) WORSPACE in particular is a big headache.

We are on Zig 0.14 and Bazel 8.2.1. We try to stay current as much as possible (even moving to nightly once std.Io async lands).
I am all for dropping WORKSPACE support and anything pre-Bazel 8.

Supporting 2 major releases is fine imho (that would be Zig 0.13-0.14 and then 0.14-0.15 when it lands). Given the pace of Zig, I'd say that's not a big issue.

[aherrmann]

What I was thinking was we start from your current master and incrementally add. Rebasing will be a lot more complex.

Agreed!

Yeah, this is what we did at https://github.com/zml/zml/blob/master/bazel/zig.bzl but it turns out it's a bit painful because it creates 2 targets and there are subtleties related to compiler rt and a few footguns.
As it turns out we can directly call cc_common.link and do the linking ourselves directly inside the rule, which is clearner.

Got it, yes, invoking cc_common.link sounds cleaner in that case.
Side note, #444 added an attribute to control compiler-rt.

This is what i've done in https://github.com/zml/rules_zig/blob/steeve/zig-build/zig/private/common/zig_build.bzl#L368. There is a new setting --@rules_zig//zig/settings:linkmode={zig,cc}.

Looking at the code I see that there is both an attribute and a setting. Have you found both paths necessary or would either just an attribute or just a setting suffice?

I'm aligned. We use use_cc_toolchain(mandatory = False).

Perfect!

Yeah, there is a target to download/sandbox ZLS and one to create the completion json

That sounds very cool!

With the deprecation of cImport and even externalization of translate-c, what we do is transitively aggregate the direct CC deps of our Zig libraries, generate a master header that imports the public headers, then call translate-c on it.

Got it, thanks for explaining! Yes, that makes sense.

I am all for dropping WORKSPACE support and anything pre-Bazel 8.

👍

[aherrmann]

To summarize, it sounds like these are the items that need work to complete the upstreaming:

• Refine static and shared library rules #285
  • Rename zig_library to zig_static_library to be explicit and consistent with zig_shared_library. #490
• rename zig_module to zig_library
• optionally use cc-toolchain linker
  • add linkmode attribute and/or setting
• Zls Support #403
• add a translate-c rule

and to ease the upstreaming:

• feat!: remove Bazel WORKSPACE support, bzlmod required #482
• feat!: drop Bazel 7 support, Bazel 8 supported #488
• feat!: remove old Zig version support, 0.13.0 & 0.14.1 are supported #485

Do you see anything missing?

[aherrmann]

I published a new release, 0.10.0, that includes the restricted support (removed WORKSPACE, removed Bazel 7, removed Zig <0.13).

I've also included the rename of zig_library to zig_static_library with a deprecation macro zig_library for a smooth transition. The next release will remove zig_library. The one after can rename zig_module to zig_library.

I hope this provides a good starting point for you and your team to upstream the features from the ZML fork.

[cerisier]

A part of the upstreaming I found a few other items to discuss:

• Use depsets to aggregate and render zig command line args. And avoid duplicate logic in zig_build and zig_module_dependencies + zig_module_specifications.
• Unify cdeps and deps into deps. Differentation can be done based on Providers.
• Removal of csrcs. So that all C compilation is done through a bazel C toolchain.
• Support of copts on zig_module since many copts are handled by the zig frontend even for compilation of zig files.
• Support for linkopts on zig_module regardless of the future linkmode that we will introduce.

[aherrmann]

Thanks @cerisier for raising these extra items. They all sound good to me with one exception:

Removal of csrcs. So that all C compilation is done through a bazel C toolchain.

Can you elaborate on the motivation for this one? As discussed above I've been careful so far to make sure that the C toolchain is optional for the Zig rules. The motivation being that C toolchains tend to be limiting and very difficult to set up when cross platform, cross compilation support, and hermeticity are a goal. So, I'd prefer to keep the C toolchain as an optional dependency.

[cerisier]

Understood. I won't touch this part.