Skip to content

ci: Move FreeBSD/NetBSD module tests from x86_64-linux to aarch64-linux #24084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

ci: Move FreeBSD/NetBSD module tests from x86_64-linux to aarch64-linux #24084

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e4ff8bf
test: Move all n32/x32 targets under -Dtest-extra-targets.
alexrp Jun 5, 2025
dfe0ad4
build: Add -Dtest-target-filter-exclude to invert -Dtest-target-filter.
alexrp Jun 5, 2025
e752738
ci: Move FreeBSD/NetBSD module tests from x86_64-linux to aarch64-linux.
alexrp Jun 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion build.zig
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,7 @@ pub fn build(b: *std.Build) !void {

const test_filters = b.option([]const []const u8, "test-filter", "Skip tests that do not match any filter") orelse &[0][]const u8{};
const test_target_filters = b.option([]const []const u8, "test-target-filter", "Skip tests whose target triple do not match any filter") orelse &[0][]const u8{};
const test_target_filters_exclude = b.option(bool, "test-target-filter-exclude", "Invert the meaning of -Dtest-target-filter to exclude targets") orelse false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're starting to write a meta language inside the build options... let's not go down the path of having build options fundamentally change the meaning of other build options.

-Dskip-netbsd -Dskip-freebsd would be the simplest approach.

Also OK would be -Dtest-target-include=... -Dtest-target-exclude=... and have the inclusions always override the exclusions since everything is included by default.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-Dskip-netbsd -Dskip-freebsd would be the simplest approach.

I thought about doing that at first, but I don't love the idea of adding options for every new target category that someone might be interested in skipping.

Also OK would be -Dtest-target-include=... -Dtest-target-exclude=... and have the inclusions always override the exclusions since everything is included by default.

You mean something like:

        var include = true;
        for (options.test_target_excludes) |include| {
            if (std.mem.indexOf(u8, triple_txt, exclude) != null) {
                include = false;
                break;
            }
        }
        if (options.test_target_includes.len != 0) {
            include = blk: for (options.test_target_includes) |include| {
                if (std.mem.indexOf(u8, triple_txt, include) != null) break :blk true;
            } else false;
        }
        if (!include) continue;

?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

        const include = (for (options.test_target_excludes) |include| {
            if (std.mem.indexOf(u8, triple_txt, exclude) != null) break false;
        } else true) or for (options.test_target_includes) |include| {
            if (std.mem.indexOf(u8, triple_txt, include) != null) break true;
        } else false;

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That snippet doesn't really make sense. It makes -Dtest-target-include=foo pretty much entirely useless, since you also somehow need to exclude everything else, lest that first for return true giving include = true.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I think the typical way of doing this (at least, it's the thing I expect?) would be for excludes to override includes, and passing no include filters implicitly includes everything (still subject to exclusions). Probably easier to show programmatically and with examples:

const include = for (options.test_target_excludes) |exclude| {
    if (std.mem.indexOf(u8, triple_txt, exclude) != null) break false; // explicitly excluded
} else for (options.test_target_includes) |include| {
    if (std.mem.indexOf(u8, triple_txt, include) != null) break true; // explicitly included (and not excluded)
} else options.test_target_includes.len == 0; // include by default if no include filter given

then you get:

<no options>                 include all targets
-Dtest-target-include=foo    include targets matching 'foo'
-Dtest-target-exclude=foo    include targets not matching 'foo'
-Dtest-target-include=foo -Dtest-target-include=bar -Dtest-target-exclude=baz -Dtest-target-exclude=qux
  ^^ include targets which match 'foo' or 'bar' but do not match 'baz' nor 'qux'

Copy link
Member

@andrewrk andrewrk Jun 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

somehow need to exclude everything else,

-Dtest-target-exclude="" (matches everything)

but I'm fine with the counter proposal suggested above

const test_extra_targets = b.option(bool, "test-extra-targets", "Enable running module tests for additional targets") orelse false;

var chosen_opt_modes_buf: [4]builtin.OptimizeMode = undefined;
Expand Down Expand Up @@ -409,7 +410,7 @@ pub fn build(b: *std.Build) !void {
test_step.dependOn(check_fmt);

const test_cases_step = b.step("test-cases", "Run the main compiler test cases");
try tests.addCases(b, test_cases_step, test_filters, test_target_filters, target, .{
try tests.addCases(b, test_cases_step, test_filters, test_target_filters, test_target_filters_exclude, target, .{
.skip_translate_c = skip_translate_c,
.skip_run_translated_c = skip_run_translated_c,
}, .{
Expand All @@ -427,6 +428,7 @@ pub fn build(b: *std.Build) !void {
test_modules_step.dependOn(tests.addModuleTests(b, .{
.test_filters = test_filters,
.test_target_filters = test_target_filters,
.test_target_filters_exclude = test_target_filters_exclude,
.test_extra_targets = test_extra_targets,
.root_src = "test/behavior.zig",
.name = "behavior",
Expand All @@ -444,6 +446,7 @@ pub fn build(b: *std.Build) !void {
test_modules_step.dependOn(tests.addModuleTests(b, .{
.test_filters = test_filters,
.test_target_filters = test_target_filters,
.test_target_filters_exclude = test_target_filters_exclude,
.test_extra_targets = test_extra_targets,
.root_src = "test/c_import.zig",
.name = "c-import",
Expand All @@ -459,6 +462,7 @@ pub fn build(b: *std.Build) !void {
test_modules_step.dependOn(tests.addModuleTests(b, .{
.test_filters = test_filters,
.test_target_filters = test_target_filters,
.test_target_filters_exclude = test_target_filters_exclude,
.test_extra_targets = test_extra_targets,
.root_src = "lib/compiler_rt.zig",
.name = "compiler-rt",
Expand All @@ -475,6 +479,7 @@ pub fn build(b: *std.Build) !void {
test_modules_step.dependOn(tests.addModuleTests(b, .{
.test_filters = test_filters,
.test_target_filters = test_target_filters,
.test_target_filters_exclude = test_target_filters_exclude,
.test_extra_targets = test_extra_targets,
.root_src = "lib/c.zig",
.name = "zigc",
Expand All @@ -491,6 +496,7 @@ pub fn build(b: *std.Build) !void {
test_modules_step.dependOn(tests.addModuleTests(b, .{
.test_filters = test_filters,
.test_target_filters = test_target_filters,
.test_target_filters_exclude = test_target_filters_exclude,
.test_extra_targets = test_extra_targets,
.root_src = "lib/std/std.zig",
.name = "std",
Expand Down Expand Up @@ -535,6 +541,7 @@ pub fn build(b: *std.Build) !void {
));
test_step.dependOn(tests.addCAbiTests(b, .{
.test_target_filters = test_target_filters,
.test_target_filters_exclude = test_target_filters_exclude,
.skip_non_native = skip_non_native,
.skip_release = skip_release,
}));
Expand All @@ -545,6 +552,7 @@ pub fn build(b: *std.Build) !void {
if (tests.addDebuggerTests(b, .{
.test_filters = test_filters,
.test_target_filters = test_target_filters,
.test_target_filters_exclude = test_target_filters_exclude,
.gdb = b.option([]const u8, "gdb", "path to gdb binary"),
.lldb = b.option([]const u8, "lldb", "path to lldb binary"),
.optimize_modes = optimization_modes,
Expand All @@ -556,6 +564,7 @@ pub fn build(b: *std.Build) !void {
.enable_llvm = enable_llvm,
.test_filters = test_filters,
.test_target_filters = test_target_filters,
.test_target_filters_exclude = test_target_filters_exclude,
})) |test_llvm_ir_step| test_step.dependOn(test_llvm_ir_step);

try addWasiUpdateStep(b, version);
Expand Down
4 changes: 3 additions & 1 deletion ci/aarch64-linux-debug.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,9 @@ ninja install
stage3-debug/bin/zig build test docs \
--maxrss 44918199637 \
-Dstatic-llvm \
-Dskip-non-native \
-Dtest-target-filter=aarch64-linux \
-Dtest-target-filter=freebsd \
-Dtest-target-filter=netbsd \
-Dtarget=native-native-musl \
--search-prefix "$PREFIX" \
--zig-lib-dir "$PWD/../lib" \
Expand Down
4 changes: 3 additions & 1 deletion ci/aarch64-linux-release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,9 @@ ninja install
stage3-release/bin/zig build test docs \
--maxrss 44918199637 \
-Dstatic-llvm \
-Dskip-non-native \
-Dtest-target-filter=aarch64-linux \
-Dtest-target-filter=freebsd \
-Dtest-target-filter=netbsd \
Comment on lines +55 to +57
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the CI scripts are very intentionally not specifying what to include, but what to skip.

Copy link
Member Author

@alexrp alexrp Jun 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The behavior we want is to build all native targets + FreeBSD targets + NetBSD targets. We can't express that with just excludes. Or well, we can, but it's going to be a lot of excludes, and I'm not sure that'd be better than this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I would also say that -Dskip-non-native isn't really saying what to skip; it's essentially just whitelisting "native". If we ignore the difference between writing native vs writing the "resolved native triple component" (e.g. aarch64 on an aarch64 host), then -Dtest-target-filter=aarch64-linux is precisely equivalent to -Dskip-non-native. If we stop ignoring that distinction, then -Dtest-target-filter=aarch64-linux clearly provides a superset of -Dskip-non-native.

In other words, you could essentially rephrase a single -Dtest-target-filter=xxxx as -Dskip-non-xxxx, and I'd argue that -Dskip-non-native is doing exactly that: "exclude all tests which do not match this filter" means exactly the same thing as "include all tests which do match this filter".

So, I struggle to buy that "the CI scripts are specifying what to skip, not what to include".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep in mind also that if we do go the exclusion route for this script, we'd have to update it every time a new kind of target is added to the test matrix, e.g. serenity, openbsd, ...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the point. Everything is supposed to be tested by default, not testing things being explicit exceptions that are obviously noted.

-Dtarget=native-native-musl \
--search-prefix "$PREFIX" \
--zig-lib-dir "$PWD/../lib" \
Expand Down
3 changes: 3 additions & 0 deletions ci/x86_64-linux-debug.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,9 @@ stage3-debug/bin/zig build test docs \
-fqemu \
-fwasmtime \
-Dstatic-llvm \
-Dtest-target-filter-exclude \
-Dtest-target-filter=freebsd \
-Dtest-target-filter=netbsd \
-Dtarget=native-native-musl \
--search-prefix "$PREFIX" \
--zig-lib-dir "$PWD/../lib" \
Expand Down
3 changes: 3 additions & 0 deletions ci/x86_64-linux-release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,9 @@ stage3-release/bin/zig build test docs \
-fqemu \
-fwasmtime \
-Dstatic-llvm \
-Dtest-target-filter-exclude \
-Dtest-target-filter=freebsd \
-Dtest-target-filter=netbsd \
-Dtarget=native-native-musl \
--search-prefix "$PREFIX" \
--zig-lib-dir "$PWD/../lib" \
Expand Down
18 changes: 13 additions & 5 deletions test/src/Cases.zig
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -515,13 +515,14 @@ pub fn lowerToTranslateCSteps(
parent_step: *std.Build.Step,
test_filters: []const []const u8,
test_target_filters: []const []const u8,
test_target_filters_exclude: bool,
target: std.Build.ResolvedTarget,
translate_c_options: TranslateCOptions,
) void {
const tests = @import("../tests.zig");
const test_translate_c_step = b.step("test-translate-c", "Run the C translation tests");
if (!translate_c_options.skip_translate_c) {
tests.addTranslateCTests(b, test_translate_c_step, test_filters, test_target_filters);
tests.addTranslateCTests(b, test_translate_c_step, test_filters, test_target_filters, test_target_filters_exclude);
parent_step.dependOn(test_translate_c_step);
}

Expand Down Expand Up @@ -600,22 +601,29 @@ pub fn lowerToBuildSteps(
parent_step: *std.Build.Step,
test_filters: []const []const u8,
test_target_filters: []const []const u8,
test_target_filters_exclude: bool,
) void {
const host = std.zig.system.resolveTargetQuery(.{}) catch |err|
std.debug.panic("unable to detect native host: {s}\n", .{@errorName(err)});
const cases_dir_path = b.build_root.join(b.allocator, &.{ "test", "cases" }) catch @panic("OOM");

for (self.cases.items) |case| {
for_cases: for (self.cases.items) |case| {
for (test_filters) |test_filter| {
if (std.mem.indexOf(u8, case.name, test_filter)) |_| break;
} else if (test_filters.len > 0) continue;

const triple_txt = case.target.result.zigTriple(b.allocator) catch @panic("OOM");

if (test_target_filters.len > 0) {
for (test_target_filters) |filter| {
if (std.mem.indexOf(u8, triple_txt, filter) != null) break;
} else continue;
if (test_target_filters_exclude) {
for (test_target_filters) |filter| {
if (std.mem.indexOf(u8, triple_txt, filter) != null) continue :for_cases;
}
} else {
for (test_target_filters) |filter| {
if (std.mem.indexOf(u8, triple_txt, filter) != null) break;
} else continue :for_cases;
}
}

const writefiles = b.addWriteFiles();
Expand Down
15 changes: 12 additions & 3 deletions test/src/Debugger.zig
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ root_step: *std.Build.Step,
pub const Options = struct {
test_filters: []const []const u8,
test_target_filters: []const []const u8,
test_target_filters_exclude: bool,
gdb: ?[]const u8,
lldb: ?[]const u8,
optimize_modes: []const std.builtin.OptimizeMode,
Expand Down Expand Up @@ -2447,12 +2448,20 @@ fn addTest(
if (std.mem.indexOf(u8, name, test_filter) != null) break;
} else return;
}

if (db.options.test_target_filters.len > 0) {
const triple_txt = target.resolved.result.zigTriple(db.b.allocator) catch @panic("OOM");
for (db.options.test_target_filters) |filter| {
if (std.mem.indexOf(u8, triple_txt, filter) != null) break;
} else return;
if (db.options.test_target_filters_exclude) {
for (db.options.test_target_filters) |filter| {
if (std.mem.indexOf(u8, triple_txt, filter) != null) return;
}
} else {
for (db.options.test_target_filters) |filter| {
if (std.mem.indexOf(u8, triple_txt, filter) != null) break;
} else return;
}
}

const files_wf = db.b.addWriteFiles();

const mod = db.b.createModule(.{
Expand Down
14 changes: 11 additions & 3 deletions test/src/LlvmIr.zig
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ pub const Options = struct {
enable_llvm: bool,
test_filters: []const []const u8,
test_target_filters: []const []const u8,
test_target_filters_exclude: bool,
};

const TestCase = struct {
Expand Down Expand Up @@ -74,11 +75,18 @@ pub fn addExact(

pub fn addCase(self: *LlvmIr, case: TestCase) void {
const target = self.b.resolveTargetQuery(case.params.target);

if (self.options.test_target_filters.len > 0) {
const triple_txt = target.result.zigTriple(self.b.allocator) catch @panic("OOM");
for (self.options.test_target_filters) |filter| {
if (std.mem.indexOf(u8, triple_txt, filter) != null) break;
} else return;
if (self.options.test_target_filters_exclude) {
for (self.options.test_target_filters) |filter| {
if (std.mem.indexOf(u8, triple_txt, filter) != null) return;
}
} else {
for (self.options.test_target_filters) |filter| {
if (std.mem.indexOf(u8, triple_txt, filter) != null) break;
} else return;
}
}

const name = std.fmt.allocPrint(self.b.allocator, "check llvm-ir {s}", .{case.name}) catch @panic("OOM");
Expand Down
14 changes: 10 additions & 4 deletions test/src/TranslateC.zig
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ step: *std.Build.Step,
test_index: usize,
test_filters: []const []const u8,
test_target_filters: []const []const u8,
test_target_filters_exclude: bool,

const TestCase = struct {
name: []const u8,
Expand Down Expand Up @@ -97,10 +98,15 @@ pub fn addCase(self: *TranslateCContext, case: *const TestCase) void {

if (self.test_target_filters.len > 0) {
const triple_txt = target.result.zigTriple(b.allocator) catch @panic("OOM");

for (self.test_target_filters) |filter| {
if (std.mem.indexOf(u8, triple_txt, filter) != null) break;
} else return;
if (self.test_target_filters_exclude) {
for (self.test_target_filters) |filter| {
if (std.mem.indexOf(u8, triple_txt, filter) != null) return;
}
} else {
for (self.test_target_filters) |filter| {
if (std.mem.indexOf(u8, triple_txt, filter) != null) break;
} else return;
}
}

const write_src = b.addWriteFiles();
Expand Down
Loading
Loading