Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rework std.Progress #20059

Merged
merged 60 commits into from
May 28, 2024
Merged
Show file tree
Hide file tree
Changes from 58 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
d6e8ba3
start reworking std.Progress
andrewrk May 3, 2024
e1e4de2
progress progress
andrewrk May 22, 2024
a3c9511
rework std.Progress again
andrewrk May 23, 2024
66c3b6a
fix terminal repainting
andrewrk May 23, 2024
582acdf
keep the cursor at the end instead of beginning
andrewrk May 23, 2024
67e08e7
fix clearing and sibling iteration
andrewrk May 23, 2024
ed36470
std.Progress: truncate trailing newline
andrewrk May 23, 2024
f071164
std.Progress: child process sends updates via IPC
andrewrk May 23, 2024
df46f5a
std.Progress: include subtrees from child processes
andrewrk May 24, 2024
3a768bd
std.Progress: save a copy of IPC data
andrewrk May 24, 2024
c01cfde
std.process.Child: fix ZIG_PROGRESS env var handling
andrewrk May 24, 2024
f6873c6
std.Progress: fix using saved IPC data
andrewrk May 24, 2024
f97c2f2
update the codebase for the new std.Progress API
andrewrk May 24, 2024
795c579
test runner: update to new std.Progress API
andrewrk May 24, 2024
a486392
std.Build.Step: don't create an empty progress node
andrewrk May 24, 2024
1cf1cb6
std.debug.Trace: follow the struct default field guidance
andrewrk May 24, 2024
b27fdf9
Compilation: only create progress nodes for work actually being done
andrewrk May 24, 2024
2233d95
main: avoid creating multiple std.Progress instances
andrewrk May 24, 2024
70e39c1
std.Progress: fixes
andrewrk May 24, 2024
352dc2c
compiler: show decl name in progress node
andrewrk May 24, 2024
2759173
zig build: lock stderr while executing the build runner
andrewrk May 24, 2024
516366f
std.Progress: skip printing root node when it is empty
andrewrk May 24, 2024
ca03c9c
std.Progress: fix race condition with IPC nodes
andrewrk May 24, 2024
9331da8
std.Build.Step.Run: don't create empty progress node
andrewrk May 24, 2024
acdf988
std.process.Child: prevent racing children from inheriting progress p…
mlugg May 26, 2024
d403d8c
Module: fix and improve progress reporting
mlugg May 26, 2024
d77f5e7
Progress: fix compile errors on windows
jacobly0 May 26, 2024
e8907f9
std.Progress: correct the top level doc comments
andrewrk May 26, 2024
e2e61f3
std.process.Child: comptime assert to protect prog_fileno
andrewrk May 26, 2024
e820173
Compilation: fix sub-compilations given wrong progress node
andrewrk May 26, 2024
eb718ce
std.process: fix compilation on 32-bit targets
andrewrk May 26, 2024
d3b03ed
frontend: fix use of undefined progress node
andrewrk May 26, 2024
7fe72d5
std.Progress: move global preallocations to thread memory
andrewrk May 26, 2024
807b613
std.Progress: move more global preallocations to thread memory
andrewrk May 26, 2024
52ed54d
std.Progress: truncate IPC data exceeding preallocated buffers
andrewrk May 26, 2024
11f8947
std.Progress: avoid scrolling the PS1 off the terminal
andrewrk May 27, 2024
ea7d8ec
std.Progress: smaller type for parents and robustify
andrewrk May 27, 2024
0ca2b4e
std.Progress: use std.log.debug rather than warn
andrewrk May 27, 2024
849693f
zig build: give a root progress name
andrewrk May 27, 2024
52ffdec
std.Progress: keep cursor on newline
andrewrk May 27, 2024
6145819
std.Progress: handle when terminal write buffer too small
andrewrk May 27, 2024
dc3a192
std.Progress: count newlines more accurately
andrewrk May 27, 2024
64c6a50
std.Progress: elide root node if empty
andrewrk May 27, 2024
c9587d3
CLI: add missing call to root progress node end()
andrewrk May 27, 2024
45b505c
std.process.cleanExit: lock stderr before exiting
andrewrk May 27, 2024
03073d6
build runner: use "configure" for the progress name
andrewrk May 27, 2024
0937992
resinator: update to new progress API
andrewrk May 27, 2024
a5e4fe5
std.Build.Step.Run: account for new environment variable
andrewrk May 27, 2024
eea7e5e
std.Progress: adjust the timings a little bit
andrewrk May 27, 2024
dcf9cae
std.Progress: handle big-endian targets
andrewrk May 27, 2024
2367a1f
std.Progress: handle short writes
andrewrk May 27, 2024
4438925
fix zig translate-c creating root progress node twice
andrewrk May 27, 2024
bb1f4d2
translate-c tests: skip_foreign_checks=true
andrewrk May 27, 2024
aca7feb
std.Progress: fix race condition with setIpcFd
andrewrk May 27, 2024
b7889f2
zig build: respect --color argument
andrewrk May 27, 2024
947a3a1
std.process.Child: fix spawning child proc with new cwd fd
andrewrk May 27, 2024
5bdfe22
CLI: actually fix translate-c creating root progress node twice
andrewrk May 28, 2024
65a0e14
std.Progress: relax some of the atomic orderings
andrewrk May 28, 2024
40afac4
std.Progress: Use Windows console API calls when ANSI escape codes ar…
squeek502 May 28, 2024
3a3d218
std.Progress: better Windows support
andrewrk May 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions lib/compiler/aro/aro/Diagnostics.zig
Original file line number Diff line number Diff line change
Expand Up @@ -528,7 +528,7 @@ const MsgWriter = struct {
config: std.io.tty.Config,

fn init(config: std.io.tty.Config) MsgWriter {
std.debug.getStderrMutex().lock();
std.debug.lockStdErr();
return .{
.w = std.io.bufferedWriter(std.io.getStdErr().writer()),
.config = config,
Expand All @@ -537,7 +537,7 @@ const MsgWriter = struct {

pub fn deinit(m: *MsgWriter) void {
m.w.flush() catch {};
std.debug.getStderrMutex().unlock();
std.debug.unlockStdErr();
}

pub fn print(m: *MsgWriter, comptime fmt: []const u8, args: anytype) void {
Expand Down
26 changes: 13 additions & 13 deletions lib/compiler/build_runner.zig
Original file line number Diff line number Diff line change
Expand Up @@ -289,13 +289,14 @@ pub fn main() !void {
.windows_api => {},
}

var progress: std.Progress = .{ .dont_print_on_dumb = true };
const main_progress_node = progress.start("", 0);
const main_progress_node = std.Progress.start(.{
.disable_printing = (color == .off),
});

builder.debug_log_scopes = debug_log_scopes.items;
builder.resolveInstallPrefix(install_prefix, dir_list);
{
var prog_node = main_progress_node.start("user build.zig logic", 0);
var prog_node = main_progress_node.start("Configure", 0);
defer prog_node.end();
try builder.runBuild(root);
}
Expand Down Expand Up @@ -385,7 +386,7 @@ fn runStepNames(
arena: std.mem.Allocator,
b: *std.Build,
step_names: []const []const u8,
parent_prog_node: *std.Progress.Node,
parent_prog_node: std.Progress.Node,
thread_pool_options: std.Thread.Pool.Options,
run: *Run,
seed: u32,
Expand Down Expand Up @@ -452,7 +453,7 @@ fn runStepNames(
{
defer parent_prog_node.end();

var step_prog = parent_prog_node.start("steps", step_stack.count());
const step_prog = parent_prog_node.start("steps", step_stack.count());
defer step_prog.end();

var wait_group: std.Thread.WaitGroup = .{};
Expand All @@ -467,7 +468,7 @@ fn runStepNames(
if (step.state == .skipped_oom) continue;

thread_pool.spawnWg(&wait_group, workerMakeOneStep, .{
&wait_group, &thread_pool, b, step, &step_prog, run,
&wait_group, &thread_pool, b, step, step_prog, run,
});
}
}
Expand Down Expand Up @@ -891,7 +892,7 @@ fn workerMakeOneStep(
thread_pool: *std.Thread.Pool,
b: *std.Build,
s: *Step,
prog_node: *std.Progress.Node,
prog_node: std.Progress.Node,
run: *Run,
) void {
// First, check the conditions for running this step. If they are not met,
Expand Down Expand Up @@ -941,11 +942,10 @@ fn workerMakeOneStep(
}
}

var sub_prog_node = prog_node.start(s.name, 0);
sub_prog_node.activate();
const sub_prog_node = prog_node.start(s.name, 0);
defer sub_prog_node.end();

const make_result = s.make(&sub_prog_node);
const make_result = s.make(sub_prog_node);

// No matter the result, we want to display error/warning messages.
const show_compile_errors = !run.prominent_compile_errors and
Expand All @@ -954,8 +954,8 @@ fn workerMakeOneStep(
const show_stderr = s.result_stderr.len > 0;

if (show_error_msgs or show_compile_errors or show_stderr) {
sub_prog_node.context.lock_stderr();
defer sub_prog_node.context.unlock_stderr();
std.debug.lockStdErr();
defer std.debug.unlockStdErr();

printErrorMessages(b, s, run) catch {};
}
Expand Down Expand Up @@ -1225,7 +1225,7 @@ fn cleanExit() void {
process.exit(0);
}

const Color = enum { auto, off, on };
const Color = std.zig.Color;
const Summary = enum { all, new, failures, none };

fn get_tty_conf(color: Color, stderr: File) std.io.tty.Config {
Expand Down
4 changes: 2 additions & 2 deletions lib/compiler/resinator/cli.zig
Original file line number Diff line number Diff line change
Expand Up @@ -108,8 +108,8 @@ pub const Diagnostics = struct {
}

pub fn renderToStdErr(self: *Diagnostics, args: []const []const u8, config: std.io.tty.Config) void {
std.debug.getStderrMutex().lock();
defer std.debug.getStderrMutex().unlock();
std.debug.lockStdErr();
defer std.debug.unlockStdErr();
const stderr = std.io.getStdErr().writer();
self.renderToWriter(args, stderr, config) catch return;
}
Expand Down
4 changes: 2 additions & 2 deletions lib/compiler/resinator/errors.zig
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@ pub const Diagnostics = struct {
}

pub fn renderToStdErr(self: *Diagnostics, cwd: std.fs.Dir, source: []const u8, tty_config: std.io.tty.Config, source_mappings: ?SourceMappings) void {
std.debug.getStderrMutex().lock();
defer std.debug.getStderrMutex().unlock();
std.debug.lockStdErr();
defer std.debug.unlockStdErr();
const stderr = std.io.getStdErr().writer();
for (self.errors.items) |err_details| {
renderErrorMessage(self.allocator, stderr, tty_config, cwd, err_details, source, self.strings.items, source_mappings) catch return;
Expand Down
6 changes: 0 additions & 6 deletions lib/compiler/resinator/main.zig
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,6 @@ pub fn main() !void {
},
};

if (zig_integration) {
// Send progress with a special string to indicate that the building of the
// resinator binary is finished and we've moved on to actually compiling the .rc file
try error_handler.server.serveStringMessage(.progress, "<resinator>");
}

var options = options: {
var cli_diagnostics = cli.Diagnostics.init(allocator);
defer cli_diagnostics.deinit();
Expand Down
31 changes: 19 additions & 12 deletions lib/compiler/test_runner.zig
Original file line number Diff line number Diff line change
Expand Up @@ -129,12 +129,11 @@ fn mainTerminal() void {
var ok_count: usize = 0;
var skip_count: usize = 0;
var fail_count: usize = 0;
var progress = std.Progress{
.dont_print_on_dumb = true,
};
const root_node = progress.start("Test", test_fn_list.len);
const have_tty = progress.terminal != null and
(progress.supports_ansi_escape_codes or progress.is_windows_terminal);
const root_node = std.Progress.start(.{
.root_name = "Test",
.estimated_total_items = test_fn_list.len,
});
const have_tty = std.io.getStdErr().isTty();

var async_frame_buffer: []align(builtin.target.stackAlignment()) u8 = undefined;
// TODO this is on the next line (using `undefined` above) because otherwise zig incorrectly
Expand All @@ -151,11 +150,9 @@ fn mainTerminal() void {
}
std.testing.log_level = .warn;

var test_node = root_node.start(test_fn.name, 0);
test_node.activate();
progress.refresh();
const test_node = root_node.start(test_fn.name, 0);
if (!have_tty) {
std.debug.print("{d}/{d} {s}... ", .{ i + 1, test_fn_list.len, test_fn.name });
std.debug.print("{d}/{d} {s}...", .{ i + 1, test_fn_list.len, test_fn.name });
}
if (test_fn.func()) |_| {
ok_count += 1;
Expand All @@ -164,12 +161,22 @@ fn mainTerminal() void {
} else |err| switch (err) {
error.SkipZigTest => {
skip_count += 1;
progress.log("SKIP\n", .{});
if (have_tty) {
std.debug.print("{d}/{d} {s}...SKIP\n", .{ i + 1, test_fn_list.len, test_fn.name });
} else {
std.debug.print("SKIP\n", .{});
}
test_node.end();
},
else => {
fail_count += 1;
progress.log("FAIL ({s})\n", .{@errorName(err)});
if (have_tty) {
std.debug.print("{d}/{d} {s}...FAIL ({s})\n", .{
i + 1, test_fn_list.len, test_fn.name, @errorName(err),
});
} else {
std.debug.print("FAIL ({s})\n", .{@errorName(err)});
}
if (@errorReturnTrace()) |trace| {
std.debug.dumpStackTrace(trace.*);
}
Expand Down
10 changes: 5 additions & 5 deletions lib/std/Build.zig
Original file line number Diff line number Diff line change
Expand Up @@ -1059,7 +1059,7 @@ pub fn getUninstallStep(b: *Build) *Step {
return &b.uninstall_tls.step;
}

fn makeUninstall(uninstall_step: *Step, prog_node: *std.Progress.Node) anyerror!void {
fn makeUninstall(uninstall_step: *Step, prog_node: std.Progress.Node) anyerror!void {
_ = prog_node;
const uninstall_tls: *TopLevelStep = @fieldParentPtr("step", uninstall_step);
const b: *Build = @fieldParentPtr("uninstall_tls", uninstall_tls);
Expand Down Expand Up @@ -2281,10 +2281,10 @@ pub const LazyPath = union(enum) {
.cwd_relative => |p| return src_builder.pathFromCwd(p),
.generated => |gen| {
var file_path: []const u8 = gen.file.step.owner.pathFromRoot(gen.file.path orelse {
std.debug.getStderrMutex().lock();
std.debug.lockStdErr();
const stderr = std.io.getStdErr();
dumpBadGetPathHelp(gen.file.step, stderr, src_builder, asking_step) catch {};
std.debug.getStderrMutex().unlock();
std.debug.unlockStdErr();
@panic("misconfigured build script");
});

Expand Down Expand Up @@ -2351,8 +2351,8 @@ fn dumpBadDirnameHelp(
comptime msg: []const u8,
args: anytype,
) anyerror!void {
debug.getStderrMutex().lock();
defer debug.getStderrMutex().unlock();
debug.lockStdErr();
defer debug.unlockStdErr();

const stderr = io.getStdErr();
const w = stderr.writer();
Expand Down
19 changes: 5 additions & 14 deletions lib/std/Build/Step.zig
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ pub const TestResults = struct {
}
};

pub const MakeFn = *const fn (step: *Step, prog_node: *std.Progress.Node) anyerror!void;
pub const MakeFn = *const fn (step: *Step, prog_node: std.Progress.Node) anyerror!void;

pub const State = enum {
precheck_unstarted,
Expand Down Expand Up @@ -176,7 +176,7 @@ pub fn init(options: StepOptions) Step {
/// If the Step's `make` function reports `error.MakeFailed`, it indicates they
/// have already reported the error. Otherwise, we add a simple error report
/// here.
pub fn make(s: *Step, prog_node: *std.Progress.Node) error{ MakeFailed, MakeSkipped }!void {
pub fn make(s: *Step, prog_node: std.Progress.Node) error{ MakeFailed, MakeSkipped }!void {
const arena = s.owner.allocator;

s.makeFn(s, prog_node) catch |err| switch (err) {
Expand Down Expand Up @@ -217,7 +217,7 @@ pub fn getStackTrace(s: *Step) ?std.builtin.StackTrace {
};
}

fn makeNoOp(step: *Step, prog_node: *std.Progress.Node) anyerror!void {
fn makeNoOp(step: *Step, prog_node: std.Progress.Node) anyerror!void {
_ = prog_node;

var all_cached = true;
Expand Down Expand Up @@ -303,7 +303,7 @@ pub fn addError(step: *Step, comptime fmt: []const u8, args: anytype) error{OutO
pub fn evalZigProcess(
s: *Step,
argv: []const []const u8,
prog_node: *std.Progress.Node,
prog_node: std.Progress.Node,
) !?[]const u8 {
assert(argv.len != 0);
const b = s.owner;
Expand All @@ -319,6 +319,7 @@ pub fn evalZigProcess(
child.stdout_behavior = .Pipe;
child.stderr_behavior = .Pipe;
child.request_resource_usage_statistics = true;
child.progress_node = prog_node;

child.spawn() catch |err| return s.fail("unable to spawn {s}: {s}", .{
argv[0], @errorName(err),
Expand All @@ -337,11 +338,6 @@ pub fn evalZigProcess(
const Header = std.zig.Server.Message.Header;
var result: ?[]const u8 = null;

var node_name: std.ArrayListUnmanaged(u8) = .{};
defer node_name.deinit(gpa);
var sub_prog_node = prog_node.start("", 0);
defer sub_prog_node.end();

const stdout = poller.fifo(.stdout);

poll: while (true) {
Expand Down Expand Up @@ -379,11 +375,6 @@ pub fn evalZigProcess(
.extra = extra_array,
};
},
.progress => {
node_name.clearRetainingCapacity();
try node_name.appendSlice(gpa, body);
sub_prog_node.setName(node_name.items);
},
.emit_bin_path => {
const EbpHdr = std.zig.Server.Message.EmitBinPath;
const ebp_hdr = @as(*align(1) const EbpHdr, @ptrCast(body));
Expand Down
2 changes: 1 addition & 1 deletion lib/std/Build/Step/CheckFile.zig
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ pub fn setName(check_file: *CheckFile, name: []const u8) void {
check_file.step.name = name;
}

fn make(step: *Step, prog_node: *std.Progress.Node) !void {
fn make(step: *Step, prog_node: std.Progress.Node) !void {
_ = prog_node;
const b = step.owner;
const check_file: *CheckFile = @fieldParentPtr("step", step);
Expand Down
2 changes: 1 addition & 1 deletion lib/std/Build/Step/CheckObject.zig
Original file line number Diff line number Diff line change
Expand Up @@ -550,7 +550,7 @@ pub fn checkComputeCompare(
check_object.checks.append(check) catch @panic("OOM");
}

fn make(step: *Step, prog_node: *std.Progress.Node) !void {
fn make(step: *Step, prog_node: std.Progress.Node) !void {
_ = prog_node;
const b = step.owner;
const gpa = b.allocator;
Expand Down
6 changes: 3 additions & 3 deletions lib/std/Build/Step/Compile.zig
Original file line number Diff line number Diff line change
Expand Up @@ -967,7 +967,7 @@ fn getGeneratedFilePath(compile: *Compile, comptime tag_name: []const u8, asking
const maybe_path: ?*GeneratedFile = @field(compile, tag_name);

const generated_file = maybe_path orelse {
std.debug.getStderrMutex().lock();
std.debug.lockStdErr();
const stderr = std.io.getStdErr();

std.Build.dumpBadGetPathHelp(&compile.step, stderr, compile.step.owner, asking_step) catch {};
Expand All @@ -976,7 +976,7 @@ fn getGeneratedFilePath(compile: *Compile, comptime tag_name: []const u8, asking
};

const path = generated_file.path orelse {
std.debug.getStderrMutex().lock();
std.debug.lockStdErr();
const stderr = std.io.getStdErr();

std.Build.dumpBadGetPathHelp(&compile.step, stderr, compile.step.owner, asking_step) catch {};
Expand All @@ -987,7 +987,7 @@ fn getGeneratedFilePath(compile: *Compile, comptime tag_name: []const u8, asking
return path;
}

fn make(step: *Step, prog_node: *std.Progress.Node) !void {
fn make(step: *Step, prog_node: std.Progress.Node) !void {
const b = step.owner;
const arena = b.allocator;
const compile: *Compile = @fieldParentPtr("step", step);
Expand Down
2 changes: 1 addition & 1 deletion lib/std/Build/Step/ConfigHeader.zig
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ fn putValue(config_header: *ConfigHeader, field_name: []const u8, comptime T: ty
}
}

fn make(step: *Step, prog_node: *std.Progress.Node) !void {
fn make(step: *Step, prog_node: std.Progress.Node) !void {
_ = prog_node;
const b = step.owner;
const config_header: *ConfigHeader = @fieldParentPtr("step", step);
Expand Down
2 changes: 1 addition & 1 deletion lib/std/Build/Step/Fmt.zig
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ pub fn create(owner: *std.Build, options: Options) *Fmt {
return fmt;
}

fn make(step: *Step, prog_node: *std.Progress.Node) !void {
fn make(step: *Step, prog_node: std.Progress.Node) !void {
// zig fmt is fast enough that no progress is needed.
_ = prog_node;

Expand Down
2 changes: 1 addition & 1 deletion lib/std/Build/Step/InstallArtifact.zig
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ pub fn create(owner: *std.Build, artifact: *Step.Compile, options: Options) *Ins
return install_artifact;
}

fn make(step: *Step, prog_node: *std.Progress.Node) !void {
fn make(step: *Step, prog_node: std.Progress.Node) !void {
_ = prog_node;
const install_artifact: *InstallArtifact = @fieldParentPtr("step", step);
const b = step.owner;
Expand Down
2 changes: 1 addition & 1 deletion lib/std/Build/Step/InstallDir.zig
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ pub fn create(owner: *std.Build, options: Options) *InstallDir {
return install_dir;
}

fn make(step: *Step, prog_node: *std.Progress.Node) !void {
fn make(step: *Step, prog_node: std.Progress.Node) !void {
_ = prog_node;
const b = step.owner;
const install_dir: *InstallDir = @fieldParentPtr("step", step);
Expand Down
2 changes: 1 addition & 1 deletion lib/std/Build/Step/InstallFile.zig
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ pub fn create(
return install_file;
}

fn make(step: *Step, prog_node: *std.Progress.Node) !void {
fn make(step: *Step, prog_node: std.Progress.Node) !void {
_ = prog_node;
const b = step.owner;
const install_file: *InstallFile = @fieldParentPtr("step", step);
Expand Down
2 changes: 1 addition & 1 deletion lib/std/Build/Step/ObjCopy.zig
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ pub fn getOutputSeparatedDebug(objcopy: *const ObjCopy) ?std.Build.LazyPath {
return if (objcopy.output_file_debug) |*file| .{ .generated = .{ .file = file } } else null;
}

fn make(step: *Step, prog_node: *std.Progress.Node) !void {
fn make(step: *Step, prog_node: std.Progress.Node) !void {
const b = step.owner;
const objcopy: *ObjCopy = @fieldParentPtr("step", step);

Expand Down
Loading