support fetch plugins to download dependencies from less common protocols

[andrewrk]

Extracted from #14265.

Zig will have built-in support for fetching dependencies from some protocols, such as http, https, gzip, tar, and some others. But there will always be new or exotic protocols. One such example would be ipfs. Not really ubiquitous enough to support directly, but could be interesting for some people to use.

For this I propose a "fetch plugin" system.

Fetch plugins would be specified in build.zig.zon like this:

.{
    // ...
    .fetch_plugins = .{
        .ipfs = .{
            .url = "git+ssh://git@example.com/foo.tar.gz#6f987aba83414319c3afba57a9f49a71f6e13c8e",
            .hash = "sha256=c9b30cffc40999d2c078ff350cbcee642970a224fe123c756d0892f876cf1aae",
        },
        .@"git+ssh" = .{
            .url = "http://example.com/bar.tar.gz",
            .hash = "sha256=9ba4f49895b174a3f918d489238acbc146bd393575062b2e3be33488b688e36f",
        },
    },
    // ...
}

The url fields within fetch_plugins must use a built-in URI scheme or a different fetch plugin from the same manifest.

Implementing this proposal will lift out this code from the zig compiler:

zig/src/main.zig

Lines 4080 to 4126 in 7cb2f92

	if (!build_options.omit_pkg_fetching_code) {
	var http_client: std.http.Client = .{ .allocator = gpa };
	defer http_client.deinit();
	try http_client.rescanRootCertificates();
	// Here we provide an import to the build runner that allows using reflection to find
	// all of the dependencies. Without this, there would be no way to use `@import` to
	// access dependencies by name, since `@import` requires string literals.
	var dependencies_source = std.ArrayList(u8).init(gpa);
	defer dependencies_source.deinit();
	try dependencies_source.appendSlice("pub const imports = struct {\n");
	// This will go into the same package. It contains the file system paths
	// to all the build.zig files.
	var build_roots_source = std.ArrayList(u8).init(gpa);
	defer build_roots_source.deinit();
	// Here we borrow main package's table and will replace it with a fresh
	// one after this process completes.
	main_pkg.fetchAndAddDependencies(
	&thread_pool,
	&http_client,
	build_directory,
	global_cache_directory,
	local_cache_directory,
	&dependencies_source,
	&build_roots_source,
	"",
	) catch |err| switch (err) {
	error.PackageFetchFailed => process.exit(1),
	else => |e| return e,
	};
	try dependencies_source.appendSlice("};\npub const build_root = struct {\n");
	try dependencies_source.appendSlice(build_roots_source.items);
	try dependencies_source.appendSlice("};\n");
	const deps_pkg = try Package.createFilePkg(
	gpa,
	local_cache_directory,
	"dependencies.zig",
	dependencies_source.items,
	);
	mem.swap(Package.Table, &main_pkg.table, &deps_pkg.table);
	try main_pkg.addAndAdopt(gpa, "@dependencies", deps_pkg);
	}

...and move it to a new file lib/fetch_runner.zig. This is similar to build_runner.zig and test_runner.zig and has the responsibility to fetch the full dependency tree. It will deal with fetch plugins by fetching them, and then rebuilding the fetch runner itself, multiple times if necessary, until everything is fetched.

Once everything is fetched, build_runner.zig proceeds as usual. Note that in the case when everything is already fetched, the fetch_runner will not be executed because the open() syscalls on the first-level dependencies based on their hashes all succeeded.

[AdamGoertz]

How would this fit into the framework proposed in #14286? It sounds like the goal in that issue is to make it safer to run un-audited build.zig files from dependencies by sandboxing them with WASM, but this proposal would also allow arbitrary code execution by downloading and running fetch plugins, similarly to running someone else's build.zig.